1. What Is MediaCrawler?

MediaCrawler is the most comprehensive open-source crawler for Chinese social media platforms, with over 64,000 GitHub stars. Built on Python + Playwright browser automation, it preserves login sessions to call platform APIs directly — no need to reverse-engineer complex JavaScript encryption.

Core positioning: A learning tool for data analysts, public opinion researchers, and content operators. It supports one-click data collection from 7 major Chinese social platforms.

Unlike general-purpose crawlers like Crawl4AI, MediaCrawler is purpose-built for Chinese social media with built-in signing logic and data parsing rules. For Western platform data collection, check out Wiseflow.

2. Supported Platforms and Feature Matrix

Platform Keyword Search Post by ID Nested Comments Creator Page Login Cache IP Proxy Pool Comment Word Cloud
Xiaohongshu (RED) ✅ ✅ ✅ ✅ ✅ ✅ ✅
Douyin (TikTok CN) ✅ ✅ ✅ ✅ ✅ ✅ ✅
Kuaishou ✅ ✅ ✅ ✅ ✅ ✅ ✅
Bilibili ✅ ✅ ✅ ✅ ✅ ✅ ✅
Weibo ✅ ✅ ✅ ✅ ✅ ✅ ✅
Baidu Tieba ✅ ✅ ✅ ✅ ✅ ✅ ✅
Zhihu ✅ ✅ ✅ ✅ ✅ ✅ ✅

All 7 platforms share identical feature support — MediaCrawler's biggest architectural advantage is its unified interface design. Switching platforms requires changing just one parameter.

3. Technical Architecture

MediaCrawler's tech stack is designed around "minimizing anti-detection risk":

┌─────────────────────────────────────────────┐
│            MediaCrawler Architecture         │
├─────────────────────────────────────────────┤
│  User Layer    CLI / WebUI Dashboard         │
├─────────────────────────────────────────────┤
│  Scheduler     Platform Dispatcher → Crawler │
├─────────────────────────────────────────────┤
│  Execution     Playwright (CDP Mode)         │
│                ├─ Login Manager (QR/Phone)   │
│                ├─ JS Signing (Browser Ctx)   │
│                └─ Anti-Detection (Real FP)   │
├─────────────────────────────────────────────┤
│  Data Layer    CSV / JSON / JSONL / Excel    │
│                SQLite / MySQL / PostgreSQL    │
└─────────────────────────────────────────────┘

Key technical principles:

  1. CDP Mode (Recommended): Connects to your existing Chrome browser via Chrome DevTools Protocol, reusing real cookies, extensions, and browsing history for optimal anti-detection
  2. Signature Reuse: Leverages the authenticated browser context to extract platform signing parameters via JS expressions — no reverse engineering needed
  3. Login State Caching: After first QR code login, cookies are saved and reused automatically on subsequent runs

4. Installation and Quick Start

4.1 Environment Setup

# 1. Install uv (recommended Python package manager)
curl -LsSf https://astral.sh/uv/install.sh | sh

# 2. Confirm Node.js >= 16 (required for Douyin and Zhihu signing)
node --version

# 3. Clone the project
git clone https://github.com/NanmiCoder/MediaCrawler.git
cd MediaCrawler

# 4. Install dependencies (uv handles Python version and deps)
uv sync

4.2 Chrome Configuration (CDP Mode)

MediaCrawler defaults to CDP mode. Configure Chrome:

  1. Install Chrome >= 144
  2. Navigate to chrome://inspect/#remote-debugging
  3. Check "Allow remote debugging for this browser instance"
  4. Confirm it shows Server running at: 127.0.0.1:9222

To disable CDP mode, set ENABLE_CDP_MODE = False in config/base_config.py.

4.3 First Run

# Search Xiaohongshu notes by keyword (default: 15 notes)
uv run main.py --platform xhs --lt qrcode --type search

# First run shows a QR code — scan with Xiaohongshu app to login
# Login state is cached automatically for future runs

5. Core Configuration

config/base_config.py is the key configuration file:

# Target platform: xhs | dy | ks | bili | wb | tieba | zhihu
PLATFORM = "xhs"

# Search keywords (comma-separated)
KEYWORDS = "Python tutorial,side hustle coding"

# Login method: qrcode | phone | cookie
LOGIN_TYPE = "qrcode"

# Crawl type: search | detail | creator
CRAWLER_TYPE = "search"

# Data storage format: csv | db | json | jsonl | sqlite | excel | postgres
SAVE_DATA_OPTION = "jsonl"

# Max notes per keyword
CRAWLER_MAX_NOTES_COUNT = 15

# Concurrency (keep at 1 to avoid rate limiting)
MAX_CONCURRENCY_NUM = 1

# Enable comment scraping
ENABLE_GET_COMMENTS = True

# Enable nested comment scraping
ENABLE_GET_SUB_COMMENTS = False

# IP proxy configuration
ENABLE_IP_PROXY = False
IP_PROXY_PROVIDER_NAME = "kuaidaili"

6. Platform-Specific Examples

6.1 Xiaohongshu (RED / Little Red Book)

# Search notes by keyword
uv run main.py --platform xhs --lt qrcode --type search

# Scrape specific note by ID
uv run main.py --platform xhs --lt qrcode --type detail

# Scrape all notes from a creator's page
uv run main.py --platform xhs --lt qrcode --type creator

Data fields include: note title, body text, image list, likes, favorites, comment count, author info, and publish time.

6.2 Douyin (Chinese TikTok)

# Search videos
uv run main.py --platform dy --lt qrcode --type search

# Specific video by ID
uv run main.py --platform dy --lt qrcode --type detail

Douyin requires Node.js for signature computation. Data includes video title, play URL, cover image, likes, comments, and shares.

6.3 Bilibili

# Search videos
uv run main.py --platform bili --lt qrcode --type search

# Specific video by BV ID
uv run main.py --platform bili --lt qrcode --type detail

Bilibili data includes: video title, BV ID, play count, danmaku count, likes/coins/favorites, and uploader info.

6.4 Weibo / Tieba / Zhihu / Kuaishou

# Weibo
uv run main.py --platform wb --lt qrcode --type search

# Baidu Tieba
uv run main.py --platform tieba --lt qrcode --type search

# Zhihu
uv run main.py --platform zhihu --lt qrcode --type search

# Kuaishou
uv run main.py --platform ks --lt qrcode --type search

All platforms use the same command format — just swap the --platform parameter.

7. Data Storage and Export

MediaCrawler supports 7 storage formats, with data saved to the data/ directory by default:

Format Best For Characteristics
JSONL Daily collection (default) Line-by-line, resume-friendly
CSV Excel analysis Open directly in spreadsheet apps
JSON Program integration Standard format, nested structures
Excel Report output Formatted spreadsheet
SQLite Local database Lightweight queries, no server needed
MySQL Production Deduplication support, scalable
PostgreSQL Enterprise Full relational database 
# Switch to MySQL storage
SAVE_DATA_OPTION = "db"

# Configure connection in db_config.py
DB_HOST = "localhost"
DB_PORT = 3306
DB_USER = "root"
DB_PASSWORD = "your_password"
DB_NAME = "media_crawler"

8. Advanced Usage

8.1 WebUI Dashboard

# Terminal 1: Start backend API
uv run uvicorn api.main:app --port 8080 --reload

# Terminal 2: Start frontend (dev mode)
cd webui && npm install && npm run dev
# Visit http://localhost:5173/

# Or build for production
cd webui && npm install && npm run build
# Only need the API server, visit http://localhost:8080

The WebUI provides a visual configuration interface — no command line needed.

8.2 IP Proxy Pool

For large-scale collection, enable proxy rotation:

ENABLE_IP_PROXY = True
IP_PROXY_POOL_COUNT = 2
IP_PROXY_PROVIDER_NAME = "kuaidaili"  # kuaidaili | wandouhttp | static

# Static proxy
IP_PROXY_PROVIDER_NAME = "static"
STATIC_PROXY_URL = "http://user:password@your_proxy:port"

8.3 Concurrency and Rate Control

# Concurrency (set carefully — too high triggers rate limits)
MAX_CONCURRENCY_NUM = 1

# Max items per keyword
CRAWLER_MAX_NOTES_COUNT = 50

# Starting page
START_PAGE = 1

8.4 Comment Word Cloud

MediaCrawler includes built-in word cloud generation from comments:

# Enable in base_config.py
# Requires wordcloud and other dependencies
ENABLE_WORD_CLOUD = True

Important: Understand these risks before using MediaCrawler:

  1. Legal compliance: Follow your local data protection laws (Cybersecurity Law, Data Security Law, PIPL in China)
  2. Platform rules: Respect each platform's robots.txt and terms of service
  3. Rate control: Keep request frequency reasonable — don't overload platforms
  4. Usage scope: For learning and research only — no commercial use
  5. Privacy: Do not scrape or distribute personal private information

10. Comparison with Other Tools

Feature MediaCrawler Snscrape Tweepy Custom Crawler
Target 7 Chinese platforms Twitter/X Twitter/X Any platform
Anti-detection CDP real browser None Official API DIY
Login QR/Phone/Cookie None needed OAuth Manual
Storage 7 formats CLI output JSON DIY
Difficulty Low (config & go) Low Medium High
Comments ✅ Nested ❌ ✅ Dev needed
Maintenance Active (64K+ Stars) Stalled Active Self-maintained

For general web scraping, try Crawl4AI; for news feed monitoring with AI summaries, try Wiseflow. MediaCrawler's unique value is its deep adaptation to Chinese social media.

11. FAQ

What Python version does MediaCrawler support?

Python 3.11 is recommended — it's the primary test version. When using uv, the correct Python version is matched automatically.

What if QR code login fails?

In CDP mode, ensure Chrome >= 144 with remote debugging enabled. If it keeps asking for re-login, close Chrome and re-enable debugging. Try setting HEADLESS = False to manually pass sliding verification.

Can I crawl multiple platforms simultaneously?

You can, but serial execution is recommended. Specify one --platform per run and switch after completion. Concurrent multi-platform crawling easily triggers rate limits.

Where is the data stored?

By default in the data/ folder under the project root, with subdirectories per platform. Customize the path via SAVE_DATA_PATH.

Can I use MediaCrawler commercially?

No. The project uses a NON-COMMERCIAL LEARNING LICENSE 1.1, explicitly prohibiting commercial use. Contact the author for commercial licensing.

12. Summary

MediaCrawler is the most complete open-source tool for Chinese social media data collection, backed by 64,000+ GitHub stars. Its core advantages:

  • Broad coverage: 7 platforms with a unified interface — zero-cost switching
  • Strong anti-detection: CDP mode reuses real browser environments
  • Easy setup: One-command install with uv, config and go
  • Flexible storage: 7 data formats for different scenarios
  • Active community: Continuous updates and responsive issue handling

For data analysts, public opinion researchers, and content operators, MediaCrawler is the best starting point for learning social media data collection. Just be sure to use it legally and ethically — respect platform rules and user privacy.