What is MediaCrawler?

In the data-driven era, social media platforms contain vast amounts of user insights, content trends, and public opinion data. However, manually collecting this data is both inefficient and unsustainable. MediaCrawler is an open-source project designed to solve this pain point—it's a powerful multi-platform self-media data collection tool maintained by developer NanmiCoder, with tens of thousands of stars on GitHub, making it the most comprehensive Chinese social media crawler framework currently available.

MediaCrawler's core value proposition: one tool covering 7 major platforms, eliminating the need to develop separate crawlers for each platform. Built on Python's asynchronous architecture, it leverages Playwright browser automation to handle login states, then calls platform APIs through preserved Cookie environments to obtain signature parameters—completely eliminating the need to reverse-engineer complex encryption algorithms, significantly lowering the technical barrier.

Project repository: github.com/NanmiCoder/MediaCrawler

Use Cases

  • Public opinion analysis: Monitor brand reputation and user feedback across major platforms
  • Content trend research: Track common characteristics of trending topics and viral content
  • Competitor analysis: Collect competitors' posting frequency and engagement metrics
  • Academic research: Study social media propagation patterns and user behavior analysis
  • Data visualization: Generate word clouds and trend charts from collected data

Supported Platforms and Feature Matrix

MediaCrawler currently supports the following platforms, with highly consistent feature coverage across each:

Platform Code Keyword Search Specific Post Nested Comments Creator Profile Login Cache IP Proxy Word Cloud
Xiaohongshu (RED) xhs ✅ ✅ ✅ ✅ ✅ ✅ ✅
Douyin (TikTok China) dy ✅ ✅ ✅ ✅ ✅ ✅ ✅
Kuaishou ks ✅ ✅ ✅ ✅ ✅ ✅ ✅
Bilibili bili ✅ ✅ ✅ ✅ ✅ ✅ ✅
Weibo wb ✅ ✅ ✅ ✅ ✅ ✅ ✅
Baidu Tieba tieba ✅ ✅ ✅ ✅ ✅ ✅ ✅
Zhihu zhihu ✅ ✅ ✅ ✅ ✅ ✅ ✅

Three crawling modes:

  • search: Search by keywords and crawl related posts/videos
  • detail: Specify post ID or URL for precise crawling of details and comments
  • creator: Crawl all content from a specified creator's profile

Three login methods:

  • qrcode: QR code scan login (recommended, most secure)
  • phone: Phone number + verification code login
  • cookie: Directly input existing Cookie

Technical Architecture: The Elegant Python + Playwright Design

MediaCrawler's technical choices reflect a pragmatic philosophy. Instead of traditional HTTP requests + signature reverse-engineering, it uses Playwright browser automation to handle login states, then calls platform APIs through preserved Cookie environments. The core advantage of this design: no need to reverse-engineer complex encryption algorithms.

Core Architecture Layers

MediaCrawler
├── main.py              # Program entry, crawler factory
├── config/              # Configuration management (platform configs, base configs)
├── media_platform/      # 7 platform crawler implementations
│   ├── xhs/            # Xiaohongshu
│   ├── douyin/         # Douyin
│   ├── kuaishou/       # Kuaishou
│   ├── bilibili/       # Bilibili
│   ├── weibo/          # Weibo
│   ├── tieba/          # Baidu Tieba
│   └── zhihu/          # Zhihu
├── store/              # Data storage layer (CSV/JSON/SQLite/MySQL/MongoDB)
├── proxy/              # IP proxy pool management
└── cache/              # Login state cache management

Workflow

  1. Login phase: Playwright opens browser, user scans QR code/enters phone number, obtains Cookie
  2. Cache phase: Cookie saved locally (cache/{platform}_cookies.json)
  3. Crawling phase: Reuses Cookie environment, obtains signature parameters via JS expressions, calls platform APIs
  4. Storage phase: Writes collected data to CSV/JSON/database

CDP Mode: The Anti-Detection Killer Feature

MediaCrawler enables CDP (Chrome DevTools Protocol) mode by default, connecting to the user's existing Chrome browser. Compared to Playwright's headless browser, CDP mode offers:

  • Real browser fingerprint: Uses the user's actually installed Chrome, including all extensions, plugins, and settings
  • Lower detection risk: Browser behavior patterns closer to real users
  • Inherited login state: Automatically reuses existing Cookies and browsing history

Enable via (config/base_config.py):

PYTHON 
ENABLE_CDP_MODE = True           # Enable CDP mode
CDP_CONNECT_EXISTING = True      # Connect to existing browser (recommended)
CDP_DEBUG_PORT = 9222            # Debug port

Before use, enter chrome://inspect/#remote-debugging in Chrome's address bar and check "Allow remote debugging for this browser instance".

Installation and Quick Start

Environment Requirements

  • Python 3.11+
  • Node.js 16+ (required for Douyin and Zhihu)
  • Chrome browser (latest version recommended)

uv is currently the fastest Python package manager with accurate dependency resolution, highly recommended.

BASH 
# 1. Clone project
git clone https://github.com/NanmiCoder/MediaCrawler.git
cd MediaCrawler

# 2. Install dependencies (automatically creates virtual environment)
uv sync

# 3. Run crawler (Xiaohongshu example)
uv run main.py --platform xhs --lt qrcode --type search

Method 2: Traditional venv Installation

BASH 
# 1. Create virtual environment
python -m venv venv
source venv/bin/activate  # Linux/macOS
# venv\Scripts\activate   # Windows

# 2. Install dependencies
pip install -r requirements.txt

# 3. Install Playwright browser drivers (needed for non-CDP mode)
playwright install

# 4. Run crawler
python main.py --platform xhs --lt qrcode --type search

Configuration File Overview

Core configuration in config/base_config.py, common parameters:

PYTHON 
# Platform selection: xhs | dy | ks | bili | wb | tieba | zhihu
PLATFORM = "xhs"

# Keyword search (comma-separated)
KEYWORDS = "AI programming,Python tutorial"

# Login method: qrcode | phone | cookie
LOGIN_TYPE = "qrcode"

# Crawl mode: search | detail | creator
CRAWLER_TYPE = "search"

# Enable IP proxy
ENABLE_IP_PROXY = False

# Enable comment crawling
ENABLE_GET_COMMENTS = True

# Enable nested comments
ENABLE_GET_SUB_COMMENTS = False

# Data storage format: csv | json | jsonl | sqlite | mysql | mongodb | excel
SAVE_DATA_OPTION = "jsonl"

# Max crawl count per platform
CRAWLER_MAX_NOTES_COUNT = 15

# Concurrency
MAX_CONCURRENCY_NUM = 1

# Crawl interval (seconds)
CRAWLER_MAX_SLEEP_SEC = 2

Platform-specific configurations are in dedicated files like config/xhs_config.py, config/dy_config.py, etc.

Platform-Specific Crawling Guide

MediaCrawler's command-line interface is unified and concise, with only three core parameters: --platform (platform), --lt (login method), --type (crawl mode). Below, we use Xiaohongshu as an example to demonstrate the complete crawling workflow.

Xiaohongshu Crawling

1. Keyword Search Mode

BASH 
# Modify config/base_config.py
PLATFORM = "xhs"
KEYWORDS = "AI art,Stable Diffusion"
CRAWLER_TYPE = "search"
ENABLE_GET_COMMENTS = True  # Enable comment crawling

# Run
uv run main.py --platform xhs --lt qrcode --type search

First run opens a browser for Xiaohongshu QR code login. After successful login, Cookie is automatically cached—no need to rescan on subsequent runs.

2. Specific Post Detail Mode

Configure target post URLs in config/xhs_config.py:

PYTHON 
XHS_SPECIFIED_NOTE_URL_LIST = [
    "https://www.xiaohongshu.com/explore/xxxxx?xsec_token=xxx",
    # Can add multiple
]

Then run:

BASH 
uv run main.py --platform xhs --lt qrcode --type detail

3. Creator Profile Mode

Configure creator profile URLs in config/xhs_config.py:

PYTHON 
XHS_CREATOR_ID_LIST = [
    "https://www.xiaohongshu.com/user/profile/xxxxx?xsec_token=xxx",
]

Run:

BASH 
uv run main.py --platform xhs --lt qrcode --type creator

Douyin Crawling

Douyin requires Node.js environment (for JS signatures). Configure config/dy_config.py:

PYTHON 
# Keyword search
KEYWORDS = "Python tutorial,programming basics"

# Specific video URLs
DOUYIN_SPECIFIED_NOTE_URL_LIST = [
    "https://www.douyin.com/video/xxxxx",
]

Run command:

BASH 
uv run main.py --platform dy --lt qrcode --type search

Douyin's login process is similar to Xiaohongshu—scan QR code, then Cookie is cached locally.

Bilibili Crawling

Bilibili represents typical video platform crawling. Configure config/bilibili_config.py:

PYTHON 
# Search keywords
KEYWORDS = "open source projects,GitHub"

# Specific video BV numbers
BILIBILI_SPECIFIED_NOTE_URL_LIST = [
    "https://www.bilibili.com/video/BVxxxxxx",
]

# Creator homepage
BILIBILI_CREATOR_ID_LIST = [
    "https://space.bilibili.com/xxxxx",
]

Run:

BASH 
uv run main.py --platform bili --lt qrcode --type search

Bilibili's data structure is relatively standardized, resulting in higher success rates.

Weibo, Kuaishou, Tieba, Zhihu

Other platforms follow similar configuration patterns—just modify the corresponding config file under config/, then run:

BASH 
# Weibo
uv run main.py --platform wb --lt qrcode --type search

# Kuaishou
uv run main.py --platform ks --lt qrcode --type search

# Baidu Tieba
uv run main.py --platform tieba --lt qrcode --type search

# Zhihu
uv run main.py --platform zhihu --lt qrcode --type search

Zhihu and Kuaishou also require Node.js environment.

WebUI Visual Interface

If you prefer not to use command line, MediaCrawler provides a WebUI:

BASH 
# Terminal 1: Start backend
uv run uvicorn api.main:app --port 8080 --reload

# Terminal 2: Start frontend
cd webui
npm install
npm run dev

Visit http://localhost:5173/ to configure parameters, view logs, and preview data in your browser.

Data Storage and Export

MediaCrawler supports 7 data storage formats, configured via SAVE_DATA_OPTION:

CSV / JSON / JSONL

Simplest file storage methods, data saved in data/{platform}/ directory.

PYTHON 
SAVE_DATA_OPTION = "csv"   # or "json" / "jsonl"

CSV is suitable for Excel viewing, JSON/JSONL for programmatic processing.

SQLite

Lightweight database requiring no additional installation, suitable for standalone use.

PYTHON 
SAVE_DATA_OPTION = "sqlite"

Automatically creates database and table structure on first run, supports deduplication.

MySQL / PostgreSQL

Recommended for production environments, supporting large-scale data storage and queries.

PYTHON 
SAVE_DATA_OPTION = "mysql"  # or "postgres"

Configure database connection in .env file:

BASH 
MYSQL_DB_HOST=localhost
MYSQL_DB_PORT=3306
MYSQL_DB_USER=root
MYSQL_DB_PWD=your_password
MYSQL_DB_NAME=media_crawler

MongoDB

Document database, suitable for storing data with non-fixed structures.

PYTHON 
SAVE_DATA_OPTION = "mongodb"

Configure .env:

BASH 
MONGODB_HOST=localhost
MONGODB_PORT=27017
MONGODB_DB_NAME=media_crawler

Excel

Directly outputs .xlsx files, convenient for non-technical users to view.

PYTHON 
SAVE_DATA_OPTION = "excel"

Data Field Descriptions

Taking Xiaohongshu as an example, collected data includes:

  • Note information: Title, content, likes, favorites, comments, publish time, tags
  • Comment information: Comment content, user nickname, likes, replies, timestamp
  • Creator information: Nickname, follower count, likes received, work count

All fields stored in JSON format for easy subsequent analysis.

Advanced Usage

Scheduled Crawling and Incremental Updates

MediaCrawler doesn't include built-in scheduling, but can be combined with cron or APScheduler:

BASH 
# Linux crontab example: crawl daily at 10 AM
0 10 * * * cd /path/to/MediaCrawler && uv run main.py --platform xhs --lt cookie --type search

The key to incremental updates is leveraging database deduplication. When using SQLite or MySQL storage, MediaCrawler automatically skips existing records (based on note ID). Simply run the same configuration periodically to collect only new content.

IP Proxy Pool Configuration

For large-scale crawling, enabling IP proxies is recommended to avoid triggering platform risk controls:

PYTHON 
# config/base_config.py
ENABLE_IP_PROXY = True
IP_PROXY_POOL_COUNT = 2  # Proxy pool count
IP_PROXY_PROVIDER_NAME = "kuaidaili"  # Supports kuaidaili | wandouhttp | static

Configure proxy service credentials in .env file:

BASH 
# Kuaidaili configuration
KDL_SECERT_ID=your_secret_id
KDL_SIGNATURE=your_signature
KDL_USER_NAME=your_username
KDL_USER_PWD=your_password

Static proxies can also be used:

PYTHON 
IP_PROXY_PROVIDER_NAME = "static"
STATIC_PROXY_URL = "http://user:password@proxy.example.com:8080"

Concurrency Control and Request Intervals

PYTHON 
# Concurrency (recommend 1-3, too high triggers risk controls)
MAX_CONCURRENCY_NUM = 1

# Request interval (seconds, random 1-2 seconds)
CRAWLER_MAX_SLEEP_SEC = 2

# Max crawl count per platform
CRAWLER_MAX_NOTES_COUNT = 15

Start with small-scale testing, gradually increase crawl volume. Excessive concurrency and frequency will result in account restrictions or IP bans.

Comment Word Cloud Generation

MediaCrawler includes built-in word cloud generation based on collected comment data:

PYTHON 
# config/base_config.py
ENABLE_GET_WORDCLOUD = True

# Custom phrases (optional)
CUSTOM_WORDS = {
    "AI art": "technology",
    "Stable Diffusion": "tool",
}

# Stop words file
STOP_WORDS_FILE = "./docs/hit_stopwords.txt"

After running, word cloud images (PNG format) are generated in the data/ directory, providing visual insight into comment hotspots.

Important notice: Crawler technology usage must comply with laws, regulations, and platform terms of service.

According to China's Cybersecurity Law, Data Security Law, Personal Information Protection Law, and the Collection of Crawler-Related Legal Cases, the following behaviors may violate the law:

  • Unauthorized collection of personal information: User privacy data (phone numbers, ID cards, addresses, etc.)
  • Bypassing technical protection measures: Cracking CAPTCHAs, spoofing identities to bypass risk controls
  • Large-scale collection for commercialization: Using collected data for commercial purposes
  • Disrupting normal platform operations: High-frequency requests causing excessive server load

Compliance Recommendations

  1. Collect only public data: Don't collect content requiring special permissions
  2. Control collection frequency: Set reasonable request intervals to avoid burdening platforms
  3. No commercial use: MediaCrawler uses NON-COMMERCIAL LEARNING LICENSE, prohibiting commercial use
  4. Respect robots.txt: Honor platform crawler protocols
  5. Protect user privacy: Don't publicly disseminate collected personal information

Disclaimer

The MediaCrawler project is for learning and research purposes only. Users assume all legal responsibilities arising from tool usage. The author is not responsible for any illegal activities.

Comparison with Other Tools

Tool Supported Platforms Tech Stack Anti-Detection Learning Curve Use Case
MediaCrawler 7 major Chinese platforms Python + Playwright CDP mode, strong Medium Chinese social media data collection
Snscrape Twitter/Instagram etc. Python Weak, easily blocked Low Lightweight overseas platform collection
Tweepy Twitter only Python Depends on official API Low Twitter official API development
Scrapy General purpose Python + Twisted Must implement yourself High Large-scale custom crawlers
Selenium General purpose Python + WebDriver Medium Medium Interactive web automation
Playwright General purpose Python/Node Strong Medium Modern web automation

MediaCrawler's Advantages

  • All-in-one coverage: One tool supporting 7 major Chinese platforms, no need to develop separately for each
  • No reverse engineering: Based on Playwright + CDP mode, bypasses complex signature algorithms
  • Ready to use: Clear configuration files, concise command-line parameters
  • Active community: GitHub stars continuously growing, timely issue responses

When to Choose Other Tools

  • Snscrape/Tweepy: Need to collect from overseas platforms like Twitter/Instagram
  • Scrapy: Need to build large-scale, high-concurrency custom crawlers
  • Selenium/Playwright: Need general web automation capabilities rather than specialized social media collection

Frequently Asked Questions

MediaCrawler itself is a legal learning tool, but usage determines legality. Collecting only public data, controlling frequency, avoiding commercial use, and complying with platform terms of service generally won't cause legal issues. However, collecting personal information, bypassing technical protections, or large-scale commercial use may violate the Data Security Law and Personal Information Protection Law. Refer to the Collection of Crawler-Related Legal Cases to understand legal boundaries.

2. Why does QR code login fail or frequently require re-scanning?

Common causes: - Cookie expiration: Platforms periodically clear login states, just re-scan - Risk control triggered: Frequent collection or abnormal behavior leads to account restrictions, reduce collection frequency - Network issues: Unstable proxy IP or platform-flagged, try switching IP or disabling proxy - Outdated browser version: CDP mode requires Chrome 144+, update your browser

3. How to improve collection success rate?

  • Enable CDP mode: Use real browser environment to reduce detection risk
  • Control collection frequency: Set CRAWLER_MAX_SLEEP_SEC = 2-5 to avoid high-frequency requests
  • Use IP proxies: Enable proxy pools for large-scale collection to distribute request sources
  • Single account, single IP: Avoid multiple accounts sharing one IP, easily triggers correlation risk controls
  • Off-peak collection: Avoid platform peak hours (e.g., 8-10 PM)

4. Can collected data be used for commercial analysis?

No. MediaCrawler uses NON-COMMERCIAL LEARNING LICENSE 1.1, explicitly prohibiting commercial use. For commercial needs, consider: - Contact platform official to purchase data services - Use platform-provided official APIs (e.g., Weibo Open Platform, Bilibili Open Platform) - Consult legal experts to ensure data collection and usage comply with regulations

5. Does it support Windows/Mac/Linux?

Fully supported. MediaCrawler is Python-based, cross-platform compatible. Windows users note: - Path separator uses backslash \ - Virtual environment activation: venv\Scripts\activate - Chrome path example: C:\Program Files\Google\Chrome\Application\chrome.exe

Mac/Linux users: - Virtual environment activation: source venv/bin/activate - Chrome path example: /Applications/Google Chrome.app/Contents/MacOS/Google Chrome

Summary

MediaCrawler is currently the most comprehensive Chinese social media crawler framework, with one tool covering Xiaohongshu, Douyin, Kuaishou, Bilibili, Weibo, Tieba, and Zhihu—7 major platforms. Its core value lies in:

  • Lowering technical barriers: Based on Playwright + CDP mode, no need to reverse-engineer complex encryption algorithms
  • All-in-one collection: Unified command-line interface, simple configuration, ready to use
  • Flexible storage: Supports 7 formats including CSV/JSON/SQLite/MySQL/MongoDB/Excel
  • Anti-detection capabilities: CDP mode + IP proxy pool, reduces ban risk

However, clearly recognize:

  • Legal risks: Must comply with laws and regulations, use only for learning and research
  • Platform risk controls: Excessive collection frequency leads to restrictions or bans
  • Non-commercial use: LICENSE explicitly prohibits commercial use

If you're a data analyst, researcher, or developer needing to collect Chinese social media data for learning, research, or personal projects, MediaCrawler is the most worthwhile tool to try. But always remember: use compliantly, respect platform rules, protect user privacy.

Project repository: github.com/NanmiCoder/MediaCrawler

If you find this article helpful, please give the project a Star to support the author!

Related reading: