MediaCrawler: A Playwright Session-Reuse Scraper for Seven Chinese Social Platforms

The problem MediaCrawler solves: signed requests without reverse engineering

Chinese social platforms sign their API requests with parameters generated by obfuscated JavaScript. A conventional scraper has to extract that algorithm, reimplement it in Python, and re-verify it every time the platform rotates its code. MediaCrawler takes the other route. According to the README, it uses Playwright to log in and persist the login state, then evaluates JS expressions inside that already-authenticated browser context to obtain the signature parameters. The README states the goal directly: no need to reverse the complex encryption algorithms, which lowers the technical bar. The audience is therefore engineers and researchers who want the data, not the deobfuscation work. The project covers Xiaohongshu, Douyin, Kuaishou, Bilibili, Weibo, Tieba and Zhihu, and the feature table lists keyword search, crawl by post ID, secondary comments, crawl by creator homepage, login-state caching, IP proxy pool and comment word-cloud generation as available across all seven. That table is the most useful page in the repository, and it is worth reading before anything else, because it is the only place the project states what each platform supports.

How the mechanism works: a real browser as the signing oracle

The architecture is a browser-automation harness rather than a request library. Playwright opens or attaches to a browser, the user logs in, and the resulting session is cached so subsequent runs reuse it. Signature parameters are not computed in Python; the program asks the page to compute them, which is why the README describes the approach as avoiding JS reverse engineering. The default transport is CDP mode, which connects to a Chrome instance the user already runs, reusing that browser's existing login state, cookies and extensions. The README argues this reduces platform risk-control detection, which is plausible: the traffic originates from a normal browser profile rather than a fresh automated one. The alternative is standard Playwright mode, toggled by setting ENABLE_CDP_MODE = False in config/base_config.py. Two consequences follow from this design. First, the crawler's throughput is bounded by browser interaction, not by HTTP concurrency, so this is not a tool for high-volume harvesting. Second, the correctness of every request depends on the platform's own JavaScript still being loadable in that page context, which means the failure mode is a broken page, not a wrong signature you can debug in isolation.

Getting it running: uv, Chrome remote debugging, and the CLI flags

The README recommends uv for dependency management. After installing uv and confirming with uv --version, you change into the project directory and run uv sync to pin the Python version and dependencies. Node.js 16.0.0 or higher is also required, and the README notes that Douyin and Zhihu specifically need it. Browser drivers are optional: under the default CDP mode you do not run playwright install, and only the standard Playwright mode needs uv run playwright install. For CDP mode you need Chrome version 144 or higher, then you open chrome://inspect/#remote-debugging and tick Allow remote debugging for this browser instance. The page shows Server running at: 127.0.0.1:9222 when ready. Runs are launched through main.py. The README gives two examples: uv run main.py --platform xhs --lt qrcode --type search for keyword search, and uv run main.py --platform xhs --lt qrcode --type detail for a list of post IDs read from the config. Login is completed by scanning a QR code in the corresponding app. The README also notes that a Chrome confirmation dialog appears after the crawler starts, that the program waits for the user, and that you have 60 seconds to accept. Comment collection is off by default; you turn it on by editing ENABLE_GET_COMMENTS in config/base_config.py, which the README says carries Chinese comments for the other options. A WebUI exists: run uv run uvicorn api.main:app --port 8080 --reload alongside cd webui && npm install && npm run dev, which serves on port 5173 and proxies /api to 8080. For a production build, npm run build outputs into api/webui/ so the API server alone serves the interface on 8080.

The 60-second dialog and other places this breaks

The confirmation dialog is the clearest limitation in the documentation. The README states that the program waits for the user and that the operation must be completed within 60 seconds. That single sentence rules out cron jobs, containerised pipelines and any unattended collection, at least in CDP mode, unless you accept the standard Playwright path with its own trade-offs. The second constraint is the login model. Login state is cached, but the README's examples all begin with a QR-code scan, so a human is in the loop at least at session start. Third, the project is explicitly scoped to public information and the disclaimer forbids commercial use and large-scale crawling, which is a stated boundary rather than a technical one but still shapes what the tool is for. Fourth, the licence metadata is unresolved: the repository reports NOASSERTION, and the README points to a LICENSE file plus a longer disclaimer section without stating terms in the text provided. Anyone planning to build on this needs to read the LICENSE file and the disclaimer before writing code, not after. Finally, the README's own framing is that this is a learning project, and it advertises a paid Pro version whose listed advantages include removing the Playwright dependency, adding resumable crawling, multi-account support and full Linux support. That list is effectively a description of the open version's gaps.

What the Pro version's feature list tells you about the open one

The README's Pro section is informative even if you never intend to pay. It lists resumable crawling and multi-account plus IP proxy pool support as headline additions, which implies the open version handles neither interruption recovery nor account rotation as a first-class feature. It also lists removal of the Playwright dependency and full Linux environment support as advantages, which suggests the open version's browser-driven design is the source of both its portability friction and its operational overhead. The README does not state that the open version fails on Linux, only that Pro adds complete Linux support, so treat that as an open question to test rather than a confirmed defect. The same section mentions an AI Agent Skill installable through OpenClaw, Claude Code or Cursor, and a comment-analysis agent marked as still in development. None of these are verifiable from the README; they are claims about a separate repository.

When a plain HTTP client is the better answer

If your target is a platform with a stable, documented or already-solved API, MediaCrawler is the wrong shape of tool. A requests or httpx script that signs its own requests has no browser to launch, no QR code, no 60-second dialog and no dependency on Node.js. It runs in a container, it parallelises, and it fails with a status code you can log. The difference in approach is not incremental: MediaCrawler trades throughput and unattended operation for the ability to skip deobfuscation entirely. That trade is correct when the signing algorithm changes often, when you need a handful of platforms rather than one, and when a human can be present at the start of each run. It is wrong when you need the same endpoint polled every fifteen minutes, when you need to run inside a CI job, or when the volume you need exceeds what a single browser session can reasonably produce. The README's own disclaimer, which forbids large-scale crawling, points in the same direction: this is a tool for bounded, supervised collection.

Maintenance cost and what the licence question actually requires

Two maintenance surfaces exist and they behave differently. The Python side is conventional: uv sync against the lockfile, and a requirements.txt path for anyone using venv, which the README marks as not recommended. The browser side is the volatile one. CDP mode depends on Chrome 144 or higher and on a remote-debugging flag that the user sets manually, and the README's instruction to reuse an existing profile means the crawler inherits whatever that profile contains. When a platform changes its page structure or its signing expression, the fix lands in the JS evaluation layer rather than in a signature implementation, which is easier to patch but harder to unit test. On licensing, the repository reports NOASSERTION and the README carries a disclaimer restricting use to learning and reference, prohibiting commercial use, and disclaiming liability. That is not a standard open source grant, and the README does not resolve what the LICENSE file actually says. Verify the licence text and the full disclaimer against your intended use before depending on this in anything that ships. The README also links to a repository collecting crawler-related legal cases in China, which is a signal about the project's own risk framing rather than a legal opinion.

Editorial conclusion

Adopt MediaCrawler if you need public post and comment data from Xiaohongshu, Douyin, Kuaishou, Bilibili, Weibo, Tieba or Zhihu and you are willing to run a real Chrome window with remote debugging enabled and scan a QR code per session. Do not adopt it if you need unattended, headless, scheduled collection, or if you cannot accept a licence file whose terms the repository metadata does not resolve. Before writing any code, verify two things: that ENABLE_CDP_MODE in config/base_config.py matches the browser you actually have, and that the platform you need is listed in the feature table with the crawl type you need, because the table is the only capability statement the README provides.

Official sources

1. Issues
2. NanmiCoder/MediaCrawler on GitHub
3. Project website
4. README