LittleCrawler crawls three Chinese platforms with Playwright

Three platforms, one crawler interface

LittleCrawler is described in the repository as a multi platform social media crawler framework built on Python async programming, and the description in the packaging metadata is narrower, naming Xiaohongshu with the code xhs and Zhihu with zhihu. The README's platform table is the most complete statement, listing three: Xiaohongshu with the code xhs covering notes, details and authors, the second-hand marketplace platform Xianyu with the code xhy covering goods, notes and authors, and Zhihu with the code zhihu covering articles, details and authors. The shape is a platform abstraction with a shared configuration, a shared storage layer and a shared web backend, and the per platform code lives in separate directories under src/platforms/ and src/storage/. One detail to check before you rely on the codes. The README table gives xhy for the marketplace platform, while the project description in the README header and the packaging metadata refer to the same platform with the code xy. The crawler directories on disk use xhy. That is a small thing, but the platform code is a command line argument, so a mismatch there is the kind of error that shows up as an empty result rather than a stack trace.

Playwright and a browser rather than an HTTP client

The dependency list explains the architecture. Playwright 1.45 is pinned, along with opencv-python, pillow and matplotlib, and the configuration exposes a CDP mode toggle, ENABLE_CDP_MODE, set to True with a comment recommending it because the anti detection is stronger in that mode. The crawler therefore drives a real browser rather than replaying signed HTTP requests, which is the pragmatic choice against platforms that sign and fingerprint their API traffic, and it is also the expensive one, because every page costs a browser context. The other side of that coin is in the configuration as well. There is an ENABLE_IP_PROXY flag for proxy use, and the repository is sponsored by a residential proxy provider, with the services/ directory containing a proxy management module alongside the cache module that supports local and Redis backed caching. The project also depends on jieba, a Chinese word segmenter, plus wordcloud, pandas and a stack of database drivers, which tells you the intended output is analysis ready text and tabular data rather than raw HTML.

Storage is a first class decision, not an afterthought

The configuration carries a SAVE_DATA_OPTION setting with the comment listing csv, json, db, sqlite, mongodb and excel, and the README turns that into a table explaining what each is for. CSV is simple and general, JSON keeps the structure complete, SQLite is lightweight and local, the db value maps to MySQL and is described as for production environments, MongoDB is for flexible scaling, and Excel is for visual analysis. The architecture behind that is a storage layer with a base directory holding shared implementations for the database, Excel and MongoDB, and per platform subdirectories for the platform specific parts, so the same crawler code writes to any of the six targets. Migrations are handled with alembic, and both an async MySQL driver and aiosqlite are present alongside motor for MongoDB and sqlalchemy for the ORM layer, so the storage layer is written against SQLAlchemy rather than raw drivers. The init command takes the target explicitly, which means the schema is created by the tool rather than by hand.

Running it, from a clone to a dashboard

The install is a clone, a dependency sync and a browser download:

git clone https://github.com/pbeenig/LittleCrawler.git
cd LittleCrawler
uv sync
playwright install chromium

Note that the clone URL in the README uses the spelling pbeenig, while the repository itself is under pbeenigg, so the URL is worth checking against the repository before you copy it. The project asks for Python 3.11 or newer and recommends uv, with pip as the alternative, and the pyproject file configures uv to use the Tsinghua PyPI mirror as its default index. The entry point takes a platform and a crawler type, and a separate flag initialises the database:

python main.py --platform xhs --type search
python main.py --init-db sqlite

The crawler types are search, detail and creator, and the login type can be qrcode, phone or cookie. The web backend is a FastAPI application served by uvicorn, and the frontend is a separate Next.js build that has to be compiled into the api/ui directory before the combined service will serve it:

uv run uvicorn api.main:app --port 8080 --reload

An API_ONLY environment variable starts the API without the frontend pages, and the web directory has its own dev server for frontend work. The screenshots in the README show a login page, a start page and a home page with run status, so the backend is a dashboard rather than a bare API.

The layout separates core, platforms, storage and services

The project structure in the README is worth reading as an architecture description, because the layering tells you where to change things. The entry point is main.py, and the config/ directory holds the configuration. Under src/ there is core/ for the base classes, the command line handling and the context variables, platforms/ with one directory per platform, storage/ with the base implementations and the per platform ones, models/ holding Pydantic data models, services/ with the cache and proxy modules, and utils/. The api/ directory is the web API backend and web/ is the frontend interface, with tests/ and docs/ alongside them, data/ for output and libs/ for third party JavaScript dependencies. Two details stand out. The presence of Pydantic models means the scraped shapes are typed, so a change in what a platform returns surfaces as a validation error rather than as silently wrong data. And the context variables in core/ suggest the per run state, including things like the browser session and the proxy, is passed through rather than held in module globals, which is what makes concurrent runs in one process plausible.

Where this approach runs out of road

The limits are not subtle. First, everything goes through a browser, so throughput is bounded by browser contexts rather than by your network, and a large collection job will be slow and memory hungry in a way a plain HTTP client would not be. Second, the scrapers are tied to page structure. The project has no contract with the platforms it reads, and a markup change on their side turns a working selector into an empty result, so someone has to maintain that code. Third, the login handling, whether by QR code, phone or cookie, means the crawler is acting with an account, and the terms of those platforms govern what you may collect with it. The project states no such position, and that is a gap you have to close yourself. The dependency pins are also old, with pillow 9.5.0, requests 2.32.3, fastapi 0.110.2 and playwright 1.45.0 all fixed in the manifests, so security updates to the transitive stack depend on someone regenerating the lock. The last push was on 2026-08-09, the repository is not archived, and there are no GitHub releases, so the commit history is the only version marker.

Editorial conclusion

Adopt LittleCrawler when you need structured public content from a small set of platforms, when you are comfortable driving a real browser with Playwright, and when you will maintain the selectors yourself as the pages change. Do not adopt it as a high volume collection system, since the whole approach runs through a browser, and do not treat the login helpers as a bypass of platform access rules. Verify first that the platform code you use matches the one the code expects, since the documentation and the configuration comments disagree on the code for the Xianyu platform, that your storage target is reachable from where the crawler runs, and that the terms of the platforms you point it at permit the collection you intend.

Frequently asked questions

Official sources

1. Issues
2. License: MIT
3. pbeenigg/LittleCrawler on GitHub
4. Project website
5. README