ShilongLee/Crawler API Server for Chinese Social Platforms

ShilongLee/Crawler's Seven-Platform API Scope

ShilongLee/Crawler is intended for developers who need programmatic access to publicly available information and for security engineers studying how collection clients interact with Chinese media and commerce platforms. The repository groups crawlers for Douyin, Kuaishou, Bilibili, Xiaohongshu, Taobao, JD, and Weibo behind one server rather than requiring each caller to integrate seven separate clients. The README says these crawlers can retrieve public user, post, and video information, with support varying by platform.

The feature matrix is more important than the project description. Douyin, Kuaishou, Bilibili, Xiaohongshu, and Weibo list keyword search, post or video details, first- and second-level comments, media downloads, and user or work information. Taobao supports keyword search, details, first-level comments, and downloads, but does not list second-level comments or user and work information. JD is narrower: it lists keyword search and downloads, while details, comments, user information, and works are marked unavailable. A caller should select the project by checking the exact platform operation, rather than assuming that all seven implementations have equal coverage.

The project is aimed at public data, not account recovery, private messaging, or authenticated personal records. That distinction is explicit in the README, which rejects privacy violations and states that users remain responsible for legal compliance. The documentation does not describe pagination limits, rate limits, retention policies, or a service-level agreement, so consumers cannot derive dependable collection volume from the feature table alone.

FastAPI, uvicorn, and the Crawler Request Path

The repository includes Python service code alongside a JavaScript runtime. Its dependency list includes FastAPI, Starlette, Uvicorn, HTTPX, Requests, Beautiful Soup, aiosqlite, PyExecJS, and the `unicorn` package. The Dockerfile also installs Node.js and npm. This combination is consistent with a web API service that exposes crawler modules implemented across Python and JavaScript execution, but the README does not document the internal module boundaries or every data transformation.

The visible startup architecture is straightforward. `main:app` is the ASGI application, and Uvicorn serves it on port 8080. Configuration is loaded from `config/docker-config.yaml` when the Docker image starts because the Dockerfile sets `FILE` to that path. Platform services, utilities, scripts, data, configuration, and tests are separate top-level directories, which keeps the HTTP entry point distinct from the collection implementation and persistence-related code.

The README says the crawler framework supports concurrent requests and refers to an IP proxy pool with a rotation algorithm. It does not provide enough implementation detail in the available documentation to state when a proxy is selected, how cookies are scoped, or whether failed requests are automatically retried. Account registration is also server-side: a new installation must call a platform-specific `add_account` endpoint before normal collection calls. That makes initial setup more involved than deploying a stateless public endpoint.

Docker Startup on Port 8080

Docker is the shortest documented path to a running service. The published image starts Uvicorn, binds the application to `0.0.0.0`, and exposes port 8080. The first command creates a detached container named `crawler` and maps the host's port 8080 to the same container port.

Run the documented image as follows:

sudo docker run -d --name crawler -p 8080:8080 shilongli0101/crawler:latest

A successful launch should create a running container named `crawler`. The repository also documents an Alibaba Cloud mirror, but that second image is a separate regional distribution option rather than a second version:

sudo docker run -d --name crawler -p 8080:8080 registry.cn-hangzhou.aliyuncs.com/shilongli0101/crawler:latest

Inspect container output with the documented command:

sudo docker logs crawler

The logs should show output from the Uvicorn service if startup reaches the documented entry point. The README does not publish a health-check endpoint, so it does not define a ready-state response that a deployment system can probe. It also does not describe how to mount configuration or persist the `data` directory when running this particular `docker run` command. That omission matters because `aiosqlite` is included and the repository contains database files under `data`, but the available documentation does not establish the retention or upgrade procedure.

Makefile Installation and the First Account

For a local installation, the Makefile creates `.venv`, installs every pinned dependency from `requirements.txt`, and starts the same `main:app` application through Uvicorn. The README separately instructs Linux users to prepare Python and a JavaScript runtime. Its exact Node.js setup sequence is:

curl -fsSL https://deb.nodesource.com/setup_20.x | sudo bash -
sudo apt-get install -y nodejs

After cloning or entering the `Crawler` directory, run:

cd Crawler
make run

The `run` target creates the environment, installs requirements, and serves the API on port 8080 by default. The `port` variable can override that value. The README does not give an exact custom-port invocation, so the only confirmed local start command above is `make run`. Logs are written under `.log/`, with crawler output recorded in `crawler.log`.

The first operational step is account setup. Call `http://localhost:8080/{platfrom}/add_account`, replacing the documented path placeholder with one of `douyin`, `kuaishou`, `bilibili`, `xhs`, `taobao`, `jd`, or `weibo`. The available documentation does not state the HTTP method or request schema, so it would be unsafe to invent a request body. Obtain those details from `docs/doc.md`, add the required account data, then call the documented platform endpoints for public search, detail, comment, user, or media operations.

Testing Scope, Platform Gaps, and Failure Modes

The test workflow is platform-selectable, but its name is easy to misunderstand. `make test` does not merely execute pre-existing assertions against an already running server. Its Makefile target depends on `venv` and `install`, then runs `test/main.py`. Before testing, the README instructs the developer to place their own cookie in `test/cookie.py`. A cookie can expire or be rejected, making this a live account-dependent check rather than a reproducible unit test with fixed fixtures.

Run all documented tests with:

make test

Run only the Douyin module with the exact `module` parameter shown in the README:

make test module=douyin

Platform breakage is a credible failure mode. These crawlers depend on public platform endpoints and request-signing behavior, while the README specifically mentions an updated `a_bogus` implementation for Douyin. The documentation does not describe automatic compatibility recovery when a platform changes its request contract. A failed account initialization, rejected cookie, or changed upstream response is therefore an operational condition that callers must distinguish from a local server failure.

Coverage is another hard boundary. JD cannot provide several operations listed for other platforms, and Taobao lacks second-level comments and user or work information. Bilibili has a documented one-click video download, Xiaohongshu documents watermark-free note-video retrieval, and Weibo has a media preview interface. These are specific documented features, but they should not be generalized into identical functionality across the server.

Licence Boundaries and Upgrade Surface

The repository does not use a standard open source licence identifier recognized by GitHub. Its README describes the terms as a non-commercial use licence, and directs readers to the `LICENSE` file for details. That restriction has direct technical and procurement consequences: internal research may fit the stated purpose, while any commercial deployment needs a separate review of the actual licence text. This article does not interpret whether a particular business use is permitted.

The dependency file pins exact Python package versions, including `fastapi==0.111.0`, `uvicorn==0.30.1`, `httpx==0.27.0`, and `PyExecJS==1.5.1`. The Dockerfile uses the moving `python:3-slim` base image and installs Node.js and npm without version pins in the visible `apt-get` command. As a result, rebuilding the same application later may not produce an identical runtime. The `:latest` Docker image tag also does not identify a stable release through its name.

The repository is not archived, and the last push was on 2026-10-16. That fact shows recent repository activity, but it does not remove platform drift risk. The listed releases are v1.1.2 from 2024-10-02, v1.1.1 from 2024-07-24, and v1.1.0 from 2024-07-14. Before upgrading an existing deployment, verify the licence text, inspect changes between pinned images, preserve account and database state according to local policy, and confirm that the exact platform endpoints used by the application remain documented in `docs/doc.md`.

ShilongLee/Crawler Versus Platform-Specific Clients

The main alternative is to integrate a separate client or directly maintained request module for each platform. That approach keeps platform-specific authentication, signing, response parsing, and update work inside separate dependencies or services. It can also reduce the risk that one server process and its shared configuration expose a common failure point across several platforms. The trade-off is that callers must assemble, configure, and maintain multiple integrations themselves.

ShilongLee/Crawler takes the opposite approach. It places platform crawlers behind one FastAPI and HTTP interface, supports calls from multiple programming languages, and supplies Docker packaging and account-management endpoints. The documented IP proxy pool can rotate proxies, while the asynchronous framework supports concurrent requests. These are meaningful differences for a developer who wants one service boundary rather than seven client integrations.

This consolidation also transfers more responsibility to the server project. Platform features are uneven, as shown by JD's missing detail, comment, and user operations. Upstream changes can invalidate shared collection logic, and custom licence terms constrain commercial use. A direct platform-specific client is the better fit when one platform is all that is needed, or when complete control over request construction and update ownership matters more than a common API.

Editorial conclusion

Developers and security engineers who need a single HTTP interface for public information from Douyin, Kuaishou, Bilibili, Xiaohongshu, Taobao, JD, or Weibo should first inspect the licence, platform-specific API documentation, and account requirements. Organizations seeking commercial use or complete JD detail, comment, and user-data support should not adopt the current project without resolving those boundaries first.

Frequently asked questions

Official sources

1. Issues
2. README
3. Releases
4. ShilongLee/Crawler on GitHub