dataabc/weibo-crawler: pulling Weibo timelines, media and comments into files

Configuring a run with config.json instead of command line flags

The whole tool is driven by one JSON file. The repository ships `config.json.example`, and the README's worked example is a single user with media downloads switched partly on:

"user_id_list": ["1669879400"],
"only_crawl_original": 1,
"since_date": "1900-01-01",
"query_list": [],
"write_mode": ["csv"],
"original_pic_download": 1,
"retweet_pic_download": 0,
"original_video_download": 1,
"retweet_video_download": 0,
"cookie": "your cookie"

Each of those keys has a specific meaning. `user_id_list` accepts one id, several ids, or a path to a file containing ids. `only_crawl_original` set to 1 collects only original posts, and 0 collects originals plus reposts. `since_date` bounds the crawl by publication date, which is what makes incremental runs cheap later. `query_list` restricts the crawl to posts containing certain keywords, and an empty list means everything. The four download flags are independent, so you can pull original images without pulling reposted ones.

The `cookie` key is optional, and the README spends a section on the difference. Without a cookie the crawler sees what an anonymous visitor sees; with one it sees what a logged-in session sees, which means more posts and, in some cases, more detail. The README also documents a separate procedure for checking whether a cookie is still valid, which is a practical concern given how long these collections tend to run.

Running it locally and what lands on disk

The entry point for a manual run is a single command, with no arguments:

python weibo.py

Dependencies come from `requirements.txt`, which is small and readable: `lxml` and `Requests` for fetching and parsing, `tqdm` for the progress bar, `schedule` for the periodic run mode, `json5` for lenient config parsing, and `piexif` for writing EXIF data into downloaded images. Three database drivers are pinned alongside, `pymongo`, `PyMySQL` and a SQLite path, because the same script can write to several backends depending on `write_mode`.

The output layout is predictable. Everything lands under a `weibo` directory, and inside it there is one folder per user named after the user's nickname. A `user_id.csv` holds the rows, an `img` folder holds downloaded images, and a `video` folder holds downloaded video. That structure is what the Docker compose file mounts, so a containerised run writes to the same place a local run does.

The CSV columns are richer than the JSON key names suggest, and the distinction that trips people up is retweets. When you crawl all posts rather than only originals, an extra set of columns appears, prefixed to mark them as belonging to the reposted source post: source user id, source nickname, source post id, source text, source image urls, and the corresponding counts. Crawling originals only means those columns are absent entirely.

The Docker image already handles scheduling and timezones

The repository includes a Dockerfile and a compose file, and between them they solve the two problems that usually make scheduled scrapers annoying. The first is timezone. The image is based on `python:3.12.0-alpine` with tzdata installed and the clock pinned to Asia/Shanghai, which matters because Weibo timestamps are all in Beijing time and a UTC default would shift every post by a day near midnight.

FROM python:3.12.0-alpine

The second is the loop interval. The Dockerfile sets a default and passes it to the entry point:

ENV schedule_interval=1
CMD python __main__.py $schedule_interval

Note the two Python files. `weibo.py` is the manual entry point, while `__main__.py` takes an interval and is what the container runs. The compose file builds from the local Dockerfile, sets `restart: always`, and passes the interval through the environment as `schedule_interval`, with one meaning minute rather than seconds:

docker compose up -d

Three volumes are mounted: `config.json`, `user_id_list.txt` and the `weibo` output directory. The compose file references `user_id_list.txt` even though the config key accepts a file path, which gives you a way to add accounts without touching the config.

An HTTP service sits alongside the crawler

The tree includes `service.py` and an `API.md`, and `Flask` is a pinned dependency, so there is a web layer as well as a batch script. The README has a section for it near the end, which tells you it is the newer or less central part of the project rather than the main path.

This matters for how you would use the tool in production. The batch model is: configure accounts, run, collect files. The service model is: collect continuously in the background, query results over HTTP. If you are building a dashboard or a dataset pipeline, the service is the piece that avoids re-reading CSV files. If you just want an archive, `weibo.py` alone is enough and the Flask dependency is dead weight.

The remaining root files are ordinary and worth a glance. `const.py` holds shared constants, `util/` holds helper modules, `logging.conf` configures logging separately from the rest of the configuration, and `.vscode/` is committed editor setup. There is also a `test_llm.py` at the root, which suggests some experimentation with model-assisted parsing rather than a settled test suite. That file being present but `requirements.txt` having no model library in it is a fair hint about how far that went.

What the field list says about what Weibo exposes

The README enumerates the captured fields in detail, and that list is the best available description of the data model. Profile fields include user id, nickname, gender, birthday, location, education history, company, sunshine credit score, registration date, post count, follower count, following count, biography, mobile profile URL, avatar URL, high resolution avatar URL, user level, membership level, verification status, verification type and verification reason.

Post fields include post id, bid, text, headline article URL, original image URLs, video URL, posting location, publication date, like count, repost count, comment count, the client used to post, hashtags, and mentioned users. Several fields have a specific convention worth knowing before you write a parser over the output: multiple image URLs are comma separated, multiple video URLs are semicolon separated, and absent values are the empty string rather than null.

Beyond the core timeline, the README lists optional collection of first level comments under both original and reposted posts, and optional collection of reposts. That is a deliberate boundary. The crawler stops at one level of comments and does not walk the reply tree, which keeps the request count manageable but means you will not get a full conversation graph without writing that yourself.

How this compares to the official API and to the sibling project

The obvious question is why not use an official Weibo API. The practical answer is scope and access: official endpoints are rate limited, gated behind developer approvals, and inconsistent across the mobile and desktop surfaces this project reads. A crawler that works against the same endpoints a browser uses needs no application review, which is why tools like this exist at all. The cost is that it can break without notice, and that the boundaries described in the Weibo terms of service apply to it.

The repository also references a sibling project, weiboSpider, and it explains a relationship rather than a fork. The README notes that the bid field in this project is the same value as the post id in the cookie based version, which is the kind of detail that matters if you are merging data from both. Two projects from the same author, one cookie based and one not, is a reasonable fallback strategy: if one stops working the other often does not.

On maintenance, the project is not archived and the last push was 2026-07-22, so it is still being touched. The 404 open issues, against 4656 stars, are the honest signal about its state. Weibo changes its response shapes frequently, and each change generates bug reports. The dependency pinning is correspondingly current, with Requests, Flask and lxml all at recent versions, so someone is keeping up with the ecosystem even as the site shifts underneath.

Editorial conclusion

weibo-crawler is a good fit for archiving a specific set of Weibo accounts over time, because it is built around incremental re-runs and a config file rather than around a one-off scrape. Start by copying `config.json.example` to `config.json`, setting `user_id_list` and leaving the download flags off for the first pass, then run `python weibo.py` and read the CSV to confirm the field set matches what you need before enabling media downloads. The media pipeline is the expensive part, so turn on one flag at a time. For scheduled collection, the Docker image and the `schedule` dependency already handle interval runs, and `service.py` plus `API.md` add an HTTP surface if you need to query results from another service. What the repository does not document is any rate limiting or ban avoidance strategy, so treat the open issue count of 404 as a signal that site changes are the main ongoing maintenance cost here.

Frequently asked questions

Official sources

1. dataabc/weibo-crawler on GitHub
2. Issues
3. README