wechat_spider: Capturing WeChat Article Data Through AnyProxy

Why wechat_spider exists

wechat_spider addresses the difficulty of collecting a public account's older article links and the data attached to those pages when there is no documented API workflow in the repository. Its stated scope includes article content, reading counts, likes,在看 counts, comments and the historical article URLs of public accounts. The intended users are people working with WeChat content who need a local collection system rather than a one-off page save. The README describes both personal-computer and server deployment, with a React interface for checking whether captured records have been saved.

The project is not a general web crawler with a conventional URL queue. Its collection route depends on traffic from a phone using WeChat through an AnyProxy instance. The phone and computer must be connected to the same Wi-Fi, and the phone must use the computer's proxy address and port. This is a significant operational boundary. It is better suited to controlled data collection and repeatable investigation of pages reached through WeChat than to a public internet service that discovers accounts autonomously.

The repository description also says that the crawler can retrieve all historical article links for a public account. That claim depends on the documented history-page process and on the data returned by the WeChat pages being used. The README does not describe account discovery, login automation, a general search endpoint or a guarantee that every public account will expose the same fields.

AnyProxy traffic interception and data flow

The central mechanism is a man-in-the-middle proxy. The README says the project uses AnyProxy 4 and parses WeChat HTTPS requests, so collection is tied to the requests generated when a phone opens WeChat article and history pages. Installing the HTTPS certificate on both the computer and the phone is required for the local setup. The Docker image adds the root certificate to the Linux container, but the phone still needs the HTTPS certificate installed.

The repository layout places the entry point in `index.js`, proxy-related and collection logic under `rule/`, persistence code under `models/`, and an Express service under `server/`. The dependencies support the described flow: `anyproxy` provides the interception layer, `cheerio` supports HTML parsing, `mongoose` connects to MongoDB, and `request` and `request-promise` provide HTTP requests. The README identifies MongoDB collections named `posts`, `profiles` and `comments`, which gives a concrete view of what the application stores.

WeChat content is captured through page requests and subsequent navigation. The README distinguishes two navigation paths: a public account history page can jump to another history page, while a WeChat article page can jump to another article page. It explicitly says those two page types cannot navigate to each other. Automatic page turning is optional through `config.js`, and the project can also control intervals, publication-date range, article-body storage and comment storage. Redis is listed as a prerequisite, although the README does not explain its exact role in the data path.

Installing Node.js, MongoDB and Redis

The README recommends Node.js 16 and recent versions of MongoDB and Redis. The normal installation path is a Git clone followed by the package installation script. These commands come directly from the README. After installation, the reader still needs to configure the proxy certificate and make the phone reachable on the same Wi-Fi network.

The documented commands are:

git clone https://github.com/lqqyt2423/wechat_spider.git
cd wechat_spider
npm install

The README does not provide a separate command for starting MongoDB or Redis, so it would be inaccurate to add one. It also does not specify a complete production deployment guide beyond the Docker example. A first run uses `npm start`, which maps to `node index.js` in `package.json`. The expected next event is a command-line message similar to `请配置代理: xx.xx.xx.xx:8101`. The phone should then use that displayed IP and port as its proxy.

The `package.json` script set also includes `npm run dev` for nodemon and `npm run prod` for `NODE_ENV=production npm start`. The README does not explain how to initialize MongoDB collections or seed configuration values, so those steps should not be treated as automatic.

First capture with AnyProxy on a phone

The first real use requires a browser session, not just a server start. After `npm start`, configure the phone to use the computer's displayed proxy on port 8101. Open a public account history page and an article page in WeChat, then watch the command-line output and refresh the bundled React interface. The README says the default interface is available at `http://localhost:8104`.

The start command is:

cd wechat_spider
npm start

The interface can be opened at the documented address after the server starts:

http://localhost:8104

The expected verification is that data appears in MongoDB and that the visualization shows the captured records after a refresh. For automatic page turning, the README recommends opening several different public account history pages first. Once the database contains the basic history information, entering a history page can allow the configured pagination logic to proceed. That sequence matters because the project does not describe a seed command that populates all required state automatically.

For the newer WeChat interface, the README gives a history URL pattern using `biz` and `mp/profile_ext`. It also says to refresh an article page after entering it. The repository does not provide a command-line API for initiating a capture, so the documented interaction remains opening pages in the phone's WeChat client while the proxy is active.

Docker deployment and certificate boundaries

Docker is supported through the repository's `Dockerfile` and `docker-compose.yml`. The compose file defines an `app` service, Redis and MongoDB. It maps ports 8101, 8104 and 8102 to the host and passes `MONGO_PATH` as the MongoDB volume path. The README gives this deployment sequence:

git clone https://github.com/lqqyt2423/wechat_spider.git
cd wechat_spider
# build image
docker-compose build
# 运行实例（mongo数据存储地址需通过环境变量MONGO_PATH传入）
MONGO_PATH=/data/mongo docker-compose up
# 终止运行
docker-compose down

The Docker image uses `node:16`, installs production dependencies and copies the repository into `/app`. Its `EXPOSE` instruction lists the same three ports. The compose configuration also uses `redis:latest` and `mongo:latest`, so the documented setup does not pin those image versions.

The certificate boundary is important. The Dockerfile installs the bundled root certificate into the container's CA store, while the README says the phone still needs the HTTPS certificate installed. The phone's proxy IP must point to the computer's address, not the IP printed inside the Docker instance. That makes Docker convenient for repeatable services, but it does not remove the need for a reachable network and manual phone-side configuration. The README does not document authentication for the management interface or a production TLS termination design.

Configuration, storage and export limits

Configuration lives in `config.js`, with comments for the available settings. The README identifies controls for automatic article or history-page navigation, navigation intervals, publication-time filtering, article-body storage and comment storage. It also states that AnyProxy's connection port was changed to 8101 and its management interface to 8102. The management interface is enabled only when `NODE_ENV=development`.

MongoDB uses the database name `wechat_spider`. The documented collections are `posts` for article data, `profiles` for public account data and `comments` for comments. The README includes a direct export example for posts. This is useful when the collection job has already run and the next requirement is to move selected fields into CSV:

mongoexport --db wechat_spider --collection posts --type=csv --fields title,link,publishAt,readNum,likeNum,likeNum2,msgBiz,msgMid,msgIdx,sourceUrl,cover,digest,isFail --out ~/Desktop/posts.csv

The project includes `test/exportData.js` as another export reference, but the README does not explain its invocation. The visualization is packaged with the application, while the front-end source is under `client`. The repository has no GitHub releases, so versioned upgrade notes and migration procedures are not documented. The `package.json` version is 1.2.0, and the last repository push was on 2026-08-14.

When wechat_spider is the wrong tool

The largest limitation is the dependency on a phone-driven proxy session. A deployment that cannot install a trusted HTTPS certificate on the phone, cannot keep the phone and collection host on the same network, or cannot open WeChat pages interactively does not match the documented workflow. Automatic history-page turning also depends on the database already containing the necessary basic public account information. The README does not promise that a single article-page visit can seed a public account's complete history.

The page-routing rule is narrower than a general browser automation system. History pages navigate to history pages, and article pages navigate to article pages. The two are not interchangeable in the documented logic. Automatic behavior can also be disabled through `config.js`, and missing comments or article text may reflect configuration rather than an error in MongoDB.

For a team that only needs records already collected by this application, a simpler alternative is to query MongoDB directly or use `mongoexport`. That approach does not interact with WeChat and does not require AnyProxy after capture, but it cannot discover new public accounts, traverse history pages or collect fields that were never stored. Another alternative would be a separate browser automation or API-based collector, but the repository supplies no such implementation, so its compatibility with WeChat cannot be established from this source.

The MIT license identifies the project's licensing terms. That status does not answer whether a particular collection use complies with WeChat rules, account permissions, privacy obligations or applicable law. Those questions must be checked separately before handling data.

Editorial conclusion

Adopt wechat_spider when you operate the required Node.js, MongoDB, Redis, AnyProxy and mobile network setup and need browser-mediated WeChat data collection. Do not choose it for unattended headless ingestion, because the documented workflow depends on proxy configuration, certificate installation and opening pages in WeChat. Before adoption, verify that your account and content use are permitted, confirm the required WeChat page types and fields are available through the current interface, and test the saved data in MongoDB first.

Frequently asked questions

Official sources

1. Issues
2. License: MIT
3. lqqyt2423/wechat_spider on GitHub
4. README