cnbattle/douyin: a Go crawler that scrapes Douyin through a real app and a MITM proxy certificate

The crawler is the app, driven with adb

This is a scheme for collecting video lists from the Douyin recommendation and search pages, and its architecture is unusual compared with a normal web crawler. There is no HTTP client talking to a web API. Instead a real Douyin app runs on an Android emulator or a real device, adb drives the interface, and the app's own traffic is intercepted with a man in the middle proxy so the responses can be read.

The stated stack is `golang` and `adb`, and the Go module confirms the shape of the implementation. The module is `github.com/cnbattle/douyin`, built for Go 1.22.0 with a toolchain line of go1.24.1, and it depends on `github.com/ouqiang/goproxy` for the proxying, `github.com/spf13/viper` for configuration, `gorm.io/gorm` with `github.com/glebarez/sqlite` for storage, and `github.com/dustin/go-humanize` among the indirect dependencies.

The repository layout matches the flow. There is an `apps/` directory for the Android side, `internal/` for the Go internals, `main.go` at the root, a `config.example.toml` at the top level, a `mitm-proxy.crt` file in the repository root, and a `download/` directory for output. The `example/example.gif` in the tree is a recording of the tool in use, which is the fastest way to understand what it does.

The design has one practical advantage and one large practical cost. The advantage is that you always receive whatever the app itself receives, including the signatures and parameters the app computes for you. The cost is that everything now depends on a real app on a real device, and on a proxy certificate that has to be trusted by that device.

Installing the certificate is step one of the setup

The usage instructions are short and the first step is the one that decides whether anything works. You install the `mitm-proxy.crt` certificate on the relevant device, whether that is a virtual machine or a real phone, and the project ships that certificate file in the repository root so you have the exact file the tool expects.

Then the configuration is copied and adjusted:

cp config.example.toml config.toml

The instructions say to modify the parameters according to your own needs, and the two headline parameters are the ones the features list names: you can set a threshold so that only videos with more than a given number of likes are collected, and you can choose whether remote files are downloaded to the local disk at all.

After that the program is run from the source tree:

go run main.go

The instructions also mention running it from an editor, which is a fair suggestion for a Go program that needs a device attached over adb.

The certificate step deserves a second thought rather than a reflex. Installing a CA certificate on a device makes that device trust traffic intercepted by the proxy, for every application on it, not just the one you are studying. If the device holds accounts, credentials or other applications' traffic, that trust is broader than the task requires. The project itself says the scheme is for personal learning and research and must not be used for commercial purposes, and the certificate should be removed from the device when the work is done.

The output is a SQLite table and three directories of files

With downloads enabled, the run produces a `database.db` file, a SQLite 3 database, whose fields are described in `model/videos.go` in the repository. The static files, meaning user avatars, video cover images and the video files themselves, are written into `download/` under the subdirectories `avatar`, `cover` and `video`.

That layout is a sensible design for a dataset. The metadata lands in a real database rather than a file of JSON lines, so you can query it, and the binaries are kept out of the database and on disk in a predictable structure. The Makefile's static build configuration and the `gorm` and `glebarez/sqlite` dependencies are what make that possible, and the choice of a pure Go SQLite driver means the program does not need a C library or a system database to run on the host.

The final step in the documented workflow is simply to upload and process the results as needed, which is another way of saying the project deliberately stops at collection. There is no deduplication, no enrichment and no scheduling here, and the `model/videos.go` file is the reference for the exact columns you get.

The repository also carries a `Makefile` that builds static binaries for a wide range of targets, with `TARGETS` defaulting to `linux windows js` and `ARCHS` covering amd64, 386, arm, arm64, mips variants, ppc64, riscv64, s390x and wasm, and it stamps the binary with a version and Drone build metadata through linker flags.

Two admitted defects: the app crashes and the emulator freezes

The README has a section titled with what is left to improve, and it is worth reading because it tells you what will happen during a long run.

The first item is that the app crashes abnormally when it is scrolled through the recommendation list continuously. The stated temporary workaround is to force close the app and reopen it after a period of crawling, and that period is configurable in the configuration file. So the crawl is not a single continuous session, it is a cycle of sessions with a restart in between, and a run that appears hung may simply be waiting for its scheduled restart.

The second item is more open ended. The author uses the LDPlayer emulator, and it freezes after long runs, with the cause unclear, either the system or the emulator. The author asks for help from anyone who can keep the environment stable.

Those two entries define the operational envelope of this project. It works, on a scale measured in sessions rather than in hours, and the stability of the whole thing rests on an app and an emulator that the project does not control. Both problems are in the app and emulator layers, not in the Go code, which is the usual situation for this kind of tool and the reason such tools age quickly.

The search page is not implemented, and an older branch behaves differently

A note near the top of the README says that adb does not yet implement the operation of entering the search page, and asks you to implement the relevant click operations and keyword input yourself according to your own technology stack. In other words, the recommendation feed is the working path and the search path is left as an exercise, with the interface for driving the UI left open rather than prescribed.

There is also a branch note: for older versions, switch to the `old` branch, which uses anyproxy to capture traffic and is said to be more suitable for most people. So the project has two interception strategies, and the current branch is not necessarily the easier one to set up. That is the kind of detail that a README only mentions when someone has already been confused by it.

The project also states its own limits in a closing section: the author says their ability is average and asks for tolerance of non standard coding, invites pull requests for modifications, and asks for issues or a WeChat contact for problems. The tone is that of a single developer project, and the `Makefile` targets for `vet`, `revive` lint, `misspell`, `gosec` and formatting checks show the kind of tooling a careful contributor would be expected to satisfy.

A personal study tool with a licence that does not match its own README

There is a tension here that any adopter should notice. The project is published under the MIT licence, and the `LICENSE` file sits at the top level. But the README closes with a statement that the project is only for personal learning and research and must not be used for commercial purposes. An MIT licence does not normally carry that restriction, so the two statements are in tension, and the practical answer is that the author's declaration in the README is a request, not a licence term, which means a commercial user relying on the MIT grant would be in a dispute rather than in a clear position.

The README also carries a prominent sponsor banner for a commercial residential proxy service, advertising precise geolocation across many countries, a free trial that does not expire, and protection accounts that avoid bans for crawling, with a discount code. That is the commercial context of the technique: the author sells nothing, but the tool's own documentation points at a paid service for reducing the risk of being blocked. It also tells you the technique is detectable, which is the more useful fact in the banner.

On maintenance, the honest reading is mixed. The last push to the repository was on 2026-05-27, and the README notes that the most recent working test was on 2026.05.25, so the code is being touched and the author is testing it. But the newest release tag is v0.2.0 from 2021-06-20, with v0.1.0 from 2020-10-11 before it, so there are no recent tagged binaries and the source tree is the only current artefact.

Editorial conclusion

Adopt this project only if you are doing personal research on how the Douyin app's API behaves, on a device you own, and you accept that the interception model depends on a trusted CA certificate installed on that device. Do not adopt it for a commercial pipeline, because the project states it must not be used for commercial purposes, and do not point it at a production or business account, since the sponsor banner in the README explicitly advertises proxy pools that avoid account bans, which is an admission that the technique is detectable. Verify first that the `mitm-proxy.crt` certificate can be installed and trusted on the device, that the app does not crash during the crawl, since that is a known defect, and that the release binaries still match the current source, given that the newest tag is v0.2.0 from 2021.

Frequently asked questions

Official sources

1. cnbattle/douyin on GitHub
2. Issues
3. License: MIT
4. README
5. Releases