Sniper: scraping a Chinese review site past its dynamic font encryption

The problem is the font, not the HTML

The repository's own summary says what it is: a Dianping crawler where the whole site can be crawled and which solves the dynamic font encryption problem, not with OCR. The note says it is continuously updated.

That detail is the interesting technical claim, and it is worth unpacking because it determines the whole architecture. The site's defence is not that the data is missing from the HTML, it is that the numbers in the page are rendered in a font whose glyph shapes are remapped for each response. A crawler that reads the HTML sees digits that are correct as characters but that do not match the shapes the browser draws, so a price or a score arrives as a string of characters that mean nothing.

OCR is the obvious answer, and the project deliberately does not use it. Reading the rendered glyphs with an OCR engine is slow, it is a dependency on an accurate model, and it introduces its own error rate into numbers that then have to be trusted. The alternative the project takes is to work out the mapping for the current response and translate through it, which turns a rendering problem into a data problem.

The project also links its own written notes on the two encryption cases, one for the search page and one for the review page, and those are the right documents to read if you want the detail. The architecture follows: you establish the mapping for a response, apply it to the extracted fields, and store the result.

The two pages are treated separately because the defence is not uniform. A search result page and a review page have different field sets and different protections, which is why the crawler has a search stage, a detail stage and a review stage, and why each can be enabled or skipped independently at run time.

The configuration surface is the real barrier to entry

The repository is candid about the cost of running it, and the admission is in bold in the documentation. Because Dianping's anti-scraping measures are very strict, the project uses a cookie pool, IP proxies and other measures against being banned, and it recommends a proxy service. The framework also gives the user a high degree of operational freedom, and that has a cost: there are more than thirty parameters to configure, although most of them can use their defaults, so the barrier to getting it running is relatively high.

The configuration is split across two INI files, and the split is meaningful. `config.ini` holds the mechanical settings, and `require.ini` selects the crawling strategy.

The `config.ini` parameters fall into groups. The `config` group has `use_cookie_pool` for whether to use the cookie pool, `Cookie` for the cookie information with the note that the capitalisation is there so browser information can be copied in without modification, `uuid` and `tcv` for the identifiers the API expects, `user-agent` for the browser UA, with the note that random UA is not currently supported, `save_mode` for the storage method which can currently only be mongo or mongodb, `mongo_path` for the database configuration, and `requests_times` for the interval between requests.

The `detail` group holds the search settings: `keyword` for the search term, `location_id` and `channel_id` for the region and channel, `search_url`, `need_first` for whether only the first result on the first page is wanted, and `need_pages` for how many search pages to fetch.

The `proxy` group is where the anti-ban measures live: `use_proxy`, `repeat_nub` for how many times an IP may be repeated, `http_extract` and `key_extract` for the two extraction modes, `http_link` for the HTTP extraction endpoint, `proxy_host` and `proxy_port` for the key mode proxy server, and `key_id` and `key_key` for the credentials.

That last group is where the operational risk sits, and it is the part a reader should think about before running anything. A key mode proxy tunnel and a cookie pool exist for one reason, to make automated requests look like distinct human sessions, and the documentation recommends a commercial proxy service that advertises residential IPs across 195 countries and 90 million addresses for bypassing regional and rate limits.

Two config files with different risk profiles

The `require.ini` file selects what to fetch, and the documentation puts a careful header on it, with a column for cautious selection.

Under `shop_phone`, there is `need` for whether the shop phone is wanted, and `need_detail` for whether the full phone number is wanted rather than the masked form such as 12345678, with the masked form being the default and the full form marked as requiring care.

Under `shop_review`, `need` is whether reviews are wanted, `need_detail` is whether the extended set of reviews is wanted rather than only the ten featured ones, and `need_pages` is how many pages of reviews to fetch, one page being thirty entries, taking the smaller of the shop's available review pages and the configured value.

And then the warning, which is the most important sentence in that part of the documentation. For the cautiously selected configuration, since a login is required to obtain it, the requests carry cookies, and frequent requests will cause the account to be banned, with the ban lifting automatically after a while.

So the two configurations are not simply about how much data you want. The `need` settings marked as cautious are the ones that need authenticated data, and the project tells you that asking for them means your account is exposed to being banned for asking. That is a real trade-off being documented honestly, and it explains why the defaults are the masked phone and the ten featured reviews.

The run modes follow the same structure. A full run is `python main.py`, which does search, then optional detail, then optional review. The three command line forms let you run only one of the stages:

python main.py --normal 0 --detail 1 --review 0  --shop_id k30YbaScPKFS0hfP --need_more False

python main.py --normal 0 --detail 0 --review 1  --shop_id k30YbaScPKFS0hfP --need_more False

python main.py --normal 0  --detail 1 --review 1  --shop_id k30YbaScPKFS0hfP --need_more False

The `need_more` flag is the one that turns on the cautious behaviour, and the shop id is the identifier that replaces the search stage when you already know which shop you want. That combination, a known shop id with need_more, is the shortest path to the data that requires a login.

Raw output, one storage backend, and a code layout that shows the stages

Two design choices in the output path are worth noting.

The first is that the data is stored raw. Because the anti-scraping measures are severe and the field formats differ between channels, much of the data is not processed during the crawl. It is saved as it is, for you to clean afterwards, with the note that newer versions introduced interface data and some fields had to be processed to line up with the interface storage format.

So you get the messy data plus the knowledge of which fields are reliable, rather than a clean table that hides the variation. For a researcher who wants to know what the site actually returns, that is the right trade. For a pipeline that wants a stable schema, it means the cleaning work is yours.

The second is the storage target. The supported write types are MongoDB, and CSV is struck through with the note that it is not supported after the version update and there is no plan to adapt it. If you need a different database, the project asks you to contact them or to add it and open a pull request. The requirements file confirms the direction, with `pymongo==3.11.3` pinned to an exact version while everything else uses a minimum.

The code layout mirrors the pipeline. The top level has `main.py` as the entry point, `config.ini` and `require.ini` for the two configuration files, a `cookies.txt` for the cookie pool, and directories for `function/`, `utils/`, `files/`, `js/` and `docs/`. The `js/` directory holds the browser userscript extension, credited to a contributor, and `docs/` holds the reference material, with a separate document for each of the areas the documentation calls out: the JSON interface details, location ids, cookie pool notes, storage notes, proxy notes, the data specification, and a page of common problems.

The dependencies are the usual set for this kind of work and they tell you the techniques in use. `lxml` and `beautifulsoup4` for parsing, `requests` for the HTTP, `tqdm` for progress, `faker` for generating the fake identity values, `fontTools` for the font handling that the encryption problem needs, and `pymongo` for the output. The presence of `fontTools` is the clearest signal of how the font encryption is handled, since working out a glyph mapping means reading the font itself.

The environment requirements are Python 3 on Windows, Linux or macOS, and the dependencies can be installed in one step with `pip install -r requirements.txt`.

The terms the project sets for itself

The repository states its own limits in a way that is worth taking at face value, because the boundaries are narrower than the feature list suggests.

The usage note says it is for learning and exchange only, and commercial use is prohibited, and reposting without authorisation is prohibited. The licence section says the project follows the GPL-3.0 open source agreement, and that you can use it for yourself, but if you profit from using it, including commercially, doing paid work for others and other private jobs, that is not permitted. The author also reserves the right to pursue liability for infringement, and says any resulting legal dispute is not their responsibility.

That is an unusual reading of GPL-3.0, because the GPL grants those rights rather than withholding them, and a licence that removes rights the licence itself grants is a matter for a lawyer rather than for a reader. The author's stated intent is clear, and the fact that the terms are stated plainly in the README means a user is not misled about the expectation.

The second limit is about maintenance. Because of work reasons this repository may not be maintained, but barring accidents the code should remain valid for a fairly long time to come. And the support policy is explicit: bug reports and proposals are welcome by email or issue, but all questions about how the program runs are to be solved by the user, with a pointer to the running instructions, and questions from people who have not read the documentation are not accepted, and when asking a question you should show the process of thinking and verifying you went through.

The third is the legal one, and the documentation does not leave it implicit. Learners must follow the law, and the closing line of that section is the part to remember. The data being collected is a commercial platform's merchant records, including phone numbers and customer reviews, and the platform's terms and the law in the relevant jurisdiction both apply to what you do with it regardless of how the technical obstacle was solved.

So the honest summary is that this is a study of a defence mechanism with a working implementation, released for learning, with terms that are narrower than the GPL header implies, and with a dependency on proxy infrastructure that exists to make large-scale automated access look human.

Editorial conclusion

Sniper fits a developer who wants to understand how a Chinese consumer site defends its data, because the font encryption handling and the configuration surface show the shape of that defence in concrete terms, and because the collected data is saved raw for the researcher to clean rather than pre-chewed. Skip it if you need a commercial data pipeline, since the project states it is for learning and non-commercial use only and the licence terms say a user who profits from it is not permitted, and skip it if you want a supported tool, since the author says plainly that the repository may not be maintained. Before you run it, read the configuration requirements, because there are more than thirty parameters and most of them have to be right, and note that the storage target is MongoDB only, since CSV writing was removed in a later version.

Frequently asked questions

Official sources

1. Issues
2. License: GPL-3.0
3. README
4. Sniper970119/dianping_spider on GitHub