Liuli: a pipeline that turns noisy subscription feeds into a clean reading feed

The pipeline is four components, and the config file is the interface

Liuli is described in the repository as a one stop build of a multi-source, clean and personalised reading environment, and the implementation section breaks the flow into four named parts.

The collector, or 采集器, monitors the public accounts, books or blog sources you follow and brings them in in a unified standard format as the input to Liuli. The processor, or 处理器, applies custom processing to that content, and the documentation gives two examples, building a machine learning ad classifier from historical ad data to tag articles automatically, and introducing hook functions that run at relevant points in the flow. The distributor, or 分发器, works through an interface layer that takes requests and returns responses, provides the personalisation configuration for the user, and then distributes automatically according to that configuration, sending the clean articles to WeChat, DingTalk, Telegram, RSS clients or a self hosted website. The backup, or 备份器, persists the processed articles, to a database or to GitHub for example.

The stated philosophy is that configuring the input, the processing and the output is what makes this work, and the feature list makes that concrete with configuration based development where you customise input, processing and output. Backups are listed as supported across sources, with GitHub and MongoDB named.

The diagram of the flow is in the repository as an SVG, and the four-stage decomposition is the real architecture. It matters because each stage is replaceable. A collector that only handles WeChat public accounts, a processor that does nothing but strip one known ad pattern, and a distributor that writes to a single self hosted site is a complete and useful system, and it is also the starting point for everything the project is eventually aiming at.

The ad classifier is the stated goal, and it is still a checkbox

The feature list in the README is honest in a way that many project READMEs are not, and the honesty is the most useful thing in it.

Four items are ticked. Configuration based development with custom input, processing and output. Information backup across sources, with GitHub and MongoDB named. Official example technical support. And a set of usage scenarios, of which the checked ones are all WeChat public accounts, covering the advertising problem with a linked article on building a clean and personalised public account reading environment, the RSS subscription problem with a linked article on building a pure RSS feed from public accounts, and book or novel follow-up subscription with a linked article on following and reading novels with Liuli.

Three items are not ticked. The machine learning features, which cover captcha recognition, ad classification and smart tagging. Reading source management, which would build a knowledge management platform. And target monitoring, which is the general case of watching arbitrary sources rather than the specific WeChat ones.

That is the honest position, and it should change how you read the project. The WeChat public account path is the worked case, with articles in Chinese describing how to solve the ad problem, the RSS problem and the novel subscription problem. The general reading source management, the target monitoring, and the machine learning features are aspirations recorded as a roadmap rather than features in the code.

The project says the same thing in a different place, in the development progress board. v0.2.0 is described as implementing the basic functionality and making the regular scenarios applicable, and v0.3.0 as implementing collector customisation, so that what a user sees is what can be collected. That second item is the interesting one, because it means the currently working path, the hard coded source list, is what is planned to become user configurable.

Playwright does the collecting, which is why it needs a browser

The acknowledgements section names the four core open source dependencies, and they explain the whole shape of the system.

Flask is the web framework, Vue is the progressive JavaScript framework for the front end, Ruia is an asynchronous crawler framework written by the same author for his own use, and playwright is used to scrape data with a browser.

That last one is the significant choice. Scraping a WeChat public account feed with a headless browser rather than with an HTTP client and a parser means the collector runs a real browser, navigates the same pages a reader would, and sees the same content. It handles the cases where the content is assembled client side, and it sidesteps a lot of what makes direct scraping of that platform awkward. The cost is that a headless Chromium has to be running, and the schedule container is built for exactly that, with a schedule_playwright.Dockerfile in the repository alongside the plain schedule.Dockerfile.

The startup log in the README shows the shape of a run. It loads the .env environment variables, then the schedule starts successfully and reports a schedule time, and then a run logs that playwright matched the public account, that it is extracting the latest article, that the article was persisted, and that the WeChat article update finished, with a count of one out of one. That log is a small but revealing artefact: the collector identifies a source by matching, extracts the latest article, persists it and reports completion. It is a scheduler running a configured set of sources.

The Ruia framework being the author's own asynchronous crawler is worth noting for a reader who wants to extend the collector. It means the crawling layer is not a well documented third party project but code the author controls, so understanding how it works means reading his code.

Docker Compose brings up an API, a scheduler and MongoDB

The quick start assumes Docker is installed and walks through creating the directories, downloading a default configuration and a compose file, and starting it:

mkdir liuli && cd liuli
# 数据库目录
mkdir mongodb_data
# 任务配置目录
mkdir liuli_config
wget -O liuli_config/default.json https://raw.githubusercontent.com/howie6879/liuli/main/liuli_config/default.json
# 配置 pro.env 具体查看 doc/02.环境变量.md
vim pro.env
# 下载 docker-compose
wget https://raw.githubusercontent.com/howie6879/liuli/main/docker-compose.yaml
# 启动
docker-compose up -d

The compose file itself shows the three services. There is an liuli_api service on the image liuliio/api:v0.1.4 with port 8765 mapped, and an liuli_schedule service on the image liuliio/schedule:v0.2.4 which mounts both the pro.env file and the liuli_config directory, and an liuli_mongodb service on mongo:3.6 with the root username and password set to liuli and the data directory mounted for persistence. All three sit on a bridge network called liuli-network, and both application services depend on the database.

The split into three services is the architecture made concrete. The API is the interface layer the distributors talk to, the scheduler is the component that polls the sources on the configured schedule, and the database holds the configuration and the persisted articles.

There is also a source install path, for people who do not want Docker:

# 确保有Python3.7+环境
git clone https://github.com/liuli-io/liuli.git --depth=1
cd liuli

# 创建基础环境
pipenv install --python={your_python3.7+_path} --dev  --skip-lock
# 配置.env 具体查看 doc/02.环境变量.md 启动调度
pipenv run dev_schedule

The Pipfile is in the repository root, which is where the fuller list of third party dependencies is, and the project pins Python 3.7 or newer as the environment for this path.

The ad sample dataset is the part you can actually contribute to

There is one part of the project that is genuinely collaborative, and it is the dataset rather than the code.

The help section says that to improve the recognition accuracy of the model, the author wants people to contribute advertising samples, and points at a file, .files/datasets/ads.csv, with a three column format: title, url and is_process, where title is the article title, url is the article link, and is_process indicates whether the sample should be processed, with a default of 0. The format table is given as a markdown table with the example rows in Chinese.

There is a practical note attached about avoiding duplicates. The same advertisement is often placed by several public accounts, so contributors are asked to check whether a record already exists before adding it, and to open a pull request.

This is a sensible way to build a labelled dataset for a classifier, and it also reveals the practical difficulty. An ad is not a thing with a fixed title and URL, it is a placement inside somebody else's feed, so the same advertiser appears with different headlines and different tracking links in different accounts. The dataset is asking for samples, and the samples are the hard part.

The other thing the dataset says is that the classifier is not trained for you. There is no model file, no training script and no accuracy figure in the documentation, only a request for labelled examples and a roadmap item for the machine learning features. So a reader who wants the ad filtering working on day one should expect to do that work, or to wait for it.

Where Liuli is the wrong choice

Three situations rule it out, and the first one is about the scope of the sources.

The first is a general feed reader. The working scenarios are WeChat public accounts, the RSS feed built from them, and novel follow-up subscription. The general case, arbitrary target monitoring, is an unticked item on the feature list, and collector customisation is the v0.3.0 milestone rather than a shipped feature. If you want to follow a hundred ordinary blogs and have them deduplicated, this is not the tool yet, and adding a new source type means working with the author's own crawler framework rather than writing a collector against a documented interface.

The second is if you need the ad filtering to work now. The machine learning features, including ad classification and smart tagging, are listed as not yet done, and the dataset is a request for contributions rather than a shipped model. A user who installs this expecting a clean feed will find the pipeline working and the classification not working, and the two are separate stages, so the plumbing will look fine.

The third is a long term dependency. There is a single tagged release, v0.2.0 from 2022-02-10, while the last push to the repository was on 2026-10-16. The project is not archived and the code is being worked on, but the version history does not tell you which release corresponds to the current state of the tree, and the compose file pins specific image tags, liuliio/api:v0.1.4 and liuliio/schedule:v0.2.4, which are separate from the repository version.

The licence is Apache-2.0, and the project description mentions WeChat as a topic. The subject matter is the reason to think carefully about the operating environment: a deployment that fetches and republishes content from a platform is doing something that platform's terms may have an opinion about, and that is a question for your own compliance review rather than for this page.

Editorial conclusion

Liuli fits a reader who is tired of the advertising in a set of followed WeChat public accounts or blogs and wants one clean feed out of them, and it fits a developer who wants the collection, processing, distribution and backup stages to be separately configurable rather than welded into one script. It is the wrong choice if you expect the ad filtering to work out of the box, because the machine learning features are listed as unchecked, and it is the wrong choice if you want a maintained general purpose feed reader. Before deploying it, read the four configuration documents under docs, since the environment variables and the distributor and backup configurations are where the behaviour actually lives, and check the project version you are running against the single tagged release.

Frequently asked questions

Official sources

1. howie6879/liuli on GitHub
2. License: Apache-2.0
3. Project website
4. README
5. Releases