Open-source project
wzdnzd/aggregator avatar
wzdnzd/aggregator

Aggregator: crawl, validate and republish free proxy subscriptions

One-stop Proxies Crawling and Aggregation Platform

7,576 stars5,704 forksPythonApache-2.0

At a glance

What is it?
Aggregator crawls public Telegram, GitHub and search sources for proxy subscriptions, keeps the entries that answer, converts them to Clash, V2Ray and SingBox formats and republishes them to a Gist, PasteGG or Imperial backend.
Who is it for?
Aggregator fits people who want the plumbing of collecting, validating, converting and republishing handled for them, and it fits badly if you need a stable proxy source or a documented extension API: the inputs are other people's public posts, the conversion and testing steps live in Subconverter and Mihomo, and the plugin refactoring is still an unticked roadmap item.
Can I use it commercially?
Yes. Apache-2.0 is a permissive licence: you can use, modify and sell software built on it, as long as you keep its copyright and licence notices.
Is it still maintained?
Yes. The repository last received commits 29 days ago.
What is it written in?
Mainly Python, according to GitHub's language statistics.

Answers come from the project's GitHub data, last synced on November 20, 2026, and from our analysis. They are not legal advice.

From daily snapshots

Star history

  • wzdnzd/aggregator7,576 ★ · +809
Star history
Repository2026-09-232026-11-20
wzdnzd/aggregator6.8k7.6k

809 Stars in the last 58 days, measured from the daily GitHub snapshot. Compare with other projects

Editorial analysis

Two entry points: collect.py for speed, process.py for control

Aggregator ships two scripts and they are not variations of the same thing. subscribe/collect.py is the short path: it collects airport subscriptions, checks them and pushes the result onward, with the destination, the token and the target formats all passed on the command line.

bash
python subscribe/collect.py \
    -g username/gist-id \
    -k your-github-token \
    -t clash v2ray singbox

subscribe/process.py is the full version and it is driven by a JSON config file rather than by flags.

bash
cp subscribe/examples/config.default.json my-config.json

export PUSH_TOKEN=your_github_token

python subscribe/process.py -s my-config.json

The README marks process.py as the recommended path and keeps collect.py for people who just want free proxies for personal daily use with no complicated configuration. That choice costs something: collect.py reads no config file, so the storage engine, the group layout and the crawl targets are not yours to set. Only process.py reaches the Telegram, GitHub, Google, Yandex and Twitter crawlers and the custom rules behind them.

Twelve protocols in, three client formats out

The protocol list is longer than the format list, and that asymmetry is the shape of the project. In: VMess, Trojan, SS, SSR, Snell, Hysteria2, VLESS, Hysteria, TUIC, AnyTLS, HTTP, SOCKS. Out: Clash, V2Ray, SingBox. Crawlers take whatever the source pages happen to publish, Mihomo acts as the proxy testing engine that decides which entries stay alive, and Subconverter does the subscription conversion.

Both of those are external projects that the README credits rather than code inside this repository. The Dockerfile copies the subconverter directory in and deletes its darwin, linux-arm and windows binaries. The practical consequence is that a protocol is only supported end to end if Subconverter knows it, so adding a protocol to the crawlers does not by itself make it publishable. The two lists are maintained separately, which is worth checking before you automate anything on the output.

Storage is a routing decision keyed by group

A site is listed by name and domain and declares which output group it feeds. That is what the top of the minimal config example looks like.

json
"sites": [
    {
        "name": "example-airport",
        "domain": "example.com",
        "push_to": ["free"]
    }
],

The group then decides which client target gets produced, and the storage block decides which backend receives it.

json
"groups": {
    "free": {
        "targets": {"clash": "free-clash"}
    }
},
"storage": {
    "engine": "gist",
    "items": {
        "free-clash": {

Two hops to read that: a site pushes into a group through push_to, and the group maps to one target name, free-clash in this case, which then appears as a key under storage.items. The Telegram branch repeats the same pattern, with crawl.telegram.users.proxy_channel.push_to sending one channel into the same free group. Because what a client consumes is a subscription URL, the GitHub token is a requirement rather than a convenience, and Gist is only one of three storage backends the feature list names.

Timeouts and thread count are the dials when output is empty

The README answers its own failure modes in a small table, and the answers are specific. A config that will not load is a syntax problem, checked with python -m json.tool config.json. A rejected Token means the GitHub Token permission or its expiry, not the proxies. A network timeout is answered with two dials: raise the timeout to -t 15000 or drop the thread count to -n 16. No proxy output at all points back at the crawl source configuration and the network connection.

bash
python subscribe/process.py -s config.json --check

python subscribe/process.py -s config.json -n 128

--check is the cheap first move because it answers whether entries survive validation without paying for the whole aggregate and push cycle. -n 128 is labelled the high performance mode, and the same knob set to 16 is what the timeout row recommends when the network is struggling, so one flag is doing double duty as a load dial and a restraint dial. One more detail from that block: the token arrives as PUSH_TOKEN in the shell, while the container image reads GIST_PAT.

The container path is linux/amd64 only

The Dockerfile is the clearest statement of the supported platform in the repository. It starts from python:3.12.3-slim, copies requirements.txt, the subscribe directory and two files out of clash/, then deletes every subconverter binary that is not linux/amd64 before installing dependencies.

dockerfile
COPY clash/clash-linux-amd clash/Country.mmdb /aggregator/clash
RUN rm -rf subconverter/subconverter-darwin-amd \
    && rm -rf subconverter/subconverter-darwin-arm \
    && rm -rf subconverter/subconverter-linux-arm \
    && rm -rf subconverter/subconverter-windows.exe

That trim keeps the image smaller, and it also closes the container route to macOS and to arm64 hosts. Country.mmdb travels into the image alongside the amd64 client binary, and geoip2 sits in requirements.txt, so country data is bundled rather than fetched at run time. The base image pins one Python version, 3.12.3, which is the version any container deployment will be running.

The image's default command uses flags the README never explains

The last line of the Dockerfile runs collect.py with three flags that appear in none of the documented invocations.

dockerfile
CMD ["python", "-u", "subscribe/collect.py", "--all", "--overwrite", "--skip"]

The documented calls use -g for the gist, -k for the token and -t for target formats, while the image depends on --all, --overwrite and --skip and takes its credentials and destination from the environment instead.

dockerfile
ENV GIST_PAT=""
ENV GIST_LINK=""
ENV CUSTOMIZE_LINK=""

Two spellings of one credential is a real papercut: set PUSH_TOKEN and the shell path works, set GIST_PAT and only the container path does. The documented build line is a docker buildx invocation that takes a mirror argument, --build-arg PIP_INDEX_URL, and the file defaults that argument to https://pypi.org/simple, which matters when the default index is slow from where you run it.

The plugin system is a roadmap item, not an interface you can code against

The feature list advertises an extensible custom crawling architecture. The roadmap in the same file lists that work as unfinished: abstracting ICrawler, IStorage and IConverter, adding BaseCrawler, BaseStorage and BaseConverter, moving the Telegram, GitHub and Google crawlers into standalone plugins, moving the Gist, PasteGG and Imperial backends into plugins, then adding automatic plugin discovery, registration and a standardised plugin config. Every one of those boxes is unticked.

The same list shows configuration is not modelled yet either, since a Pydantic typed config, completeness validation and templated config are all future items, as are structured logging, a unified exception hierarchy with error codes, Black with isort and flake8, and a rework of thread pool scheduling. The README's plugin development note says the project already has a flexible plugin system for custom crawl targets and welcomes crawler plugins, but a plugin written today would be written against module internals that the same roadmap plans to move.

Where the proxies come from and what the project asks of you

The inputs are public pages and channels: Telegram, GitHub, Google, Yandex and Twitter for the crawlers, plus airport subscriptions which, following the workflow diagram, mean registering an account and fetching a subscription before anything is validated. The dependency list is short and carries no version constraints.

text
PyYAML
tqdm
geoip2
pycryptodomex
fofa-hack

fofa-hack stands out as the only name in that list that is not a general purpose library, and the absence of any pin means a rebuild can pick up different code than the one you tested. The disclaimer in the README limits the project to learning crawler techniques, forbids commercial use, and asks users to follow local law and respect the terms of the sites being crawled. For ready made subscriptions rather than a build of your own, the project points readers at Issue #91. There are no GitHub releases, and the last push to main was 2026-10-23.

Editorial conclusion

Aggregator fits people who want the plumbing of collecting, validating, converting and republishing handled for them, and it fits badly if you need a stable proxy source or a documented extension API: the inputs are other people's public posts, the conversion and testing steps live in Subconverter and Mihomo, and the plugin refactoring is still an unticked roadmap item. Before scheduling it, check three things: whether your host is linux/amd64 if you plan to use the image, whether you are setting PUSH_TOKEN or GIST_PAT, and what -n 128 does to the sites being crawled rather than to your own connection.

Frequently asked questions

What does wzdnzd/aggregator collect, and where does the output go?

It crawls proxy resources from Telegram, GitHub, Google, Yandex and Twitter, checks which entries are alive, converts them to Clash, V2Ray and SingBox formats and stores them through the Gist, PasteGG or Imperial backends. The protocol list covers VMess, Trojan, SS, SSR, Snell, Hysteria2, VLESS, Hysteria, TUIC, AnyTLS, HTTP and SOCKS.

Should I run collect.py or process.py in aggregator?

collect.py is the simplified path, taking -g for the gist, -k for the GitHub token and -t for the target formats. process.py is the full version, started with -s on a JSON config file, and it is the script that reaches the multi source crawlers, the custom rules and the Gist, PasteGG or Imperial backends.

How do I fix a config error or a rejected token in aggregator?

For a config file that will not load, validate the syntax with python -m json.tool config.json. For a token the run refuses, check the GitHub Token permission and its expiry. Note that the shell path reads PUSH_TOKEN while the container image reads GIST_PAT.

Why does aggregator produce no proxies, and what do -n and -t change?

The README points first at the crawl source configuration and the network connection. For timeouts it gives two dials, raising the timeout to -t 15000 or cutting the thread count to -n 16, while -n 128 is the documented high performance setting. The --check flag validates proxy liveness without running the full aggregate and push.

Can I run the aggregator container on macOS or an arm64 host?

Not through the image, as it is written. The Dockerfile starts from python:3.12.3-slim, copies clash/clash-linux-amd, and the build step deletes the darwin, arm and windows subconverter binaries, so the container route is linux/amd64. Running subscribe/collect.py or process.py directly needs whatever the host already provides.

Official sources

  1. Issues
  2. License: Apache-2.0
  3. Project website
  4. README
  5. wzdnzd/aggregator on GitHub
Add this badge to your README

If you maintain this project, the badge below links readers to this analysis and shows its maintenance status from the daily GitHub snapshot. Paste the markdown into your README; add ?metric=license or ?metric=stars to the image URL for a different field.

Add this badge to your README

markdown
[![Hysen Labs](https://hysenlabs.com/badge/wzdnzd-aggregator.svg)](https://hysenlabs.com/projects/wzdnzd-aggregator)

Star history chart for your README

markdown
[![Star history](https://hysenlabs.com/chart/wzdnzd-aggregator.svg)](https://hysenlabs.com/projects/wzdnzd-aggregator)

Was this useful?