Leon406/SubCrawler for Locally Filtered Proxy Subscriptions

SubCrawler's Public-Node Subscription Pipeline

SubCrawler addresses a specific problem: collecting proxy nodes already published in public sources, testing their availability, filtering configurations that tend to produce bad results, and exposing the results in subscription-oriented files. Its intended audience appears to be individual users rather than commercial node operators. The README explicitly limits the project to personal use, prohibits profit-making use and unlawful activity, and asks people not to fork it unless they intend to perform further development.

The generated data includes separate VLESS and Hysteria2 files, a combined file, and several hosts files. The README labels the VLESS and Hysteria2 outputs as untested because LiteSpeed does not support those protocols. That boundary matters. An endpoint appearing in a raw subscription is evidence that the repository published it, not evidence that it works from the user's network.

The repository is primarily Kotlin and includes Gradle files, a Gradle wrapper, workflow files under `.github/`, local filter scripts, a `config/` directory, a `litespeed/` directory, and generated content under `sub/`. Based on that layout, generation and local filtering are separate stages. The GitHub workflow produces results on an overseas runner, while the user-facing scripts provide a way to filter again on the reader's own system.

Google Ping Testing and LiteSpeed Configuration

The documented local speed-test path uses LiteSpeed with a `pingonly` test mode and Google ping as its ping method. The documented `config.json` also fixes concurrency at 1024, uses a five-second timeout, deduplicates output, and sorts according to `rspeed`. This is a liveness-oriented pipeline, not a throughput benchmark. The README explicitly recommends disabling bandwidth testing for this workflow.

LiteSpeed writes a node list to `output.txt`. The documented sequence copies that file into an nginx resource directory and base64-encodes it into a V2Ray subscription. nginx then serves both the plain list and the encoded file as static resources. That architecture is easy to understand, but it also creates an operational dependency: the filtering process is local, while publication depends on a separately configured web server.

The README does not explain how SubCrawler obtains and merges its upstream public sources. It points users who want to build a node pool to the separate `Leon406/proxypool` repository and its `source.yaml`. Consequently, the exact crawl source, scheduler implementation, retry policy, and source-deduplication behavior cannot be confirmed from SubCrawler's README. The project description says it crawls and checks public nodes daily, while the repository contents indicate GitHub Actions and Kotlin source, but the README does not walk through those implementation details.

Running SubCrawler's First Local Filtering Session

The README does not document a complete installation procedure such as cloning the repository, selecting a JDK version, or invoking Gradle. It instead directs readers to the repository's local scripts and generated subscription links. The repository includes `localFilter` for Linux and macOS and `localFilter.bat` for Windows. Those scripts are the documented entry point for the first real use.

On Linux or macOS, use one of the documented forms below. The first invokes Bash directly, while the second adds execute permission before launching the script:

bash localFilter

chmod +x localFilter && ./localFilter

On Windows, the README names this command:

localFilter.bat

After filtering, the default output is base64-encoded for clients such as V2rayN, SS, and SSR. Other client formats require one of the listed subscription converters. The README links locally built subconverter, `sub2clash`, acl4ssr-sub, sub v1, 品云, and 肥羊转换 for this purpose. It does not specify which converter output corresponds to every client, so the conversion choice remains a compatibility decision.

For a first published test on a Linux server, the README documents these commands:

./lite --config config.json --test 生成链接

cp output.txt  /usr/share/nginx/res/nodes.txt

base64  /usr/share/nginx/res/nodes.txt > /usr/share/nginx/res/node.txt

The first command starts LiteSpeed in the documented mode. The second publishes the generated list under nginx's resource path, and the third creates the encoded subscription. The README does not show nginx installation, directory creation, service startup, or permissions, so those operations cannot be prescribed from the available documentation.

SubCrawler Protocol Filters and Output Boundaries

SubCrawler removes protocol combinations that may cause speed-testing problems. For SSR, the excluded methods are `none`, `rc4`, and `rc4-md5`. For SS, they are `aes-128-cfb`, `aes-256-cfb`, and `rc4-md5`. For VMESS, the filter excludes `none`, `grpc`, `h2`, and `auto`. This is a predefined compatibility policy rather than a claim that every remaining node is reliable.

Protocol support differs across the published files, LiteSpeed, and client applications. The README identifies Google, Shadowsocks, ShadowsocksR, Trojan, V2Ray, and VLESS among the repository topics. It also points to Hysteria2 output, but explicitly says that file is untested because LiteSpeed lacks support. For Windows, the README recommends V2rayN and lists SS, Trojan, VMess, and VLESS. Other listed clients have different protocol sets, including sing-box and Clash Meta clients described as supporting all protocols or the broader protocol set.

The hosts outputs have separate purposes: an advertising-blocking hosts file, a reselected Google hosts file, and a file for GitHub and common domains. The README links these files through `raw.fastgit.org`. It does not document how often those hosts entries are refreshed, how conflicts are resolved, or whether each target is verified from China. They should therefore be treated as generated resources, not as independently audited network data.

SubCrawler also lists IP anonymity, DNS leak, WebRTC leak, IPv6, and port-scan services. It does not claim that running any one test validates a node, and it does not provide evidence that generated nodes pass those services. Local testing addresses reachability. Privacy properties require separate, explicit verification for the particular node and client configuration.

Why a Second Local Test Is Required

The central limitation is location dependence. The README states that GitHub Action node tests run on an overseas server and that domestic availability is not guaranteed. It therefore advises users to run local secondary speed testing. A node can respond to Google ping from one network and fail or route poorly from another because of DNS resolution, packet loss, firewall policy, or client compatibility. The README does not quantify any of these effects, so the workflow provides no guarantee about performance.

There is another operational limitation: the project does not pin published nodes to an immutable release. No GitHub releases are listed, and the generated URLs point directly to repository content. Raw files can change as the workflow or filtering logic changes. Users who need reproducibility would have to record their own inputs and outputs, but the README does not document such a procedure.

A real alternative is `beck-8/subs-check`, which the README identifies as a local speed checker supporting VLESS and Hysteria2. Compared with SubCrawler's documented LiteSpeed workflow, subs-check covers two protocols that LiteSpeed does not test in this repository. Another alternative is building the source pipeline with `Leon406/proxypool` and publishing the result separately. That approach separates collection from SubCrawler's generated artifacts, but the README does not provide enough implementation detail to claim that it offers more reliable collection.

For users who only need a protocol-aware GUI client, SubCrawler is an intermediate data source rather than a proxy engine. `GUI.for.SingBox`, `clash-verge-rev`, V2rayN, V2rayNG, NekoBoxForAndroid, and the listed iOS clients consume subscriptions, while SubCrawler performs collection, filtering, and file generation.

SubCrawler Maintenance, Releases, and GPL-3.0 Use

The repository is not archived, and its last push was on 2026-10-15. That recent repository activity does not amount to an API or data-format stability guarantee. No GitHub releases are available, and the README does not publish a changelog, compatibility matrix, supported Gradle version, or migration procedure. Users who depend on the raw files therefore face an upgrade cost whenever generated output or source behavior changes.

The project uses GPL-3.0. That license is significant for anyone considering redistribution or incorporation into another distributed work. This review does not interpret the license or provide legal advice. The README's separate personal-use declaration is narrower than a technical description of GPL permissions, and the repository description's request not to fork should not be treated as a replacement for the license terms. Anyone redistributing or incorporating the project should review GPL-3.0 obligations separately from the README statement.

Protocol churn also raises maintenance work. The repository already distinguishes protocols that LiteSpeed can test from VLESS and Hysteria2 outputs it cannot test. Client support changes across sing-box, Clash Meta, V2ray, and other applications. Adopters should therefore inspect the current generated files and client compatibility rather than assuming that every published entry is interchangeable.

The practical upgrade boundary is clear: SubCrawler provides a current repository artifact and local scripts, not a versioned software release. Pinning a local copy may preserve prior outputs, but the README does not specify whether generated files, scripts, or upstream changes need separate handling.

Editorial conclusion

SubCrawler suits individual users who already understand proxy subscriptions and want a repeatable collection, filtering, and local testing workflow. It is a poor fit for managed service operators, legal teams seeking commercial permission, or users who require guaranteed domestic reachability. Verify first that your network can access the raw subscription files, that your chosen client supports the included protocols, and that the local filtering scripts run correctly on your operating system. The repository had no GitHub releases, so adoption should also account for unpinned generated changes and the GPL-3.0 obligations.

Frequently asked questions

Official sources

1. Issues
2. Leon406/SubCrawler on GitHub
3. License: GPL-3.0
4. README