monosans/proxy-scraper-checker for Rust proxy collection and validation

What proxy-scraper-checker actually solves

This project targets users who already have access to proxy source pages, raw text files, HTML documents, JSON documents, URLs, local files, or CIDR ranges, and want to turn that mixed input into a list of usable proxy endpoints. Its scope is narrower than building a managed proxy service. It does not provide proxies, maintain an endpoint pool, or guarantee that a proxy remains reachable after the run. Instead, it extracts candidate addresses, checks each candidate against a configured URL, and writes the candidates that complete the request in full.

The supported protocols are HTTP, SOCKS4, and SOCKS5. The README also states that SOCKS4a, SOCKS5h, proxy credentials, and CIDR ranges are understood, although the precise configuration for every protocol variant is documented in `config.toml`. The input format is deliberately permissive. A source does not need a dedicated API or a fixed schema, because the checker pattern-matches proxy-looking values from raw text, HTML, and JSON. Results are deduplicated across sources, which matters when several lists contain the same endpoint.

This is useful for building a temporary research list, refreshing a private inventory, or producing a machine-readable handoff for another program. It is a poor fit for applications that require authenticated proxies with guaranteed ownership, geographic policy, or long-term service levels. The tool records metadata about a proxy, but it does not turn that metadata into a service contract.

Extraction, deduplication, and async validation

The implementation is based on Rust and Tokio. The dependency list in `Cargo.toml` includes `tokio`, `reqwest` with SOCKS support, `serde`, `serde_json`, `ipnet`, `regex`, and `maxminddb`. That combination indicates a pipeline rather than a simple sequential grep. Sources are parsed for proxy patterns, candidates are normalized and deduplicated, and checks are performed asynchronously.

The README gives the most important validation rule: every proxy must fetch a real URL in full before it survives. This is stricter than merely opening a socket. A candidate that accepts a connection but stalls while transferring the body is discarded. The design is sensible for proxy lists because many endpoints advertise access but fail under an actual HTTP transaction. It also means that a proxy can pass one run and fail later because the target URL, remote service, route, or proxy availability has changed.

Each accepted proxy is associated with response time and, when `check_url` returns one, an exit IP. ASN and city-level geolocation are resolved through offline MaxMind databases. The README specifically says this avoids per-proxy API calls. The repository includes a `config-schema.json`, so configuration is a first-class part of the project rather than a set of undocumented flags. Optional TUI dependencies include `ratatui` and `crossterm`; the README describes an interactive view with per-protocol progress and logs.

First use with the prebuilt macOS binary

The README directs users to the latest release and says to choose an archive for the processor. The project publishes a single binary, and the documented macOS workflow is to extract it into a dedicated folder, edit `config.toml` in that folder, and run the binary from Terminal. The command below is the documented invocation. Run it from the directory containing the edited configuration so the checker can find `config.toml` in the current directory.

./proxy-scraper-checker

The README does not show a complete minimal `config.toml`, so this article does not invent source entries or replacement values. Before starting, review the inline options in `config.toml`, especially `check_url` and `max_concurrent_checks`. The checker should display progress in its TUI and later write its configured outputs. If the binary is blocked by macOS quarantine, the README documents this command:

xattr -d com.apple.quarantine proxy-scraper-checker

That command removes the quarantine attribute from the downloaded file. It does not configure a proxy source or alter the checker. After the run, inspect the configured output directory. The README says results are written as JSON with metadata and as plain text lists ready to paste elsewhere.

Configuration is the interface, and the README is not a sample

The executable reads `config.toml` from the current directory. To place the file elsewhere, the README specifies the `PROXY_SCRAPER_CHECKER_CONFIG` environment variable. The repository also contains `config-schema.json`, which is the appropriate place to look when validating editor completion or checking supported keys. The README explicitly says that every option is documented inline in `config.toml`.

That design is convenient for repeatable runs, but it creates a documentation boundary. This article can confirm the environment variable name, the configuration filename, `check_url`, and `max_concurrent_checks` from the available project text. It cannot safely provide a complete configuration file because the contents of `config.toml` were not included. Copying a guessed source URL, output path, protocol flag, or concurrency value would make the tutorial unreliable.

The project is also distributed as a single binary, which reduces runtime setup on Windows, Linux, macOS, Android, FreeBSD, NetBSD, illumos, and Solaris according to the README. Docker support is visible in the repository through `Dockerfile` and `compose.yaml`. The Compose file mounts `./config.toml`, maps `./out` to `/app/out`, and persists a cache volume. It does not, however, document a complete command that should be used as a first run, so the binary workflow is the clearer starting point.

Network load and proxy safety limits

The checker can open hundreds of simultaneous connections to untrusted hosts. The README warns that an ISP may classify this as abusive traffic and that inexpensive routers can lose connectivity when their NAT table fills. This is a real operational constraint, not a theoretical caveat. Running the tool on a home connection can therefore create a false impression of a network fault if a router, firewall, or upstream provider is reacting to connection volume.

The README specifically identifies `max_concurrent_checks` as the setting to lower when a run destabilizes the network. The documentation also recommends considering a VPN. That recommendation does not make the traffic anonymous, legal, or harmless, and it does not change what the tool does. It simply addresses the network location from which the connection storm is sent. The project is not suitable as a low-volume foreground utility when left unattended.

Proxy reliability is also time-sensitive. The validation target is controlled by `check_url`, and a successful full fetch proves only that the candidate worked during that check. It does not prove that the proxy belongs to the advertised organization, supports a particular site, preserves privacy, or will remain online. Treat output as a short-lived inventory. Re-running the checker is part of normal use, especially when endpoints disappear quickly.

License, maintenance, and operational trade-offs

The project is distributed under the MIT license, and `Cargo.toml` declares `license = "MIT"`. The license permits reuse and modification under its terms, but it does not grant rights to proxy owners, websites checked through a proxy, or networks that receive the traffic. Users remain responsible for complying with applicable terms and laws. No legal advice is needed to state the practical boundary: the software license covers the checker, not the behavior or ownership of discovered proxies.

The repository is not archived. Its last push was on 2026-10-15, and the listed `build-3788` release was also published on 2026-10-15. Those facts indicate recent project activity, but they do not establish a compatibility promise for every future release. The README says pre-built archives come from the latest commit on `main` through CI, so a release artifact may represent a moving branch build rather than a separately versioned package.

The upgrade cost is primarily operational. A new binary may change configuration expectations, output shape, platform coverage, or MaxMind database behavior, so the `config.toml` and existing automation should be checked after each upgrade. The repository uses exact dependency versions in the visible `Cargo.toml` entries, while the package version is `0.1.0` and `publish = false`. That suggests a binary-oriented project rather than a conventional published library.

A practical alternative: consume the maintained proxy list

The README points to `monosans/proxy-list` when users want proxies without running this checker. That is a different workflow. `proxy-list` supplies a maintained list, while proxy-scraper-checker performs local collection, validation, deduplication, response-time measurement, and metadata generation from sources configured on the user's machine. The related list can save local network load and setup time, but it gives the user less control over which sources are queried and how aggressively candidates are tested.

Another reasonable alternative is to write a small application around a single known proxy source and use `reqwest` directly. That avoids broad scraping and hundreds of simultaneous checks, but it does not provide this project's raw-text, HTML, JSON, CIDR, protocol, TUI, and offline MaxMind pipeline. The choice is therefore between a prepared external list, a narrow custom checker, and proxy-scraper-checker for broad local discovery.

Editorial conclusion

Adopt proxy-scraper-checker when you need a configurable Rust utility that turns mixed proxy sources into tested lists, especially if offline MaxMind data and a single executable are priorities. Do not treat it as a proxy provider or assume that a successful check guarantees stable service. Verify first that your sources are trustworthy, your network can tolerate the configured concurrency, and the `check_url` and output paths match your intended use.

Frequently asked questions

Official sources

1. Issues
2. License: MIT
3. monosans/proxy-scraper-checker on GitHub
4. README
5. Releases