siteone-crawler is one Rust binary that crawls, audits, clones and gates

Ten tools in one binary, after a rewrite

The project describes itself as a website analyzer, cloner and converter for developers looking for security and performance insight, for SEO specialists looking for optimisation opportunities, and for site owners who want reliable backups and offline versions. The package description in the manifest is blunter about the scope, one dependency free binary for all major platforms, ten tools in one. The most consequential fact in the readme is the rewrite, from PHP with Swoole to Rust, which the project says produced twenty five percent faster execution and thirty percent lower memory consumption while producing identical output. That last clause is the important one for anyone comparing versions, since the claim is that the behaviour did not change. The platform story is a single native binary for Windows, macOS and Linux on both x64 and arm64 with no runtime dependencies, which is what makes it usable in a container image without a language runtime layered on top.

Three ways to drive the same engine

The readme describes three front ends over one engine, and the choice is about who is using it. Running the binary with no arguments opens an interactive wizard with eleven preset modes, which is how somebody who has never seen the tool gets a working audit in a minute. The command line interface is the extensive one, and it is the mode the release artefacts and the videos demonstrate, with a separate link for the video walkthrough. And there is a desktop GUI application for visual control, which lives in its own repository rather than in this one. That separation is worth noticing, because it means the GUI and the CLI can move at different speeds, and it keeps the Rust binary in this repository free of a GUI toolkit dependency. The documentation also links a project website, a configuration section listing the command line options, and separate JSON and text output specifications.

The analyzer looks at pages and at the site

Two levels of analysis come out of one crawl. At the URL level the crawler reports the facts you would collect by hand, the status code, the response time, the size, the custom headers and the titles. At the site level the analyzer reports strange or error behaviour and useful statistics, the 404s, the redirects, the bad practices, the SEO and the security issues, and the heading structures. The performance angle is the one that usually pays for the setup time, since the same crawl that finds a broken link also tells you which pages are slow, and the tool exposes specialised analyzers for the fastest and the slowest URLs, an SEO and OpenGraph analyzer, and a built-in HTTP server for previewing results locally. There is also a DevOps mode that turns the crawler into a load testing tool, with configurable concurrent workers and a request rate limit, and that combination is why the disclaimer at the end of the readme exists.

The quality gate and the cache warmer

The CI/CD story is two features in one, and they are unrelated. The first is the quality gate, enabled with the `--ci` flag, with configurable thresholds, and a failure exits with code 10, which is chosen so that a deployment pipeline can block on it. The point of the exit code is that it is distinct from the generic failure codes, so a pipeline can tell a failed audit from a crashed tool. The second is cache warming, where crawling the whole site after a deployment populates the reverse proxy or CDN cache, which turns the crawler from a reporting tool into a deployment step. Both are the kind of thing that justifies a single binary, because the thing that runs in the pipeline and the thing that produces the report for a human are the same executable. The report itself comes in three shapes, an interactive HTML audit report with sortable tables and a quality score from 0.0 to 10.0, a detailed JSON for programmatic consumption, and coloured text for a terminal, and a built-in SMTP mailer can send the HTML report straight to an inbox.

Offline clones and markdown export

The two features that most distinguish this from a general crawler are the offline website generator and the markdown converter. The generator clones an entire site to browsable local HTML files, assets included, so you can read the documentation without a server, and it supports multi-domain clones, including subdomains or external domains, with intelligent cross-linking so the links still work when the site is opened from disk. The converter exports a whole site as browsable text markdown, or as a single file, with smart header and footer deduplication, and the readme is explicit about the use case, feeding the content to AI tools. There is a dedicated repository of markdown examples for a site, and a built-in web server in the argument list. The rest of the surface is a sitemap generator and an email summary of what was found, and the deep crawl is described as covering every page and asset rather than stopping at a link frontier.

The dependency list is a map of the tool

The manifest shows no runtime dependencies, in the sense of a language runtime, but the library list is long and each entry tells you what the tool actually does. The asynchronous core is tokio with the full feature set, and the HTTP client is reqwest with gzip, brotli and deflate decoding, rustls instead of the system TLS stack, SOCKS support, cookie handling, streaming, a blocking mode and multipart uploads, which is more than a crawler needs and is what the load testing, the uploads and the SMTP mailer each require. HTML parsing is the `scraper` crate, the text processing is `regex`, the command line is clap, and the data structures are serde with serde_json. The TLS stack is rustls with the ring provider, and there is a native certificate loader, so the tool verifies real certificates on a real machine. The mailer is lettre over SMTP, the compression is flate2 and brotli, and the markdown output uses pulldown-cmark, which is the last detail that tells you the markdown export is generated rather than assembled as text.

State of the project and the limits of the readme

The manifest says version 2.6.1 on edition 2024 with a minimum Rust version of 1.94, and the author is Ján Regeš, while the release list shows v2.5.1, v2.5.0 and v2.4.1 in June 2026, so the tree is ahead of the last tag. The last push to the repository was on 2026-09-27. The tree carries a `CLAUDE.md`, a `prompts/` directory, a `rust-toolchain.toml` and a `rustfmt.toml`, which is the usual sign of a project that has settled on its toolchain. Two limits are worth naming. The first is that the documentation is long and partly self-referential, with a table of contents that runs to several dozen argument groups, so the configuration reference rather than the readme is what you will actually use. The second is the one the project states itself, in a disclaimer section, and it is about crawling: the tool sends real requests, it can be configured to send a lot of them, and pointing it at a site you do not own is your responsibility rather than the tool's.

Editorial conclusion

siteone-crawler suits a team that wants one tool to run in a pull request, to produce an HTML report a non engineer can read, to keep an offline copy of a documentation site, and to dump a site into markdown for a model or a text editor, since those are four jobs that usually mean four dependencies. It is a poor fit if you want a library to embed in your own crawler, because it is an application rather than a library, and it is a poor fit if your site is behind authentication, since the crawler fetches what a server returns unless its optional browser rendering is enabled. Before you point it at anything, check three things: that you are allowed to crawl the target, since the tool sends real traffic and can be configured to send a lot of it, which is a matter of permission rather than of tooling, what the quality gate thresholds should be, since a failing gate exits with code 10 and can block a deployment, and the output you want in CI, the JSON specification being the one to read if you are going to parse it.

Frequently asked questions

Official sources

1. janreges/siteone-crawler on GitHub
2. License: MIT
3. Project website
4. README
5. Releases