Crater builds the crates.io ecosystem twice and diffs the results

Two compiler versions, one corpus, and a diff of the results

Crater is a tool to run experiments across parts of the Rust ecosystem. Its primary purpose is to detect regressions in the Rust compiler, and it does that by building a large number of crates, running their test suites and comparing the results between two versions of the Rust compiler.

The mechanism is a differential test at ecosystem scale. A change to the compiler, to the standard library or to a build related behaviour produces a report: this crate built before and does not build now, that crate compiled and its test suite now fails, this test output changed. The unit of evidence is a real project from the wild rather than a synthetic case, which is what makes it useful for catching the changes that no targeted test would have thought to write.

The current feature list names the steps. Crater discovers Rust codebases on crates.io and on GitHub, executes experiments on custom Rust toolchains, runs cargo build and cargo test over all the discovered codebases, can build and test without dependency updates or network access, can run arbitrary tests over all the discovered codebases, generates HTML reports with results and logs, and isolates tests in Docker containers.

That capability list is aimed at a specific user: the people who change rustc and need evidence at scale before a release. It is also usable by anyone maintaining a widely used crate, who wants to know what their change does to every other project that depends on them, and that second use is the one most outsiders never consider.

Crater runs untrusted code, and the README says so in capitals

The README carries a warning in capitals: do not run Crater in an unsandboxed environment, because Crater executes malicious code that will destroy what you love.

That sentence is the most important operational fact about the project. The corpus is crates published by strangers, and the thing Crater does with each one is compile it and run its test suite. A build script can do anything, and a test suite can do anything. Crater is therefore a tool that must run inside a boundary, and the boundary is the point of the Docker isolation feature in the feature list.

This is a real risk rather than a theoretical one, and it is worth being explicit about the shape of it. The code being run is not selected by you and is not reviewed by you. Even a crate that looks benign, a small utility that happens to be a dependency of everything, can contain a build script that writes outside its directory. A sandboxed run confines that to a container you throw away afterwards.

The same reasoning applies to the network. One of the listed capabilities is to build and test without dependency updates or network access, which is both a reproducibility measure and a containment measure: a build script that tries to phone home during the experiment has nothing to phone home to.

The result is a tool that only makes sense inside a properly isolated machine, and the documentation reflects that with a separate page for setting up a new Crater agent machine and a specification for the agent HTTP API.

Discovery is crates.io and GitHub, and the corpus is built twice

The discovery step is the first thing Crater does, and it is deliberately broad. It finds Rust codebases on crates.io and on GitHub, which means a corpus that includes published libraries and projects that are not published at all.

The execution step then runs cargo build and cargo test over every discovered codebase against the toolchain under test. Because the same corpus is built with both compilers, the comparison is apples to apples, and the report is a per project list of what changed.

Two design options make the work tractable. Crater can build and test without dependency updates, which pins the dependency versions so that a failure is attributable to the compiler change rather than to a new release of some transitive crate. And it can build without network access, which removes both the network as a confounder and the network as an escape route for the code under test.

There is also a mode for running arbitrary tests over all the discovered codebases, which is what makes Crater usable as a general experiment harness rather than only as a compiler comparison tool. The feature list does not describe what such a test looks like in practice, so a reader who wants that mode will be reading the CLI documentation rather than the README.

Output is an HTML report with results and logs, and the documentation includes a report triage procedure, which tells you something about the intended workflow: Crater produces a large volume of findings and a human decides which matter.

Local mode needs Docker, cloud mode needs an agent fleet

Crater can operate locally, with Docker as the only dependency, or distributed on the cloud. That choice shapes the work more than any other.

Local mode is the entry point. The repository ships a Dockerfile that builds the release binary and then copies it into a second, empty image, and the feature list says tests are isolated in Docker containers. The result is that a developer can reproduce a run on a laptop, and the only external requirement is Docker itself.

Distributed mode is what the project uses at scale. The operations documentation includes a page on setting up a new Crater agent machine and a technical specification for the agent HTTP API, which is the contract between the coordinating side and the machines that do the building. The Rust dependency list makes the shape visible: aws-sdk-s3 and aws-config for object storage, warp for the HTTP server, prometheus for metrics, and rusqlite with the bundled feature for local state, plus tokio for the async runtime and rayon for parallel work.

The documentation is split along the same lines. User documentation covers local and CLI usage, the GitHub bot usage and the report triage procedure. Operations documentation covers the legacy operational workflow and agent machine setup. Technical documentation covers minicrater, a smaller test harness, and the agent HTTP API.

The GitHub bot usage page is worth noting for anyone evaluating the tool: a bot that reacts to a pull request by running a large experiment is a different commitment from running Crater by hand.

The container build pins a nightly toolchain and defeats cargo mtime caching

The Dockerfile is where the practical details of building Crater itself become visible, and there are two of them that anyone reproducing the build will hit.

The first is the toolchain. The build image is ubuntu:24.04 with build-essential, git, cmake, pkg-config, libsqlite3-dev and libssl-dev, and the toolchain is installed by downloading rustup-init and running it with the nightly toolchain and the minimal profile. Crater is a tool about toolchains, and it pins its own to nightly so it can build against current compiler internals.

The second is the dependency caching trick. The Cargo.lock and Cargo.toml are copied into the image along with the .cargo directory, and a dummy main.rs and build.rs are written, so that cargo fetch and cargo build --release populate the dependency layer. The real source is copied in a later step, and then every .rs file is touched with find, because, as the comment says, without it cargo assumes the source did not change thanks to mtime weirdness.

That is the kind of detail that is invisible until you build the project and watch cargo skip a rebuild it should have done. It is also a fair indicator of the care taken in this repository.

The second stage of the image is minimal, ubuntu:2 as the base with the binary copied in, and the project ships templates/ and assets/ directories that are copied into the build, which is where the HTML report generation finds its templates.

The dependency list is a readable map of the architecture

Crater's Cargo.toml is worth reading as documentation, because it names the subsystems without explaining them.

For finding and fetching code, crates-index at 3.11.0 gives access to the crates.io index, cargo_metadata at 0.23.1 reads manifests, cargo-util-schemas parses the schema of a manifest, and docsrs-metadata is pulled directly from the rust-lang docs.rs repository as a git dependency, which is how the tool knows the documentation metadata of a published crate. rustwide, with the unstable and unstable-toolchain-ci features, is the Rust team tool for running commands across many toolchains, and it is the component that makes the two compiler comparison work.

For running the experiments, rayon provides the parallelism, tokio the async runtime, and the nix crate with mman, resource and fs features is used for the process and memory handling that sandboxed builds need.

For reporting, tera is the template engine, minifier is used with its html feature, and the MIME and percent-encoding crates point to handling of the files and URLs involved in a corpus.

For the state around the experiments, rusqlite with the bundled feature, zstd and rawzip for archives, aws-sdk-s3 for object storage, ctrlc for signal handling, env_logger and log for output, and the prometheus crate for metrics. The presence of docsrs-metadata and rust_team_data as git dependencies into other rust-lang repositories is a small but telling detail: this tool is built by the Rust project, for the Rust project.

A successor with two names, Linux only, and a licence question

Crater has a lineage that the README records. It is a successor to taskcluster-crater, and it was subsequently named cargobomb before resuming the Crater name. The renaming history explains why you may find older references under a different name, and the find-bad-crates.py script and local-crates/ directory in the tree suggest the corpus selection is still an active part of the work.

The platform constraint is stated directly: it only works on Linux at the moment. That is not arbitrary. A corpus build has to be reproducible across a fleet, and the Linux container image is what makes the sandboxing and the toolchain pinning consistent, so Windows and macOS hosts are out of scope by construction.

On licensing, the README says Crater is licensed under both the MIT and Apache 2.0 licenses, which is the standard dual licence that lets a downstream user pick either. No licence identifier is recorded for the repository itself, so the file in the tree is the place to check the exact terms if your use is commercial.

Maintenance is steady rather than rapid. The last push was on 2026-07-25 and the repository is not archived, but it publishes no GitHub releases, and the package version in Cargo.toml is still 0.1.0. A tool that runs for hours over thousands of crates is normally consumed as a pinned checkout or a built container rather than as a versioned release, and the documentation for setting up an agent machine is written on that assumption.

Editorial conclusion

Crater fits a compiler developer or a standard library maintainer who needs to know whether a change to the toolchain breaks real crates in the wild, and who has a Linux machine and the disk and time budget to build a large corpus twice. It does not fit an application team that wants faster CI, and it is the wrong tool for anyone without a sandbox, because the README states plainly that Crater executes malicious code that will destroy what you love. Before you run it, read docs/cli-usage.md, put the run inside Docker, decide whether you will use the local mode or an agent machine per docs/agent-machine-setup.md, and check the licence terms in the repository, since the README describes the project as licensed under both MIT and Apache 2.0 while no licence identifier is recorded for the repository itself.

Frequently asked questions

Official sources

1. Issues
2. Project website
3. README
4. rust-lang/crater on GitHub