Maelstrom, a test runner that puts every test in its own container

The five reasons the project gives

Maelstrom is a suite of tools for running tests in isolated micro-containers, locally on your machine or distributed across arbitrarily large clusters. There are test runners for Rust, Go and Python, with more on the way.

The README lists five reasons to use it, and they are worth reading in order because they describe a progression from convenience to a change in model.

The first is that it is easy. Maelstrom provides drop-in replacements for cargo test, go test and pytest, and in most cases it just works with your existing tests with minimal configuration.

The second is reliability. Every test is run isolated in its own lightweight container, which eliminates the confusing errors caused by inter-test or implicit test-environment dependencies. This is the reason the project exists, and it is the one worth thinking about longest, because a test that passes on your machine and fails in CI is very often a test that depends on a file left behind by the previous test, on a port, on an environment variable, or on a service that happened to be running.

The third is scalability. Maelstrom can be run as a cluster, and adding worker machines increases test throughput linearly.

The fourth is cleanliness, and it is the most unusual claim. Maelstrom has built a rootless container implementation from scratch, in Rust, not relying on Docker or runc, and optimised it for low overhead and fast startup.

The fifth is speed, and the README is careful about it. In most cases Maelstrom is faster than cargo test or go test even without clustering, but its test-per-process model is inherently slower than pytest's shared-process model. Maelstrom provides test isolation at a low performance cost. That distinction is stated plainly rather than hidden, and it is the key thing to weigh against the reliability argument.

The project is Linux only, on x86 and ARM, because it relies on namespaces to implement containers. The licence is Apache-2.0, and the README also carries a badge for the MIT licence, which is worth checking against the LICENSE files in the repository if you need to know which applies to which component.

Installing the runners

To run your tests with Maelstrom you need a test runner binary, and the recommended way to get one is cargo-binstall, which downloads a pre-built binary rather than compiling from source.

For Rust tests:

cargo binstall cargo-maelstrom

For Go tests:

cargo binstall maelstrom-go-test

For Python tests:

cargo binstall maelstrom-pytest

Each of those installs a pre-built binary from the GitHub releases page, which is the fastest route and avoids needing a Rust toolchain at all.

There is a distribution package path as well. On Arch Linux:

pacman -S cargo-maelstrom
pacman -S maelstrom-go-test
pacman -S maelstrom-pytest

And the README notes that if you do not have cargo-binstall you can download the binaries manually, pointing at the installation page in the book for the other routes.

The binaries are published to crates.io as cargo-maelstrom, and the project has a Discord community and a book at maelstrom-software.com with the full documentation, including separate pages for the installation and for each runner's configuration specification.

One thing to note is that the three runners are separate binaries rather than one with a language flag. That makes sense given the cargo subcommand convention for the Rust one, and it means the Go and Python runners can be installed on a machine that has no Rust at all, which is the point of shipping pre-built binaries.

Running the Rust and Go runners

The Rust runner is used as a cargo subcommand:

cargo maelstrom

This runs in standalone mode, meaning all the tests run locally. Each test is run in its own container, configured with a few common dependencies, and the README says it may work for your project without any further configuration.

The Go runner is a standalone binary with the same shape of behaviour:

maelstrom-go-test

Also standalone mode, also one container per test, also configured with a few common dependencies, and also described as likely to work without further configuration.

Then there is the paragraph in both sections that tells you what to do when it does not work, and it is the most practically useful sentence in the README. If some tests fail, it likely means those tests have dependencies on their execution environment that are not packaged in their containers. You can remedy this by adding directives to the configuration file.

For Rust, the file is cargo-maelstrom.toml, and the init command creates it:

cargo maelstrom --init

For Go, the file is maelstrom-go-test.toml and the command is:

maelstrom-go-test --init

Each of those links to the corresponding specification page in the book, which is where the directive syntax is documented.

The workflow that follows from this is the one to plan for. Running the tool for the first time on an existing suite will produce failures, and those failures are not noise. They are the tool telling you exactly which tests had undeclared dependencies on the environment, which is the information you wanted in the first place. Each failure becomes a directive in the configuration file, and the configuration file is the record of what your tests actually need.

That is a different shape of work from adding a test framework. It is closer to writing down the environment contract your suite has always had implicitly, and it is more work up front and less afterwards.

The Python runner and its image choice

The Python runner needs a little more setup, and the README explains why in two steps.

First you generate the configuration file:

maelstrom-pytest --init

Then you update the image in the file to have the version of Python you want, and the example given is:

[[directives]]
image = "docker://python:3.11-slim"

The default configuration and the example both use an image from Docker Hub, which is a detail worth pausing on for a tool whose selling point is a container implementation that does not rely on Docker. The distinction is that Maelstrom does not require a Docker daemon to run the containers, but the image you point it at can still be a Docker image reference, which is the common way images are distributed.

The second step is making sure your project's own Python is included, because the tests have to be able to import your code from inside the container. You update the added_layers setting in the file to make sure it includes your project's Python.

The general shape of the configuration file is worth understanding, because it is the same shape across all three runners. It is a list of directives, and each directive carries an image and a set of added layers. The image is what the container starts from, and the added layers are the extra files from your machine that get copied in so the test can see your code and its dependencies.

For Python that means a directory of your project plus whatever your dependencies live in, whether they are in a virtual environment or installed system-wide. For Rust it means the compiled test binaries and the shared libraries they need. For Go it means the built test binary. The reason the layers are explicit is the isolation model: a container with only the base image knows nothing about your machine, so everything a test needs beyond the base has to be declared.

Beyond testing and how the clustering works

The README makes a point that matters for anyone considering the project for something other than testing. While the focus so far has been on running tests, Maelstrom's underlying job execution system is general-purpose. There is a command line utility to run arbitrary commands, a gRPC-based API, and Rust bindings for programmatic access and control.

That reframes the project. It is not a test runner with a container layer attached; it is a job execution system with test runners built on top of it, and the ability to distribute work across a cluster is a property of the system rather than a feature of the test runner.

The scalability model is the straightforward one. You add worker machines and test throughput increases linearly, which is the property that makes the per-test container cost acceptable in a large suite. The parallelism within one machine comes from the containers being lightweight, which is where the rootless Rust implementation earns its place. A container that starts in a few milliseconds and does not need a daemon is what makes running one per test viable at all, and the README's claim that Maelstrom is usually faster than the native runners despite the per-test overhead is a consequence of that start-up cost being small.

The trade-off, stated in the README, is that the per-test isolation is a different model from what pytest does. Pytest shares a process across the tests in a file, and that is fast but means a test that mutates global state affects its neighbours. Maelstrom gives up the shared process and gets determinism back. For a suite with a small number of tests that do not interfere, that is a bad trade and the native runner is faster. For a suite where a tenth of the tests fail intermittently for reasons nobody can find, it is the only trade worth making.

The book is the place to read more, with pages for installation, for the cargo subcommand specification, for the Go runner specification and for the Docker deployment, and the project also runs a short survey asking how people want to use it, which is an invitation to say what the missing language support should be.

Editorial conclusion

Adopt Maelstrom if your test suite has flaky failures caused by shared state or by a test depending on the machine it runs on, and you can accept the per-test container cost. Skip it if your suite is already isolated and fast, since Maelstrom's own README admits its test-per-process model is slower than pytest's shared-process one. Verify first by running it in standalone mode with --init to generate the configuration file, then by adding the dependencies your tests actually need before you migrate a whole suite.

Frequently asked questions

Official sources

1. License: Apache-2.0
2. maelstrom-software/maelstrom on GitHub
3. Project website
4. README
5. Releases