Hermit deterministic Linux execution for reproducible testing

Hermit's controlled execution boundary

Hermit addresses a specific class of testing problem: Linux programs often receive implicit inputs that change between executions. Thread scheduling can expose different interleavings, clocks can advance at different times, random data can differ, and file metadata can change without the program's source code changing. Hermit places an x86-64 Linux guest under a special execution environment that observes and controls these sources of nondeterminism.

The intended users are engineers working on flaky tests, concurrent software, replay and debugging systems, and artifact generation. The README describes repeatable execution, controlled concurrency testing, record and replay experiments, and diagnosis of schedule-sensitive failures as use cases. Hermit does not change the guest program before execution. Its value comes from controlling the environment around that unmodified program.

The project is tied closely to Linux host facilities. The documented requirements include user, PID, and mount namespaces, parent-child ptrace, seccomp filters, libunwind, LZMA development packages, and user-space performance counters for precise scheduler preemption. Performance counters are optional according to the README, but their absence reduces preemption points for CPU-bound workloads. That distinction matters when deciding whether Hermit is suitable for a particular test suite.

How ptrace controls Linux nondeterminism

The default execution path uses the Reverie ptrace backend. Based on the repository layout and README, Hermit launches the guest beneath a controlling runtime that intercepts system calls and relevant execution events. The runtime can regulate thread scheduling, time, random data, CPUID results, and selected file metadata.

The `detcore` crates provide the core model and execution machinery, while `hermit-cli` supplies the command-line interface. `hermit-verify` handles comparison and verification-related logic. The repository also contains specialized backend directories, including `detcore-dbt`, `detcore-sabre`, and `liteinst-runtime-build`. This separation indicates that backend mechanics are not all implemented in the CLI layer.

Hermit uses a general command-oriented interface. A program is supplied after `run --`, so the host can distinguish guest arguments from Hermit options. The `--` separator is recommended because it prevents guest arguments that begin with `-` from being interpreted as Hermit options. Programs may create threads and child processes through `clone`, `clone3`, and `fork`, with those operations handled by the ordinary ptrace lifecycle in the LiteInst description.

Determinism is not created by every Hermit option. For clock-reading programs, the README specifically says that the starting host time is captured on each invocation unless `--epoch` is fixed. Randomness and metadata are controlled by the execution environment, but the README does not claim that every host interaction is automatically deterministic.

Building Hermit on Linux with Reverie

Hermit documents installation from its maintained source repository. Rust nightly is required through `rustup`, and `rust-toolchain.toml` selects the repository toolchain automatically. On Debian or Ubuntu, install the native development dependencies first:

sudo apt-get update
sudo apt-get install -y libunwind-dev liblzma-dev

The README then gives this source installation sequence:

git clone --recurse-submodules https://github.com/rrnewton/hermit
cd hermit
cargo install --locked --path hermit-cli
hermit --version

The `--recurse-submodules` option is significant because the project builds against Reverie and also uses workspace submodules. An existing clone can initialize them with:

git submodule update --init --recursive

For a local build without installing into Cargo's binary directory, use:

cargo build --workspace
./target/debug/hermit --version

An optimized workspace build stages backend runtime resources under `target/install_pkg/`, including SaBRe and e9patch rewriters, Detcore libraries, and DynamoRIO files. The README notes that staging resources does not enable optional backends in the CLI. The staged `hermit` entry is a symlink, so a standalone copy may require dereferencing it with a command such as `cp -aL target/install_pkg/ DESTINATION`.

Running a deterministic guest command

The first practical use is to place Hermit before a Linux command. The following example prints `hello` and returns the guest's exit status:

hermit run -- /bin/echo hello

This is the default `ptrace` path. Hermit's current defaults are strict and deterministic. `--strict` remains available as an explicit compatibility spelling, but the README says it does not enable a stronger mode:

hermit run --strict -- /bin/echo hello

The flag name can therefore be used in scripts when explicit intent is preferable, although it does not change the documented default policy.

Programs that read the clock require an additional decision. Hermit captures the current host time by default, so a fixed starting time is needed for repeatability across invocations. The README points to the `examples/README.md` date example for this case. The repository's `examples/` directory includes scripts for dates, random data, races, and timed progress bars, which indicates the intended way to explore these behaviors is through small guest programs.

The default backend is the sensible starting point because the README calls it the most thoroughly tested and says it needs no special setup. Specialized backends should be selected only after the ordinary command path works.

Backend selection and experimental boundaries

Hermit exposes several low-level execution backends through the global option `--backend=ptrace|dbt|liteinst|sabre|kvm|e9patch`. The backend must appear before the subcommand. Omitting it selects `ptrace`. Every documented backend supports `run`, while KVM additionally supports `analyze` and `bisect`, e9patch supports `record`, and SaBRe supports `strace`.

The CLI fails closed when a requested backend or command combination is unsupported. Hermit does not silently fall back to ptrace after an explicit backend request. That behavior is useful for automation because a failed configuration is less likely to produce results from an unintended execution mechanism.

LiteInst is described as an experimental ptrace-hosted hybrid for dynamically linked Linux x86-64 guests. It requires the optional `liteinst` CLI feature, included by `third-party-backends`. The README gives a staging command that uses the `liteinst-runtime-build` path:

./scripts/stage-liteinst-runtime.sh dev \
  "$PWD/target/debug/libreverie_liteinst.so" \
  "$PWD/target/liteinst-runtime-build"

The executable build and run commands are documented separately:

cargo build --locked -p hermit --features liteinst --bin hermit
./target/debug/hermit --backend=liteinst run --strict --verify -- /bin/echo hello

The LiteInst section also documents unsupported cases: `vfork` and `exec` from a thread other than the process leader remain unsupported. The backend's verification behavior is diagnostic, not strict determinism. Strict verification requires `--verify-strict --verify-json REPORT.json`, `bitwise_parity: true`, and nonzero compared-message counts.

Maintenance, licensing, and upgrade cost

The repository is not archived, and its last push was on 2026-10-15. That is a current maintenance signal under the repository's published status, but it does not by itself establish release maturity. The repository has no GitHub releases, so users should not expect a published release sequence from the repository metadata.

The license information needs direct review. The repository description classifies the license as other because of a custom `LICENSE` file, while `Cargo.toml` declares `BSD-3-Clause` for the workspace package. This difference means an engineer must inspect `LICENSE` and the relevant package metadata before redistribution or embedding. The available repository information does not support a legal conclusion about obligations.

Hermit also creates upgrade work beyond the Rust crate itself. Reverie is pinned through a Cargo git dependency and a workspace submodule. Updates therefore require attention to submodule revisions, backend runtime resources, optional feature flags, and native dependencies. The default developer build deliberately excludes third-party dependencies associated with DBT, SaBRe, and LiteInst. Enabling those backends changes the build scope and the set of resources that must be staged.

Editorial conclusion

Adopt Hermit when your Linux x86-64 tests depend on repeatable schedules, controlled time, or reproducible random and metadata behavior. Do not choose it for unsupported architectures, non-Linux hosts, or workloads that require unrestricted host integration without checking backend support first. Verify the selected backend, required namespace and ptrace capabilities, resource staging, and the repository LICENSE before incorporating it into a production toolchain.

Frequently asked questions

Official sources

1. facebookexperimental/hermit on GitHub
2. Issues
3. README