# turmoil: a deterministic simulator that injects latency, partitions and crashes into your tests > Turmoil runs several concurrent hosts inside a single thread and injects hardship, meaning latency, drops, partitions, crashes and torn writes, into a simulated network and filesystem, under manual control or a seeded RNG. Its selling point is determinism, and its cost is that everything your code touches has to be one of the simulated pieces. **tokio-rs/turmoil** — Add hardship to your tests - Repository: https://github.com/tokio-rs/turmoil - Stars: 1,297 · Forks: 79 - Language: Rust - License: MIT - Published: 2026-10-19 · Updated: 2026-10-19 · Language: en - Canonical page: https://hysenlabs.com/projects/tokio-rs-turmoil ## One thread, several hosts, and a clock you control Turmoil is a family of crates for deterministic simulation testing of distributed systems. It runs multiple concurrent hosts within a single thread, and injects hardship into the simulated network and filesystem, under manual control or a seeded RNG. That sentence contains the whole design, and it is worth taking apart in order, because each clause rules something out. Running multiple concurrent hosts within a single thread is what makes the simulation deterministic. If each host were a real process on a real scheduler, the order in which their events interleaved would vary from run to run, and a test that reproduces a race would reproduce it once. With everything in one thread and one simulated clock, the interleaving is a function of the fault schedule and the seed, so a failure is a fixture you can commit. The second clause is the inject list. Latency, drops, partitions, crashes, torn writes. Those are not abstract failures, they are the specific ways a distributed system breaks: a request that takes too long and times out, a packet that never arrives, a split brain where half the nodes cannot reach the other half, a process that dies mid-message, and a write acknowledged by the server but lost before it reaches the disk. A test suite that can produce those on demand is testing something different from a suite that hopes for them. The last clause is the control surface. Under manual control means the test says when the latency happens and how long it lasts. Under a seeded RNG means the fault pattern is random but reproducible from the seed. Both are useful, and the difference matters: manual control is how you write a regression test for a specific bug you already found, and the seeded RNG is how you run a thousand random schedules looking for the one that breaks something. That is the value proposition, and it is narrow and sharp. Turmoil is not a general chaos testing tool and it is not a load generator. It is a way to write the failure you were worried about as code, and have it happen every time. ## Four crates, and two of them are drop-in replacements The crate table is short and the descriptions tell you how to enter the system. The turmoil crate is the full simulation testing framework, and the table says to start here. It is the orchestrator, the one that owns the hosts, the clock and the fault injection. turmoil-net is a deterministic simulated socket stack, described as a drop-in replacement for tokio::net. That phrase is doing a lot of work. It means your code changes its imports rather than its structure, which is what makes the approach usable on an existing codebase. If a service already speaks TCP over tokio::net, you can point it at the simulated stack and its traffic becomes controllable. turmoil-fs is the same idea for storage, a deterministic simulated filesystem described as a drop-in replacement for std::fs and tokio::fs. This is the crate that gives you torn writes, because a filesystem that can be told to acknowledge a write and then not persist it is the only way to test the code path that handles a partially durable state. The fourth is turmoil-io-uring, a deterministic simulated io_uring, with an optional fs feature that wires it to turmoil-fs. This one is the narrowest and the most recent, and the wiring detail is the interesting part: it exists so that a program using io_uring for its file IO can be simulated too, rather than being the one part of the stack that escapes the simulation. The split into four crates rather than one is the architectural decision. A library that wants to simulate everything for the convenience of its users will be a dependency of the whole binary. A library that ships the socket stack and the filesystem separately lets a team adopt only the layer they need to simulate, and lets a team that has its own mock filesystem keep it. The workspace manifest makes the layout explicit, with members of crates/* and examples/*, so the examples are built alongside the crates in the same workspace and cannot drift from them. ## The examples, which show what kind of system this is for The examples directory at the top of the repository is the best evidence of what the tool is aimed at, because the directory names are the workloads. There is an axum example, a cluster example, a grpc example, and three UDP examples, udp_vpv4_broadcast, udp_vpv4_multicast and udp_vpv6_multicast. The axum example places the tool alongside a service, which is the common case: an application where you want to test what happens when the client is slow or the database is unreachable. The grpc example does the same for a binary protocol. The cluster example is the multi host case, several services talking to each other over the simulated network, which is where partitions matter most. The three UDP examples are the most interesting for a different reason: broadcast and multicast are exactly the traffic patterns where a partition is hard to reason about, because one host's view of the network is not another's, and a simulator that keeps every host on one thread can make that inconsistency explicit and repeatable. The README points at examples/ for end-to-end samples, and at each crate's tests/ directory for smaller, feature-focused tests. That split is the right one for a tool like this. An end-to-end example is where you see several hosts, a fault schedule and an assertion about the outcome, and a focused test is where you check that a particular simulated socket behaves in a particular way. The examples are also workspace members, so they are built by the same cargo commands as the crates. That means an example that stops compiling is a build failure rather than a documentation drift, which is a meaningful difference in a repository that is explicitly trying to be a reference for how to use the framework. ## Getting it, and what the README does not tell you The crates are published on crates.io and the documentation is on docs.rs, and the README links both for each of the four crates. The turmoil crate is marked as the one to start with, and the others are linked with their own badges. What the README does not do is give you an installation snippet or a first example, and it does not describe the API surface at all. The crate descriptions are one line each, and the whole of the usage documentation is delegated to the documentation site and the examples directory. That is a deliberate choice for a project of this kind, since a deterministic simulator has a conceptual model that has to be explained before any function signature means anything. You need to understand what a host is, how the simulated clock advances, how a fault is scheduled, and how a test asserts on the outcome, and none of that is compressible into a README section. It does mean you should budget for the reading. The practical sequence is to add the turmoil crate, put your existing service behind it, replace the network and filesystem layers with the simulated ones, and then write a test that injects the failure you are worried about. The one thing the README does tell you about contribution is straightforward. Unless you explicitly state otherwise, any contribution intentionally submitted for inclusion in Turmoil by you shall be licensed as MIT, without any additional terms or conditions. For a project under the tokio-rs organisation that is the expected arrangement, and it means the licence terms of the whole suite are uniform, which matters when you are reviewing a dependency chain that includes all four crates. ## Where the simulation stops, and why that is the real constraint The honest limitation is the coverage boundary, and it follows directly from the crate list. turmoil-net replaces tokio::net, and turmoil-fs replaces std::fs and tokio::fs. Everything your service does outside those two surfaces is not simulated. If your service opens a PostgreSQL connection, talks to a message broker over a real client library, calls an HTTP API, reads a file through a crate that does not use std::fs, or touches a clock directly rather than through tokio, none of that is under the simulator's control, and a fault you thought you were injecting may not reach the code that matters. The io_uring crate is a good illustration of the direction of travel. Rather than leave one specific IO path outside the simulation, they added a crate to bring it inside. The pattern suggests that coverage is something the project grows over time, and each new crate is an admission that something was previously outside. There is a second, subtler limit, which is the one the deterministic design implies. Turmoil controls the clock and the network, so it can reproduce an ordering of events. It cannot reproduce the timing of a real machine under real load, so a test that passes under Turmoil tells you the code handles that ordering, not that it handles that ordering under contention with other work on the host. The value is in the ordering, in the partition, in the crash at that point, and a team should be clear that this is what it buys. A third point follows from the fault list. A crash is injected, so your process can die. A torn write is injected, so a write can be half visible. A partition is injected, so a request can hang. These are failures your code should survive, and Turmoil is a way to check that it does. It is not a way to check that your service is fast, or that it scales, or that its retry policy is tuned. ## Against real fault injection, and against testing in production Two alternatives are worth naming, and the difference in approach is instructive. The first is fault injection against real infrastructure, typically by putting a proxy that can delay, drop and partition traffic between your real processes. The difference in approach is that this approach keeps everything real: real kernels, real clocks, real network, real ordering that you cannot predict. What you get is a faithful environment and a test you may have to run many times to catch the interleaving you care about. What Turmoil gives you is the opposite: a controlled environment where the interleaving is a function of your fault schedule and a seed, so the failing case runs every time and a failure report is reproducible on another machine. The cost is that the environment is only as good as the simulation, and the coverage boundary described above is where that runs out. The second alternative is not testing at all, in the sense of letting the problem appear in production and then writing the regression test from the incident. This is a real practice and it has a virtue that a simulation cannot: the failure is real, so the test is grounded. Its cost is that it happens at the worst time, and that the regression test you write afterwards may not reproduce the conditions, in which case it is a test that passes without proving anything. Turmoil sits between those two, and its pitch is the reproducibility. The test for a partition bug is a file that, when run, produces the partition. That is a stronger guarantee than a flaky test that reproduces once in fifty runs, and it is a much stronger guarantee than a test written from a production incident that cannot reproduce the conditions. For a team whose failures are timing failures, that is the whole argument. A team whose failures are logic failures, a wrong price calculation or a bad SQL query, does not need a simulator and would be better served by an ordinary unit test with a fake dependency. ## Conclusion Turmoil fits a Rust team whose failure modes are timing related, a partition that only happens under load, a write that is acknowledged and then lost, and who need those failures to reproduce on every run rather than intermittently. It does not fit a service that talks to a real database, a real message broker or a real kernel, because the simulated filesystem and socket stack only cover what the crates model. Check three things before you commit. Check which of your dependencies the simulated layer actually replaces, because turmoil-net is a drop-in replacement for tokio::net and turmoil-fs for std::fs and tokio::fs, and anything else your code opens, such as a database client, will not be covered by the simulation. Check whether determinism is what you need, because the seeded RNG and the manual control over each fault are the reason the tests are reproducible, and a test that does not inject anything is not getting the value. And check the release cadence against your own, since the newest listed release is 0.7.2 from 2026-04-24 while the last push was on 2026-10-14. The last push was on 2026-10-14, the license is MIT, and the project lives under the tokio-rs organisation. ## FAQ ### What is turmoil used for? It is a family of crates for deterministic simulation testing of distributed systems. It runs multiple concurrent hosts within a single thread and injects hardship, meaning latency, drops, partitions, crashes and torn writes, into the simulated network and filesystem, under manual control or a seeded RNG. ### Which crates make up turmoil? Four. The turmoil crate is the full simulation testing framework and is the one to start with, turmoil-net is a deterministic simulated socket stack that is a drop-in replacement for tokio::net, turmoil-fs is a deterministic simulated filesystem replacing std::fs and tokio::fs, and turmoil-io-uring is a deterministic simulated io_uring with an optional fs feature that wires it to turmoil-fs. ### What examples does the turmoil repository ship? The examples directory holds end-to-end samples covering axum, a cluster, grpc, and three UDP cases, udp_vpv4_broadcast, udp_vpv4_multicast and udp_vpv6_multicast. Each crate's tests/ directory holds smaller, feature-focused tests. ### Is turmoil deterministic? Yes. It runs multiple concurrent hosts within a single thread and injects faults under manual control or a seeded RNG, so the ordering of events is a function of the fault schedule and the seed rather than of the host scheduler, which is what makes a failure reproducible. ### What license does turmoil use? MIT, with a LICENSE file at the root of the repository, and any contribution intentionally submitted for inclusion is licensed as MIT without additional terms or conditions. ### How current is the turmoil project? The last push was on 2026-10-14 and the newest listed release is 0.7.2 from 2026-04-24, preceded by 0.7.1 from 2026-01-29 and 0.7.0 from 2025-11-03. ## Sources - [Issues](https://github.com/tokio-rs/turmoil/issues) - [License: MIT](https://github.com/tokio-rs/turmoil/blob/main/LICENSE) - [README](https://github.com/tokio-rs/turmoil/blob/main/README.md) - [Releases](https://github.com/tokio-rs/turmoil/releases) - [tokio-rs/turmoil on GitHub](https://github.com/tokio-rs/turmoil) --- Hysen Labs editorial analysis, written from the project's own repository and release notes. Cite the canonical page: https://hysenlabs.com/projects/tokio-rs-turmoil