# Corrosion: gossip-based state for a globally distributed network

> Corrosion is the tool Fly.io built to replace the central state database in Consul with eventually consistent state spread across every host, using a SQLite database per node, gossip to propagate changes, CR-SQLite for conflict resolution and a SWIM protocol for membership. The interesting design decision is the move away from a consensus log, and the README explains the latency argument for it in numbers.

**superfly/corrosion** — Gossip-based service discovery (and more) for large distributed systems.

- Repository: https://github.com/superfly/corrosion
- Website: https://superfly.github.io/corrosion/
- Stars: 1,875 · Forks: 105
- Language: Rust
- License: Apache-2.0
- Published: 2026-10-16 · Updated: 2026-10-16 · Language: en
- Canonical page: https://hysenlabs.com/projects/superfly-corrosion

## A central database replaced by distributed state

Corrosion is described as gossip-based service discovery, and more, for large distributed systems, and the README explains the motivation in the first person. It was built specifically for service discovery across a large global network, replacing Consul's central state database with eventually consistent state distributed across the hosts.

That is a strong claim about an existing piece of infrastructure, and the rest of the section explains why the team made it. Fly.io runs machines in many regions, and a service discovery entry that lives in one place is a cross-ocean request every time something needs to look it up.

The new tool had to deliver three things, and they are stated as requirements rather than as wishes. Fast reads and writes, where getting state from a central remote source can be incredibly expensive, at least 300 milliseconds for a round trip to something on the other side of the world, but usually less than 1 millisecond from a local source. Fast, eventual consistency, on the grounds that many use cases can cope with eventual consistency, especially if a consistent state is attained sooner than later, with the explicit note that Raft fell short for some use cases at Fly.io because round trips to a centralized location are too expensive. And flexibility, since global state for a distributed system is not one-size-fits-all and flexible schemas and queries are essential.

That third requirement is the one that shapes the API, because the state is queried with SQL rather than through a fixed key-value interface.

## Five moving parts, and what each one does

The How Corrosion works section is a list of five, and it is short enough to read as an architecture.

Corrosion maintains a SQLite database on each node. It gossips local changes throughout the cluster. It uses CR-SQLite for conflict resolution with CRDTs. It uses Foca to manage cluster membership using a SWIM protocol. And it periodically synchronizes with a subset of other cluster nodes, to ensure consistency.

Each of those choices solves a specific problem. The per-node SQLite database is what makes the sub-millisecond local read possible, and it also means the node is usable even when the network is not. Gossip is the propagation mechanism, and gossip has the property that the network keeps converging without anyone needing to be a leader.

CR-SQLite is where the conflict resolution happens, and choosing a CRDT-based approach rather than a consensus protocol is the crux of the design. A consensus protocol would give strong consistency at the cost of the round trips the whole project exists to avoid.

Foca handles membership with SWIM, which is the protocol designed for large-scale membership with failure detection, and the periodic synchronisation with a subset of nodes is the mechanism that stops gossip drift from leaving two halves of the cluster disagreeing indefinitely.

The peer-to-peer transport is QUIC, using the Quinn implementation, and the README lists secure peer-to-peer communication as a feature, which is the right thing to insist on for a protocol that carries your service registry.

## SQL, streaming subscriptions and templates

The feature list describes what the API gives you, and the first item explains the shape of the whole thing: a flexible API to read from and write to the store using SQL statements.

That is the direct answer to the flexibility requirement, and it is a bigger design decision than it looks. A SQL API over a gossiped store means the query language is familiar, the schema can change, and a query can be anything SQLite can express. The cost is that queries have to be designed with eventual consistency in mind, since a join across two tables may see two different points in the history.

On top of that there are file-based schemas with on-the-fly updates, so the schema is a file in the repository rather than a migration chain, and HTTP streaming subscriptions based on SQL queries, which is how a program subscribes to changes matching a condition rather than to a whole table.

Then there is the template system, live population of configuration files from Corrosion's state with user-defined Rhai templates. That is a genuinely interesting use of a distributed store, because it turns a cluster-wide configuration into a rendered file, and a rendering engine is a better model for that than a database read on every node.

Finally there is the Consul interoperability: storage and propagation of state from locally registered Consul services, replacing the central database with Corrosion's distributed state. That is the migration path, and it is the reason an existing Fly.io deployment could adopt this at all, since the services are already registered and the state is what moves.

## An agent on every host, and an HTTP API next to it

The usage overview is short and it is a two-tier model. You run the Corrosion agent on every node or host in the cluster, and other programs running on the node use Corrosion's HTTP API to query the local Corrosion SQLite database, add and update data, and subscribe to change notifications.

The locality is the point. An application on a host does not make a network call to find out the state of its neighbours, it queries the database next to it. The network is used for propagation, not for reads, and that is the difference between a 300 millisecond round trip and a sub-millisecond local query.

Administration is a separate surface. The Corrosion CLI provides commands for administration and access to the database and the features, and the quick start is three steps: prepare the Corrosion configuration file, specify the initial database schema, and start the Corrosion agent. The README points at the documentation and calls it work in progress, which is worth bearing in mind for a project whose own documentation site describes itself that way.

The configuration is a TOML file, and the repository ships config.example.toml alongside two example templates, one Rhai and one CSV, which are the shape of the configuration rendering the features list describes.

## Building it, and what the workspace contains

Building is two commands from a clone:

```
cargo build --release && mv target/release/corrosion ./
```

The workspace layout tells you what the project is made of. The members are the crates/ directory, the integration-tests directory, and the rust-load-generator client under .antithesis, which is worth noting because it means the project runs its own load generator against itself as part of testing.

The dependency list is where the design is visible. axum with http2, websockets and tracing for the HTTP and streaming API. deadpool-sqlite for the connection pool. papaya for a concurrent map, which is the usual choice for a shared in-memory index. foca with bincode and serde codecs for the SWIM membership. governor for rate limiting. metrics with a Prometheus exporter for observability. hyper and hyper-rustls for the HTTP stack, config for TOML configuration, hickory-resolver for DNS, bincode for serialisation and build-info to stamp the version into the binary.

The build and test infrastructure is unusual for a Rust project and is a good sign. There is a .buildkite/ directory and a .buildkite-antithesis/ directory for continuous integration, a .antithesis/ directory with the client, integration-tests/ as a workspace member, and a flake.nix for a reproducible development environment with a pinned rust-toolchain.toml.

The release list shows the shape of the project. v1.0.0 was tagged in May 2026, and before that there were a series of v0.1.0+test builds from October 2025, which is the pattern of a project that spent a long time in testing releases before calling version one.

## Where Corrosion is the wrong choice

The first wrong-tool case is a workload that needs strong consistency. Corrosion trades a consensus protocol for CRDTs on purpose, and the requirement it lists is fast, eventual consistency. A ledger, a balance, a lock, a leader election or anything where reading a stale value is a correctness problem rather than a latency problem, does not belong on this foundation, and no amount of tuning the gossip will change that.

The second is a small cluster. The argument is about a round trip to the other side of the world taking at least 300 milliseconds. A handful of nodes in one datacentre, or a fleet in one region, has a central store that is fast enough, and a gossiped CRDT store with membership management, conflict resolution and periodic reconciliation is a lot of machinery for a problem that does not exist yet.

The third is a team that wants a service discovery protocol rather than a state store. Corrosion describes itself as service discovery and more, and the more is substantial. If the need is a DNS-style or HTTP-style discovery endpoint that answers who is where, a conventional discovery service is simpler and better documented, and the Consul interoperation path in the feature list is a hint that the service discovery use case is a special case rather than the centre.

There is also the documentation caveat. The README calls the documentation work in progress, and the project reached version 1.0 in May 2026, which is recent enough that the operational experience of running it across a large fleet is still accumulating.

## Conclusion

Adopt Corrosion if you run a fleet spread across continents and your current design makes every read a cross-ocean round trip, and if your state model tolerates eventual consistency. Do not adopt it if you need strong consistency, for instance a balance that must never be read stale, because the whole design trades that away deliberately. Before you commit, read the requirements section, because the 300 millisecond argument that justifies the design is also the boundary of its applicability, and check the Consul interoperability path, since migrating an existing deployment is a larger job than running a new agent on every host.

## FAQ

### What is Corrosion?

A Rust tool for gossip-based service discovery and shared state in large distributed systems, built at Fly.io to replace the central state database in Consul with eventually consistent state distributed across the hosts.

### How does Corrosion resolve conflicts between nodes?

Each node keeps a SQLite database, local changes are gossiped throughout the cluster, and conflicts are resolved with CR-SQLite, which applies CRDT semantics, so the merge does not need a consensus round trip. Cluster membership is handled by Foca using a SWIM protocol, and nodes periodically synchronise with a subset of the cluster.

### Why did Fly.io move away from Raft for Corrosion?

Because round trips to a centralised location are too expensive across a global network, at least 300 milliseconds to the other side of the world against under 1 millisecond from a local source. Raft fell short for some use cases there, and many use cases can cope with eventual consistency, especially if a consistent state is attained sooner than later.

### How do you build and run Corrosion?

Clone the repository and run cargo build --release, moving the binary to the repository root. The quick start then prepares the configuration file, specifies the initial database schema, and starts the agent, and you run that agent on every node in the cluster, with other programs using the local HTTP API.

### What licence is Corrosion released under?

Apache-2.0, set in the workspace package metadata. Version 1.0.0 was released on 2026-05-14 after a long run of test releases, the last push was on 2026-10-15, and the README describes the documentation as work in progress.

## Sources

- [License: Apache-2.0](https://github.com/superfly/corrosion/blob/main/LICENSE)
- [Project website](https://superfly.github.io/corrosion/)
- [README](https://github.com/superfly/corrosion/blob/main/README.md)
- [Releases](https://github.com/superfly/corrosion/releases)
- [superfly/corrosion on GitHub](https://github.com/superfly/corrosion)

---

Hysen Labs editorial analysis, written from the project's own repository and release notes. Cite the canonical page: https://hysenlabs.com/projects/superfly-corrosion