mmap-sync gives one writer and many readers a wait-free channel over shared memory

The single writer, many readers shape of the problem

mmap-sync is a Rust crate for managing high-performance, concurrent data access between a single writer process and multiple reader processes. The three techniques in its description are the whole story: memory-mapped files, wait-free synchronization and zero-copy deserialization.

The use case is narrower than a general IPC library and that is a strength rather than a limitation, because the constraint is what makes the implementation fast. One process owns the data and publishes it. Any number of other processes read it. Everyone sees the same bytes, and nobody waits for a lock.

Cloudflare states it uses the crate for large-scale machine learning and points to a blog post on scalable machine learning at Cloudflare for the details. That context is instructive: a model or a feature set is trained or updated by one process and then consulted by many, at a rate where copying and serializing the data per request would dominate the cost. The repository's own tags name the problem as interprocess-communication, memory-mapping, synchronization, wait-free and zero-copy.

The crate sits on crates.io with docs on docs.rs, and the repository is small: a Cargo.toml, a src directory, a benches directory, an examples directory and a LICENSE file. The whole design is in a few hundred lines of Rust, which is the right size for a primitive you put under a whole service.

It comes from Cloudflare, and the two listed authors are Alex Bocharov and Austin Hartzheim, the latter with a Cloudflare address in the Cargo.toml.

The Synchronizer is the entire public surface

The interface is a single struct with a write method and a read method, and the README shows the signatures:

impl Synchronizer {
    /// Write a given `entity` into the next available memory mapped file.
    pub fn write<T>(&mut self, entity: &T, grace_duration: Duration) -> Result<(usize, bool), SynchronizerError> {
        …
    }

    /// Reads and returns `entity` struct from mapped memory wrapped in `ReadResult`
    pub fn read<T>(&mut self) -> Result<ReadResult<T>, SynchronizerError> {
        …
    }
}

Both are generic over T, and the README is precise that T can be any Rust struct implementing or deriving certain rkyv traits. That is the type-level contract of the crate: you cannot pass an arbitrary struct, you pass one that rkyv can serialize.

The write method takes a grace_duration alongside the entity, and returns a tuple of a usize and a bool. The README does not spell out what those two values mean, so the shape has to be inferred from the type signature and from the example output, where the writer prints a byte count and a reset flag. A grace duration on a write, in a design like this, is the window the writer gives existing readers before it recycles a buffer, which is the one piece of the protocol that needs a time component rather than a counter.

The read method returns a ReadResult, which is a wrapper rather than a bare T, and that wrapper is where the concurrency contract lives. A reader that is looking at a buffer the writer is about to overwrite has to be told so, and the wrapper is the natural place for that information.

Both methods take &mut self, so a Synchronizer is owned by one task at a time even though many processes share the underlying memory.

Two data copies and a third file for state

The synchronization design is the interesting part of the crate, and the README explains where it comes from: the wait-free data access pattern draws inspiration from the Linux kernel's Read-Copy-Update pattern and from the Left-Right concurrency control technique.

The implementation maintains two copies of the data in separate memory-mapped files. A single writer manages write access, and multiple readers access the data concurrently. The critical detail is that the writer never overwrites the copy a reader is looking at, because there is always a second copy, and the writer alternates between them.

The synchronization state lives in a third memory-mapped file, referred to as state. That file contains an atomic 64-bit integer representing an InstanceVersion, plus a pair of additional atomic 32-bit variables tracking the number of active readers for each data copy.

The bit budget is worth reading, because it explains the design constraints. The InstanceVersion consists of the currently active data file index in 1 bit, the data size in 39 bits, which accommodates data sizes up to 549 GB, and a data checksum in 24 bits. So the entire coordination state, the pointer to the current copy, the length of the data and a checksum, fits in one 64-bit atomic word that a reader can load in a single operation.

The reader counters are what make the wait-free claim work. A reader announces which copy it is using by incrementing the corresponding counter, reads the data, and decrements on the way out. The writer checks the counter for the copy it wants to reuse, and the grace duration on write is the additional slack in that protocol.

A reader that observes a consistent state can proceed without blocking and without retrying, which is what wait-free means here. That is a stronger property than lock-free, where a thread might make progress, and it matters when a stalled reader must not be able to hold up the writer.

Zero-copy reads with rkyv and the byte-level checks

The crate uses rkyv for zero-copy deserialization, directly referencing bytes in the serialized form, which the README says significantly reduces the time and memory required to access and use the data. The dependency list confirms rkyv at 0.7.40 with the validation and strict features enabled, alongside memmap2 for the mapping, thiserror for the error type, bytecheck for byte-level validation and wyhash for hashing.

The validation and strict features are the interesting choice. A reader that maps a file another process wrote is reading bytes it did not produce, and a reader that maps a file that was half-written or truncated is reading garbage. Enabling rkyv's validation means the deserialized view is checked against the archived form, and the presence of a checksum inside the InstanceVersion means the reader can detect that the data it is looking at does not match the version it read.

That combination is what makes the wait-free design safe in practice. The version word is read once, atomically, and it tells the reader both which copy is current and whether the bytes behind it are the bytes that were meant to be there. If they do not match, the reader knows not to trust the result rather than silently consuming a partially written buffer.

The dependency on bytecheck, which is a separate crate for validating archived data at the byte level, reinforces that the crate treats hostile or torn input as a case to handle rather than an impossibility.

The data is stored in shared mapped memory, and the README is clear that the point of memory-mapped files here is to let different processes access the same memory space, bypassing the need for costly serialization and deserialization. In other words, the serialization happens once, at write time, and the readers pay nothing for it.

Running the writer and reader examples

The README gives a complete runnable path, and the examples share a common module that defines the data structure being written and read.

Add the crate to Cargo.toml:

[dependencies]
mmap-sync = "2.0.0"

And import the type:

use mmap_sync::synchronizer::Synchronizer;

Then run the two examples, the writer first and the reader in the same way, with cargo run --example writer and then cargo run --example reader.

The README states the expected terminal output, which is a useful check that your setup is correct:

# Writer example
written: 36 bytes | reset: false
# Reader example
version: 7 messages: ["Hello", "World", "!"]

And the files that get created, which shows the three-file layout in practice:

$ stat -c '%A %s %n' /tmp/hello_world_*
-rw-r----- 36 /tmp/hello_world_data_0
-rw-r----- 36 /tmp/hello_world_data_1
-rw-rw---- 16 /tmp/hello_world_state

Two data files of 36 bytes each and a 16 byte state file, which matches the design: the state file holds the atomic 64-bit version plus the two 32-bit reader counters, and the two data files hold the same payload.

The version number in the reader output is worth noticing. A reader that starts after the writer has published several times will see whatever the current version is, which is how it knows which copy is live.

tmpfs backing and running the benchmarks

The performance tuning section is short and points at the usual culprit. Using a tmpfs volume reduces disk I/O latency because it operates directly on RAM, offering faster read and write than conventional disk storage. The synchronizer is pointed at a path in the shared memory object, and the README notes that /dev/shm is typically where tmpfs is mounted on most Linux systems:

let mut synchronizer = Synchronizer::new("/dev/shm/hello_world".as_ref());

If /dev/shm does not have enough space, or you want a dedicated instance, the README shows how to create one with a chosen size:

sudo mount -t tmpfs -o size=1G tmpfs /mnt/mytmpfs

This is a meaningful change rather than a micro-optimisation. The whole design assumes readers map the file and read bytes directly, and if those pages come from disk rather than RAM, every read fault becomes a disk access and the latency advantage over a socket evaporates.

The benchmarks run through Criterion, and the README gives the tool install and the run command:

cargo install cargo-criterion

cargo criterion --bench synchronizer

The published numbers were executed on a Linux laptop with a 13th Gen Intel Core i7-13800H processor and compiler flags set to RUSTFLAGS=-C target-cpu=native, and the four reported cases are write at about 251 nanoseconds, write_raw at about 145 nanoseconds, read with byte checking at about 40 nanoseconds and read without it at about 27 nanoseconds.

Those read figures are the ones to look at. A read that costs tens of nanoseconds is roughly the cost of the atomic load and the memory access itself, which is the point of the whole exercise. The Cargo.toml shows the bench uses criterion with html_reports, and dev-dependencies include proptest and pprof for flamegraphs, so the benchmark harness is set up for profiling as well as timing.

Where mmap-sync is the wrong tool

There are three boundaries, and each of them follows from the design rather than from a missing feature.

The first is the single writer. Write access is managed by a single writer, and the protocol assumes it. A workload with several producers needs a different structure, because two writers alternating between two copies need a different arbitration mechanism than a writer that knows it is alone. The crate does not claim to solve that, and pretending otherwise would break the wait-free property for readers.

The second is platform. The crate depends on memmap2 and, on Unix targets, on libc. The examples write to /tmp and the tuning advice is about /dev/shm, which is a Linux convention. A portable deployment across platforms without shared memory semantics is not what this crate offers, and the README says nothing about Windows or other platforms, so assume Unix.

The third is that the data has to fit the model. The version word carries a 39-bit size field, so a single published structure is capped at 549 GB. That is generous, but it does mean the design is one large object rather than a stream or an incremental update log, and a workload that publishes a growing time series would be pushed toward something else.

The alternative approach is worth naming because it is often the right one. A plain channel, a Unix socket or a shared-memory queue gives you the same one-to-many distribution without any of the machinery, and it is far easier to reason about. The difference in approach is that mmap-sync removes the copy and the deserialization step by making the readers map the producer's bytes, and it removes the lock by making the producer alternate buffers. Whether that is worth the extra protocol depends entirely on the size of the data and the rate at which it is read. For a small configuration blob refreshed once a second, it is not worth it. For a large model consulted on every request, which is the case Cloudflare describes, the difference is the whole reason the crate exists.

Editorial conclusion

Adopt mmap-sync if you have a Rust service with one producer and many consumers that all need the same large data set, and if the platform gives you memory-mapped files and a shared memory filesystem, because the double-buffered layout with a checksummed version word gives readers a wait-free path with no locks and no copying. Do not adopt it for a multi-writer workload, since the design assumes a single writer, and do not adopt it where the data set is small enough to fit in memory, since a channel or a socket would be simpler. Verify first that your platform can create the mapped files, that the crate version you pin matches the code you read, since the Cargo.toml records 2.0.4 while the most recent published tags are v2.0.1 and v2.0.0, that your data type implements the rkyv traits the crate requires, and that you mount the backing store on tmpfs if disk latency matters to you.

Frequently asked questions

Official sources

1. cloudflare/mmap-sync on GitHub
2. Issues
3. Project website
4. README
5. Releases