Raft Engine, a log-structured store for Multi-Raft replicas

A storage engine scoped to one job, not a general database

Raft Engine is described as a persistent embedded storage engine with a log-structured design similar to bitcask, built for TiKV to store Multi-Raft logs. That framing matters, because it says exactly what the crate is and, just as importantly, what it is not.

It is not a general-purpose embedded key-value database. The feature list is short and specific: APIs for storing and retrieving protobuf log entries with consecutive indexes, key-value storage for individual Raft Groups, minimum write amplification, collaborative garbage collection, lz4 compression over log entries, and support for file system extension.

Key-value storage exists, but it is scoped. It is key-value storage for individual Raft Groups, not a general store across groups. If you need range scans over arbitrary keys, secondary indexes or a query surface, this is not the crate, and no amount of configuration will produce those.

The audience is a developer implementing a Raft-based replicated log in Rust and looking for the storage layer that TiKV itself uses. The strongest argument for adopting it is not that it is general, but that it is specific: the same engine that stores TiKV's Multi-Raft logs is available as a crate, and the design decisions that survived contact with a production database are already made.

The crate is published on crates.io as `raft-engine` with documentation on docs.rs, and the licence is Apache 2.0, copyright PingCAP, Inc.

The other thing to notice is the maturity. The recorded releases are 0.4.0 in September 2023, 0.4.1 in September 2023 and 0.4.2 in April 2024, and the last push to the repository was on 2026-09-10. The crate is pre-1.0, which in Rust terms means the API may still move, and it has been on the 0.4 line for a while.

Memtables over a shared log stream

The architecture is two constructs, and the whole design follows from how they interact.

Raft Engine consists of two basic constructs: memtable and log file. In memory, each Raft Group holds its own memtable, containing all the key value pairs and the file locations of all log entries. On storage, user writes are sequentially written to the active log file, which is periodically rotated below a configurable threshold. And the detail that changes everything is that different Raft Groups share the same log stream.

That last point is the design decision with the most consequences. A Raft Group is a replication group, a set of replicas for one shard of a keyspace. In TiKV there are thousands of them. Giving each one its own file would mean thousands of files and thousands of write streams, so instead they share a single stream, and each group tracks where its own entries landed.

The memtable is the index. It holds the current key-value state for a group and, at the same time, the file locations of the group's log entries, which is how a later read or a later compaction knows where the data is on disk.

The write amplification claim follows from the layout. Because writes go to one log sequentially and the memtable is updated in memory, there is no per-key index structure to rewrite on every write, which is the source of write amplification in a tree-shaped store. The feature list puts it as minimum write amplification.

The trade is that the log is the source of truth and the memtable is derived. A crash means recovering state by replaying the log, and the memtable has to be rebuilt. That is a different recovery model from a store that fsyncs a b-tree page, and it is a good fit for a replicated log where the entries are durable by design and the state machine can rebuild itself.

Batching, queue leadership and the fdatasync decision

The write path is where the concurrency design shows, and the documentation walks through it in three steps.

The writing of one log batch can be broken down into three steps: optionally compress the log entries, write to log file, and apply to memtable. The first step is where lz4 compression happens, and it is optional per batch.

The second step is where concurrency is handled, and the mechanism is worth understanding because it is not obvious. To group concurrent requests, each writing thread must enter a queue. The first in line automatically becomes the queue leader, responsible for writing the entire group to the log file. So the batching is emergent, not something the caller arranges: threads queue up, one of them is elected by position, and that thread writes everybody's data in one pass.

The third step is per-thread. After its data is written, each writing thread proceeds to apply the changes to the memtable on their own. So the memtable update is parallel across threads while the file write is serialized under one leader. That split is what keeps the critical section short: the leader does the sequential I/O, and the CPU-side state update is spread across the threads that were waiting anyway.

Durability is a per-batch decision. Both synchronous and non-sync writes are supported. When one write in a batch is marked synchronous, the batch leader will call `fdatasync()` after writing, so buffered data is guaranteed to be flushed out onto the storage. The consequence for a user is that a synchronous write in a batch raises the bar for the whole batch, because the leader syncs after the group is written.

The atomicity model is similar to RocksDB's, and the documentation says so: users can stash the changes into a log batch before submitting, so a batch either lands in the log as a unit or not at all.

Garbage collection that the user schedules and the user answers

This is the part of the design that most distinguishes Raft Engine from an embedded database, and it is the part a new user has to understand before anything works properly.

After changes are applied to the local state machine, the corresponding log entries can be compacted from Raft Engine logically. Because multiple Raft Groups share the same log stream, these truncated logs will punch holes in the log files. During garbage collection, Raft Engine scans for these holes and compacts log files to free up storage space. Only at this point are the unneeded log entries deleted physically.

The two-stage design is the point. Compaction of a log is a state-machine decision, because only the state machine knows which entries are no longer needed. Physical reclamation is a storage concern. The engine separates them and asks the user to bridge the gap.

The timing is controlled by the user. Raft Engine consolidates and removes its log files only when the user voluntarily calls the `purge_expired_files()` routine, and the documentation gives the reference that TiKV calls it every 10 seconds by default.

The second half is the feedback loop, and it is the part that surprises people. Each time the GC routine is called, Raft Engine examines itself and returns a list of Raft Groups that hold particularly old log entries. Those log entries block the GC progress and should be compacted by the user.

So the contract is: you call purge, the engine tells you which groups are holding the space hostage, and you must go compact those logs in your state machine and then call purge again. Skip the middle step and the space is never reclaimed. This is a cooperative design, and it means the engine is not usable as a black box that reclaims space on its own.

Adding it to Cargo.toml, and the features that change the build

The dependency declaration is the whole installation story, because this is a library rather than a service.

[dependencies]
raft-engine = "0.4"

The available Cargo features are where the real configuration surface is, and each one changes what gets compiled in rather than what you set at runtime.

`scripting` compiles with Rhai and enables script debugging utilities including `unsafe_repair`. `nightly` enables nightly-only features including `test`. `internals` re-exports key components internal to Raft Engine, and the manifest enables it when building for docs.rs. `failpoints` enables fail point testing powered by tikv/fail-rs, which is how the project injects failures at specific points in the write path. `swap` uses `SwappyAllocator` to limit the memory usage of Raft Engine, with the memory budget configured through a `memory-limit` setting, and it depends on the nightly feature.

The manifest also shows the shape of the crate. The edition is 2024 and the minimum supported Rust version is 1.85.0, which is a recent floor. Protobuf 2 is the log entry encoding, byteorder and crc32fast handle the on-disk format, crossbeam and parking_lot provide the concurrency primitives, rayon is there for parallel work, and the log crate is configured with `max_level_trace` and `release_max_level_debug`, so trace-level logging is available in debug builds and debug-level in release builds.

For a first real use, the examples directory is the place to start, and the manifest names them: append-compact-purge, which walks the whole lifecycle of writing entries, compacting them and then purging the files, raft-engine-fork, which shows the fork mechanism, and json-codec, which uses a custom codec and requires the serde-json feature.

The fork example deserves a note, because forking is the one feature with a lasting operational consequence. Forking lets you take a snapshot of the log stream, which is how a Raft log gets captured for a backup or handed to a new replica, and the process of a file system extension is a related idea about growing the underlying storage.

The Makefile is the test policy, and the release line is old

The contributing section in the documentation is unusually specific about how tests are run, and the Makefile is where the detail lives.

Tests are automatically run against changes, and some can be run locally. The commands in the documentation are short.

# run tests with nightly features
make
# run tests on stable toolchain
make WITH_STABLE_TOOLCHAIN=force
# filter a specific test case
make test EXTRA_CARGO_ARGS=<testname>

The Makefile explains what those flags mean. The `WITH_STABLE_TOOLCHAIN` variable has three states: auto uses the current default toolchain and disables nightly features, force explicitly uses the stable toolchain and disables nightly features, and anything else leaves nightly features on. The build then probes whether the active rustc is a nightly and picks `+nightly` or `+stable` accordingly.

The targets are format, clippy and test, with `all` running the three in sequence. Clippy runs with `-D clippy::all` and a whitelist that allows `clippy::bool_assert_comparison`, across all targets and the nightly_group, serde_group and failpoints features, or the reduced set on stable. The failpoint tests are run with a single test thread, which is the point of fail points: they inject failures at chosen code paths, and a deterministic order is what makes the result meaningful.

There is also a separate stress tool, documented as a standalone stress test tool used to demonstrate system performance, with its own command line.

cargo +nightly bench --all-features <bench-case-name>
cargo run --release --package stress -- --help

And the project asks that for changes that might induce performance effects, you quote the targeted benchmark results in the pull request description.

Two practical warnings. First, the crate is still on 0.4 and the last tagged release was 0.4.2 in April 2024, while the repository has been pushed to as recently as 2026-09-10, so the code on master is ahead of the published crate and a consumer on 0.4.2 does not get those changes. Second, the crate is written against Rust 2024 with a 1.85.0 floor, which constrains the toolchain of anything that depends on it.

Editorial conclusion

Raft Engine suits a Rust developer building a replicated log service that has the same shape as TiKV's, that is, many Raft Groups writing into one stream with per-group key-value state, and who wants that pattern without a general-purpose embedded database. It does not suit an application that wants a general key-value store with range scans and secondary indexes, since the key-value interface is scoped to individual Raft Groups, and it does not suit a team that wants the engine to reclaim space on its own, because garbage collection only happens when the user calls it. Verify first that your workload tolerates the log-file layout, that you can supply the compaction feedback the engine returns, and that the version you pin, 0.4.2, is the one your Rust toolchain and edition support.

Frequently asked questions

Official sources

1. Issues
2. License: Apache-2.0
3. README
4. Releases
5. tikv/raft-engine on GitHub