AgateDB ports Badger to Rust so TiKV can keep its own storage engine

A storage engine for TiKV, written as a Badger port

AgateDB is an embeddable, persistent and fast key-value database written in pure Rust. That description is accurate but does not say the interesting part, which is why it exists.

It is designed as an experimental engine for the TiKV project, and it will bring aggressive optimizations for TiKV specifically. The whole plan, as the README puts it, is to port Badger in Rust first and then port the optimizations that have been made in unistore. Badger is a key-value store in Go, and unistore is TiKV's own storage layer, which is itself based on Badger.

The dependency chain is the whole design. TiKV gets its storage from unistore, unistore is built on Badger's design, and the optimisations that TiKV needed were made in unistore in Go. AgateDB exists to bring that optimised design into Rust, which is the language TiKV itself is written in.

The audience is therefore narrow and specific. This is a research and development vehicle for the TiKV project, and the people who should be looking at it are the ones who work on TiKV's storage layer or want to understand what a Rust Badger looks like. The repository is under the tikv organisation, the primary language is Rust, the licence is Apache-2.0, and the last push was on 2026-09-14.

The project status section says AgateDB is still under early heavy development and points to a GitHub Project board for the development progress, and there are no GitHub releases, with the version in the manifest sitting at 0.1.0.

Why Rust, stated as two practical reasons

The README has a section titled Why not X, and it is worth reading because it is the project arguing with itself about the design.

The text is disarmingly direct. X is a great project, and the motivation of this project is to ultimately land the optimizations that were made to unistore in TiKV. Unistore is based on Badger, so the project starts with Badger. In other words, the alternative considered and rejected was not a better database, it was a different ecosystem, and the answer was that the goal was the optimisation work rather than the storage engine per se.

The Rust decision is then explained in two sentences. Rust can bring memory safety out of the box, which is important during rapid development, and TiKV is also written in Rust, so it will be easier to integrate with each other, like supporting async and await and sharing global thread pools.

Both reasons are about the development process rather than about the finished system. Memory safety matters most when the code is changing quickly, and this is a codebase that is being rewritten from another language while its design is still moving. The integration argument is about the long run, since a storage engine that has to be called across an async boundary from TiKV, with a shared thread pool, is a different engineering problem from one that has to be called from Go.

The current capability, as the README states it, is that AgateDB is under active development on the develop branch, and can currently be used as a key-value store with MVCC, implementing most of the functionalities of Badger managed mode. That last phrase is the one to note. Badger managed mode is the subset where the library owns the LSM tree and the value log, rather than leaving them to the caller, and implementing most of it is a meaningful chunk of Badger's behaviour, but not all of it.

The dependency list reads like a description of the engine

The Cargo.toml is short and every entry earns its place, so it is a good map of how the engine is built.

[package]
name = "agatedb"
version = "0.1.0"
authors = ["Jay Lee <[email protected]>"]
edition = "2018"

The edition is 2018, which is the Rust 2018 edition, and the version is 0.1.0.

The dependency choices map onto the components. bytes is the byte buffer type the KV store works in. crc and crc32fast are the checksums, and farmhash is the hash function, which is a detail that matters in an LSM engine where the hash decides where a key lives in the memtable. memmap2 is the memory mapping crate, which is how a value log or a table is mapped rather than read. parking_lot and crossbeam-channel are the concurrency primitives. prost is the protobuf implementation, and there is a local proto package, so the wire format between components is defined in this repository rather than borrowed. skiplist is a local path dependency, which is the memtable structure. rand is for randomisation in tests and in the engine, tempfile for test files, thiserror for error types, and enum_dispatch for the kind of trait-object dispatch an engine with several table types needs.

The one dependency that deserves a note is yatp:

yatp = { git = "https://github.com/tikv/yatp.git", rev = "b793461ea4abd202798cda6e20e97a054103f2ad" }

It is pinned to a specific commit of a TiKV repository rather than a published version, which is a signal about how tightly this engine is coupled to the TiKV organisation's own work. It is the async thread pool that the README's integration argument was about, in dependency form.

And rocksdb is optional, behind the enable-rocksdb feature, which exists so the same workload can be run against both engines.

A workspace with its own memtable, proto and benchmark crates

The repository is a Cargo workspace, and the members list in the manifest says what has been factored out of the main crate:

[workspace]
members = [
    "agate_bench",
    "proto",
    "skiplist"
]

Three members. agate_bench is the benchmarking crate, proto is the protocol buffer definitions shared across components, and skiplist is the skip list, which is the in-memory ordered structure that a KV engine like this needs for the memtable and for range scans.

That the skip list is a separate crate rather than a module is a design signal. It is the data structure most likely to be replaced or tuned, and giving it its own crate means it can be worked on and benchmarked without touching the rest of the engine.

The benchmarks themselves are declared in the manifest, and there are five of them: bench_common, bench_table, bench_iterator, and bench_common's counterpart for the comparison run, benches_agate_rocks. Each sets harness = false, which is the criterion that tells Cargo not to use the built in test harness for a benchmark that manages its own timing. The comparison benchmark name tells you the shape of the exercise, AgateDB against RocksDB, the engine TiKV actually uses in production.

The bench profile is tuned for measurement rather than for size. opt-level is 3, lto is thin, codegen-units is 1, and incremental is false, with debug assertions and overflow checks turned off. Those are the settings you use when you want the number to reflect the code rather than the instrumentation, and the same profile is what makes a comparison against RocksDB meaningful.

The top level file listing matches this structure, with a src/ directory for the engine, benches/ for the benchmark suite, a proto/ directory, a skiplist/ directory, an agate_bench/ directory, and scripts/ for the helper tooling.

Clippy treats warnings as errors, and there is a sanitizer target

The Makefile is where the engineering standards of the project are visible, and they are stricter than a typical Makefile.

clippy:
	cargo clippy --all-targets --all-features --workspace -- -D "warnings"

test:
	cargo test --all-features --workspace

test_sanitizer:
	RUSTFLAGS="$(SANITIZER_FLAGS)" cargo test --all-features --workspace

The -D warnings flag turns every clippy warning into an error, and the invocation covers all targets, all features and the whole workspace rather than the default package. The all-features flag matters here, because the optional rocksdb dependency is a feature, so a lint or test run that skipped features would skip a whole storage engine.

The sanitizer target is the second thing worth noticing. The flags are set to -Zsanitizer=address, and there is a matching sanitizer target for the benchmarks. Running the test suite under the address sanitizer is how a memory bug in a storage engine gets caught, and with a project that exists precisely because of memory safety concerns, that is a coherent choice.

There is also a formatting target that does more than cargo fmt. It installs rustfmt and cargo-sort through rustup and cargo install, and then sorts the Cargo.toml files, which is a small but effective way to keep a long dependency list reviewable in diffs, because it means a new dependency always appears in the same place rather than at the end of a list someone appended to.

The dev target chains format, clippy and test, so the full local check is a single command.

The benchmark against RocksDB is the published comparison

The README does not make performance claims in prose. It points at a page, tikv.github.io/agatedb/dev/bench/, and says the benchmark result with RocksDB can be found there.

That is a deliberate choice and a good one for an early stage project. An experimental engine has no stable story about throughput or latency yet, so publishing a continuously updated benchmark page, rather than a number in a README that goes stale, keeps the claim honest and the comparison live.

RocksDB is the right yardstick here for a specific reason. It is the storage engine TiKV uses in production, so the question AgateDB has to answer is not whether it is fast in the abstract, but whether a Rust Badger can reach the performance of the engine the project already relies on. That is also why the rocksdb dependency exists at all, as an optional feature so the same workload can be run against both engines, and why there is a dedicated benches_agate_rocks benchmark alongside the internal ones.

For an evaluator, the benchmark page is the single most useful document in the project, more useful than the source. It answers the question that decides whether the port is worth continuing, and it is the thing to look at first when the question is whether AgateDB is a serious effort or a sketch.

The other signal in the repository is the coverage badge, which points at codecov for the master branch, so the test suite is instrumented and the coverage is tracked over time.

Editorial conclusion

Adopt AgateDB if you are working inside the TiKV ecosystem and want to experiment with a Rust storage engine that can absorb the unistore optimizations, and if you can live with an API that is still moving. Do not adopt it for production use outside that context, because the README states the project is under early heavy development, there are no tagged releases and the published version is 0.1.0. Before you build against it, read the project status page linked from the README, check the RocksDB benchmark page for the comparison the maintainers publish, and plan for the develop branch rather than master, since the README says active development happens there.

Frequently asked questions

Official sources

1. Issues
2. License: Apache-2.0
3. README
4. tikv/agatedb on GitHub