# Bitalostored: Redis on disk, built on a self-written B+ tree engine

> Bitalostored is Zuoyebang's Redis-compatible distributed store, written in Go on top of its own bitalosdb key-value engine and its own Raft implementation. It keeps the client protocol unchanged and moves the working set to disk, at the cost of shipping half its consistency model to a paid edition.

**zuoyebang/bitalostored** — Bitalostored is a high-performance distributed storage system, core engine based on bitalosdb(self-developed), compatible with Redis protocol.

- Repository: https://github.com/zuoyebang/bitalostored
- Website: https://bitalos.zuoyebang.com
- Stars: 2,170 · Forks: 154
- Language: Go
- License: Apache-2.0
- Published: 2026-10-14 · Updated: 2026-10-14 · Language: en
- Canonical page: https://hysenlabs.com/projects/zuoyebang-bitalostored

## Redis protocol over disk, for clients that cannot change

Bitalostored exists to solve one problem: keep speaking Redis while the working set outgrows DRAM. The project presents itself as a high-performance distributed storage system whose core engine is bitalosdb, a storage library Zuoyebang wrote itself, and it frames the whole thing as an alternative to Redis that keeps data on low-cost hard disk, spreads work across cores, and cuts infrastructure cost. That framing puts the target user in a specific spot: a team whose clients already speak the Redis protocol, whose application code you do not want to rewrite, and whose key count no longer fits in memory at a price the budget can absorb.

Integration cost stays low precisely because nothing on the client side moves. The command surface covers most Redis commands, including LUA and distributed transactions, so the two pieces a protocol-compatible store has to reimplement are both claimed to survive the change. Horizontal scaling runs to 1024 shards, and shard expansion is described as taking seconds rather than a maintenance window.

Three processes make up the system. Dashboard is the visual management platform, stored is the storage service, and proxy is the proxy service that fronts clients. Proxy is what a Redis client actually talks to, dashboard is the admin web console, and stored is the node that owns shards and runs the engine. Everything below, from consensus down to the structure that indexes keys, lives inside stored.

## install.sh brings dashboard, proxy and stored up on one box

The fastest path to a running system is the shell script at the root of the repository. It targets a single machine that is connected to the Internet, and its prompts ask for three numbers: the number of shards, called group, the number of slave nodes, and the number of witness nodes. All three services come up from that one run, so the dashboard and proxy are part of the evaluation rather than a later exercise.

```bash
install.sh
```

Once it finishes, two addresses matter. The admin web console listens on 127.0.0.1:8080 and both the default user and the default password are demo, which is fine on a laptop and wrong the moment that port is reachable from another host. The Redis endpoint is 127.0.0.1:8790, so an unmodified client works:

```bash
redis-cli -h 127.0.0.1 -p 8790
```

Teardown is the other script sitting beside it:

```bash
uninstall.sh
```

What the script never documents is itself. There is no flag list, no non-interactive mode and no configuration file reference in the README, so a provisioning pipeline cannot learn the prompts from documentation and has to keep a human in the loop. The go.mod file and the Makefile at the root are the alternative for building from source, which the next two sections take apart.

## Under the proxy: bitalosdb, bitalostree and Bitalos-Raft

The dependency list is the clearest statement of the architecture. Module path `github.com/zuoyebang/bitalostored` on toolchain `go 1.25.0` pins `github.com/zuoyebang/bitalosdb/v2` at `v2.0.0` for the key-value engine and `github.com/zuoyebang/bitalosraft` at `v1.0.0` for consensus. The storage layer and the replication layer are separate Zuoyebang projects pinned at fixed versions, and no Redis implementation appears anywhere in the dependency list. Shard routing sits in the proxy process, which is where a 1024 shard topology becomes key to node decisions.

Inside bitalosdb, a short list of named pieces carries the work. bitalostree is a B+ tree built on ultra-large pages with an index compression scheme that is said to remove write amplification in B+ trees. bitaloslog is the log engine, and it indexes its log through bitalostree so writes land in a structure built for throughput. bithash is a KV separation layer on a compact index structure with O(1) retrieval and its own garbage collection, so a large hash workload stops rewriting index pages the way an in-memory table would. A separate K-KV index adds a multi-level vector index on bitalostree to balance compression ratio against retrieval speed, and the KV index runs its vector computation in hand-written assembly.

Redis composite types get their own compressed storage structure, which is where the I/O reduction argument comes from: lists, sets, sorted sets and hashes are laid out for sequential access instead of pointer chasing. Code written against the internal layout of an in-memory Redis dataset will not find that layout here.

## The stored binary only builds with CGO and two Go experiments

Building from source is not a plain go build. The Makefile sets three environment variables before compiling stored, and the link line names native compression libraries:

```bash
CGO_ENABLED=1 $(DEBUG) $(EXPERIMENT) $(CGOLDFLAGS) $(GOBUILD) -o bin/bitalostored ./stored/cmd
```

The variables behind that line are the operational detail that catches people out:

```bash
CGOLDFLAGS=CGO_LDFLAGS="-lstdc++ -lm -lz -lbz2 -lsnappy -llz4 -O2"
EXPERIMENT=GOEXPERIMENT=arenas,greenteagc
DEBUG=GODEBUG=containermaxprocs=0
```

Snappy, lz4, bzip2 and zlib are linked from the system, so a build host or a slim container image has to provide those libraries before the stored binary links at all. The two experiment names pin experimental runtime behaviour, which ties the artifact to the exact toolchain build that still accepts them. Only the stored target sets `CGO_ENABLED=1`; dashboard, fe and proxy compile as ordinary Go binaries, and the fe target copies a prebuilt frontend out of `dashboard/cmd/fe-vue/dist` into `bin/`.

Each target also runs a dependency step that executes a script named `version`, which carries no extension and is tracked at the repository root:

```bash
mkdir -p bin && bash version
```

The clean target then deletes three generated files, `proxy/internal/utils/version.go`, `stored/internal/utils/version.go` and `dashboard/internal/utils/version.go`, so version strings are build output rather than source anyone should edit. Apache-2.0 is the licence named in the Makefile header, so a patch written into this tree carries the same terms.

## The published benchmark runs with WAL and Raftlog disabled

The performance section is specific about hardware and method, which makes it useful, and specific about switching durability off, which makes it a weak stand-in for production. Hardware is one Intel Xeon Platinum 8255C at 2.50GHz with 384GB of memory and two 3.5TB NVMe SSDs. Load comes from memtier_benchmark, the official Redis tool, run with 8 threads inside an 8 core cgroup over a total data size of 40GB.

The comparison covers SET, GET, LPUSH, SADD, ZADD and HSET at two value shapes, 1KB across 40,672,000 keys and 128B across 335,544,320 keys, plus one INCR pass. A command line looks like this:

```bash
./memtier_benchmark -t 16 -c 32 -s 127.0.0.1 -p xxxx --distinct-client-seed --command="set __key__ __data__" --key-prefix="performance_test_key_prefix_" --key-minimum=1 --key-maximum=40672038 --random-data --data-size=1024 -n 81920
```

What the configuration blocks say is worth reading twice. The two comparison products get a 40GB cache, while Bitalostored's own block lists threads, memtable, WAL and Raftlog and carries no cache line at all:

```bash
Threads:8
Memtable：1GB
WAL：disable
Raftlog：disable
```

A run with the log disabled against a competitor holding 40GB of cache measures throughput for data that fits the server buffer, not what happens when a write must reach stable storage, and it leaves a reader unable to tell whether Bitalostored's data was cached, spilled or simply left at a default. The QPS outcome is published as an image, `./docs/benchmark-bitalostored-qps.png`, so the numbers exist only as pixels and cannot be diffed across runs. The two rivals are masked in the README as t\*d\* and \*i\*a, so the same comparison cannot be rebuilt without first guessing which products those are.

## Three services are documented, more directories are shipped

Documentation covers three projects, dashboard, stored and proxy, and the repository root ships more than that: butils/, conf/, dashboard/, paas/, proxy/ and stored/, plus docs/, go.mod, go.sum, install.sh, uninstall.sh, version, Makefile, README.md and README_CN.md. paas/ has no counterpart in that three-project description and no mention anywhere in the prose, and conf/ appears without a single documented key, so a configuration managed deployment cannot be assembled from repository documentation and has to be read out of the source.

The default sizing line is the second thing worth reading closely. It gives the defaults as proxy * 1, group * 2 (master * 2, slave * 2 , witness * 2). Read literally, each group runs two masters, two slaves and two witnesses, so two groups come to twelve storage processes plus one proxy and the script leaves a thirteen process cluster on a single machine. It also stays unclear whether the 2 in group * 2 counts groups or each role, since the two statements do not agree on the unit. A third inconsistency sits in the prompts themselves: the script asks for group, slave and witness counts, yet the default line introduces a master count that is never the answer to any prompt.

On a laptop demo none of that matters. For the multi-data-room or multi-cloud layout the same README describes, those numbers do not tell an operator how many machines a given shard count actually consumes.

## The 1PB figure arrives with no hardware behind it

Two different evidence standards live in the same README. The benchmark section names a CPU model, a memory size, a disk count and a load tool. The practice section, which covers what Zuoyebang runs internally, names none of that: peak QPS of 30 Million, 8000Gbps of network, 1PB of stored data, an average read of 100μs and an average write of 150μs, availability of 99.999%, and zero online incidents since V1.0 was released in 2019.

Read as a production claim, that list has holes a buyer will notice. No incident log sits behind the zero incidents figure, no record says what share of traffic is reads against writes, and no topology explains how 1024 shards turn into 1PB. The availability number is an aggregate rather than a per-shard figure, and nothing says how many shards a single machine failure would take out. The latency figures carry no percentile, and an average read of 100μs says nothing about the tail a user actually waits on. Architecture detail is pushed to bitalos.zuoyebang.com, which means none of those numbers are backed by a document inside the repository itself.

## Multi-master CRDT sits behind the Enterprise label

The most useful line in the README is also the most awkward one. The introduction calls the open source version stable and says it provides a complete industrial grade solution. A few lines later, the multi-master architecture built on CRDT, with adaptive conflict resolution during multi-master writes and eventual consistency, is marked as supported in the Enterprise Edition. The consistency model available from source is therefore the single-writer-per-shard one built on Bitalos-Raft, with witness nodes as the third node role the installer asks about. Nothing in the repository covers CRDT merge behaviour, conflict policy or the sync protocol, because that path is closed in the open source build.

Failure handling is the other half. Disaster recovery is described as multi-data room or multi-cloud deployment with a complete downgrade and disaster recovery scheme, and the install script asks for a witness count next to the group and slave counts. What is missing is any threshold, any failover time and any split rule, so an operator has no way to reason about a data centre becoming unreachable. The last push to this repository was on 2026-04-03.

## Conclusion

Adopt Bitalostored if your clients already speak Redis and the dataset no longer fits in memory; protocol compatibility and the single-machine install.sh path can be checked in an hour. Pass on it if you need multi-master writes, since the CRDT layer carries an Enterprise Edition label, or if you need durability evidence, since the published benchmark runs with WAL and Raftlog disabled. Open the QPS image and the conf/ directory first: the first holds the only benchmark result, the second documents no keys.

## FAQ

### How do I start a local Bitalostored cluster to try it out?

Run install.sh on a machine connected to the Internet and answer the prompts for group, slave and witness counts, which brings up dashboard, proxy and stored together. The admin console is on 127.0.0.1:8080 with demo as both user and password, and the Redis endpoint is 127.0.0.1:8790, reachable with redis-cli -h 127.0.0.1 -p 8790. uninstall.sh removes the deployment.

### Which Redis commands does Bitalostored support?

Support is stated for most Redis commands, including LUA scripting and distributed transactions, which is why existing clients can switch over without a rewrite. The published benchmark exercises SET, GET, INCR, LPUSH, SADD, ZADD and HSET.

### How many shards can a Bitalostored cluster hold?

Horizontal scaling is documented at up to 1024 shards, and shard expansion is described as completing in seconds.

### Does the open source build support multi-master writes?

Not according to the README. The CRDT based multi-master architecture, with adaptive conflict resolution and eventual consistency, is listed as supported in the Enterprise Edition, so the open source build gives you single-writer shards over Bitalos-Raft.

### What hardware was the Bitalostored benchmark run on?

The README gives one Intel Xeon Platinum 8255C at 2.50GHz with 384GB of memory and two 3.5TB NVMe SSDs, with memtier_benchmark running 8 threads in an 8 core cgroup over 40GB of data. WAL and Raftlog were disabled for Bitalostored and the two masked comparison products were given a 40GB cache, so the run measures throughput rather than durability.

## Sources

- [License: Apache-2.0](https://github.com/zuoyebang/bitalostored/blob/main/LICENSE)
- [Project website](https://bitalos.zuoyebang.com)
- [README](https://github.com/zuoyebang/bitalostored/blob/main/README.md)
- [Releases](https://github.com/zuoyebang/bitalostored/releases)
- [zuoyebang/bitalostored on GitHub](https://github.com/zuoyebang/bitalostored)

---

Hysen Labs editorial analysis, written from the project's own repository and release notes. Cite the canonical page: https://hysenlabs.com/projects/zuoyebang-bitalostored