Apache BookKeeper: append-only storage for WAL and distributed messaging

Append-only is a design choice, not a limitation

Apache BookKeeper describes itself as a storage service "optimized for append-only workloads," and that phrase is load-bearing. An append-only storage layer writes new entries at the end of a log and does not support in-place updates or deletions. That constraint is the reason BookKeeper can promise low latency on writes: without the overhead of lock management, index updates, and random-access write paths, the write path stays short.

Write-ahead logging for HDFS NameNode is the first use case the README names. When NameNode records a metadata change, it needs the log entry durably stored before it proceeds. A storage layer that supports updates in place carries locking overhead and fragmentation that a sequentially-written log does not. BookKeeper trades away update flexibility to keep that overhead out of the write path.

This trade-off is permanent, not configurable. Callers that need to overwrite a stored value have to build that logic on top of BookKeeper's append-only entries or use a different storage system entirely. Nothing in the README or the repository suggests a mode that relaxes this constraint.

How WAL, message store, offset store and snapshot use share the same requirements

Apache BookKeeper's README lists four concrete scenarios where the service fits: WAL, message store, offset/cursor store, and object/blob store for state machine snapshots. Three of those four name specific projects that use them.

HDFS NameNode and Pravega both rely on BookKeeper for WAL. A write-ahead log must survive node failure, confirm writes durably, and replay entries in order after a crash. Those requirements align directly with BookKeeper's append-only, fault-tolerant design. Losing the NameNode's edit log would corrupt the entire HDFS file system's metadata, so the storage layer underneath it cannot afford to acknowledge a write before the entry is actually durable.

Apache Pulsar uses BookKeeper twice: as the message store for topic data and as the offset/cursor store for consumer positions. A cursor that tracks how far a consumer has read is an append-only workload: the offset only advances forward. Pulsar chose an external storage substrate rather than managing durability in its own broker code, which means BookKeeper handles the replication and durability guarantees that a broker would otherwise need to build.

The object/blob store pattern for state machine snapshots follows the same reasoning. A snapshot written once and read back later is a natural fit for append-only storage with fault tolerance, and no update path is needed.

Operators need bookkeeper.apache.org/docs/getting-started/concepts before the setup guide

No command appears in the README for downloading, configuring or starting BookKeeper. Instead, it points to two URLs: bookkeeper.apache.org/docs/getting-started/installation for the setup guide and bookkeeper.apache.org/docs/getting-started/concepts for the foundational concepts. According to the project's own phrasing, starting with the concepts page "will help you to fully understand the other parts of the documentation," which makes the sequence deliberate: read the concepts page before touching the setup guide, not after.

For a project that spans a main server module, HTTP integration, protocol definitions, metadata drivers, a streaming interface, native I/O support, Docker deployment files, and multiple benchmark suites, a single README section would not be sufficient to cover what the setup guide presumably covers in detail. BookKeeper is a system with its own vocabulary for how data is organized and replicated, and that vocabulary governs every configuration decision.

Operators approaching BookKeeper for the first time should use the external documentation site as the real starting point, because the repository README covers only the project's purpose and four named use cases.

A multi-module Maven project with native I/O, benchmarks and pluggable metadata support

Repository top-level entries show more than thirty items: Maven sub-modules, configuration, deployment scripts and testing infrastructure. BookKeeper is built with Java using Maven, with pom.xml at the root.

Among the module directories, several are specific in scope. Both bookkeeper-benchmark/ and microbenchmarks/ indicate the project maintains benchmarking infrastructure for evaluating performance characteristics. A module named bookkeeper-proto/ holds what that name implies: protocol definitions. Two directories, native-io/ and native-library-common/, signal the presence of native code alongside the Java implementation. A storage system optimized for low-latency writes has reason to use native I/O paths, and these directories confirm that native support exists in some form.

Containerized and scripted deployment are documented paths: docker/ and deploy/ both appear at the root. Configuration lives under conf/. Site source is under site3/, alongside the code rather than in a separate repository. A module named metadata-drivers/ suggests the metadata layer supports more than one backend, which is architecturally significant for operators who already run a particular metadata service in their stack.

Callers evaluating BookKeeper should note that deploying it means running a server process, not adding a library to an existing application. That operational weight is visible in the repository structure.

Update workloads, embedded use cases and teams new to distributed systems are outside the scope

BookKeeper is not the right storage layer for three categories of use case.

First, any workload that requires updating a stored record in place. The append-only design does not support overwriting an existing entry. A caller that needs to update a configuration value, modify a status flag, or change any stored record by key should use a key-value store or database instead.

Second, any case where the storage layer must be embeddable with no separate deployment. BookKeeper runs as a server. Repository entries include bin/ scripts, a docker/ directory and a deploy/ directory, all of which point to it being a service you run, monitor and maintain separately from your application. Embedding it into a single-process application is not the pattern it is designed for.

Third, any team without prior experience managing a distributed system. BookKeeper depends on a metadata service, runs across multiple nodes, and requires monitoring. It is a storage substrate for teams that already operate distributed infrastructure, not a drop-in component for teams new to that territory.

Release 4.18.1 on 2026-09-16 shows maintenance across two concurrent version lines

Three recent releases appear on the repository: 4.18.1 on 2026-09-16, 4.17.4 on 2026-07-28, and 4.18.0 on 2026-06-10. Two of those are patch-level updates to different minor versions, 4.17.4 and 4.18.1, meaning the project actively maintains at least two release branches at once. A caller on 4.17.x can receive bug fixes without upgrading to 4.18.x.

The last push to the repository was on 2026-10-03. BookKeeper is an active project by any reasonable measure.

Apache Software Foundation projects follow a defined release process. Releases require a community vote, and the foundation provides legal and infrastructure support. For enterprise teams, ASF governance is relevant: the project does not belong to a single company and cannot be unilaterally deprecated by a vendor. Bug reports go to the GitHub issue tracker at github.com/apache/bookkeeper/issues. The mailing list and Slack workspace described in the README are additional channels, but the issue tracker is where recorded bugs accumulate.

Apache-2.0 licence, three support channels, and no inline operational guidance

Repository metadata lists the licence as Apache-2.0. No commercial tier, dual licensing, or contributor licence exception appears anywhere in the README or the repository listing. For teams that need to redistribute a modified build, LICENSE at the repository root is the file to read.

Three support channels are documented in the README. User questions go to [email protected]. Design discussions and contributions use [email protected]. A Slack workspace, at the link in the README, provides faster direct contact with committers.

What the repository does not provide is an inline operational guide. No example configuration snippet appears in the README, no tuning parameters are documented there, and no deployment architecture diagram is embedded. Configuration files live in conf/, but reading them requires the online documentation to know what each key controls. Teams evaluating BookKeeper for a production system should plan for a non-trivial ramp-up period with the external site. As a storage substrate used by other major Apache projects, deployment examples from that ecosystem exist, but none are in the README.

Editorial conclusion

Use BookKeeper if you are building a system that needs fault-tolerant, low-latency durable writes with sequential read access: WAL for a metadata service, message persistence for a queue, or snapshot storage for a state machine. Avoid it if your workload requires in-place updates, random reads at low latency, or a storage layer that can be embedded without its own operational footprint. Before adopting it, work through the concepts page at bookkeeper.apache.org/docs/getting-started/concepts first, because the project says that starting there will help you fully understand the rest of the documentation.

Frequently asked questions

Official sources

1. apache/bookkeeper on GitHub
2. License: Apache-2.0
3. Project website
4. README
5. Releases