cockroachdb/pebble: a Go LSM key-value store built for CockroachDB

Who Pebble is for, and who it quietly excludes

Pebble is a key-value store in Go that follows the LevelDB and RocksDB design: sorted string tables on disk, a log-structured merge tree, and compaction. The README is explicit that it is "focused on performance and internal usage by CockroachDB." That sentence is the whole positioning. It is not a general purpose RocksDB clone in Go, and it does not try to be.

The intended user is a Go program that needs an embedded storage engine and whose access pattern resembles CockroachDB's: point reads, range scans, prefix iteration, snapshots, and heavy writes that need concurrent compaction. If that describes your workload, the feature list in the README covers you. It includes block-based tables, checkpoints, indexed batches, iterator bounds and table filters, level-based compaction, manual compaction, a merge operator, prefix and table-level bloom filters, range deletion tombstones, reverse iteration, SSTable ingestion, single delete, snapshots, and range keys.

The exclusion list matters just as much. Pebble has no transactions, no column families, no backups, no FIFO or universal compaction styles, no persistent cache, no hash table or plain table SSTable formats, and no forward or tailing iterators. A team that picks Pebble expecting RocksDB parity will discover the gap after writing code, not before.

How the LSM is organised, and where Pebble diverges from RocksDB

Writes land in a memtable and a write-ahead log; flushes turn memtables into SSTables; background compactions merge those SSTables down through levels. That much is inherited. The differences are in the details, and the README lists them as advantages over RocksDB.

Reverse iteration is faster because the memtable skiplist carries backwards links. The commit pipeline is built for better concurrency. Indexed batches merge into iteration seamlessly, with batch mutations conceptually occupying another memtable level. L0 is split into sublevels so flushes and compactions out of L0 can run concurrently, which the README frames as reduced read amplification under heavy write load. File metadata lives in a copy-on-write B-tree, which the README describes as faster LSM edits when the LSM holds large numbers of SSTables. Delete-only compactions drop whole SSTables that fall inside the bounds of a range deletion. Block-property collectors and filters let iterators skip tables, index blocks, and data blocks that a user-defined property marks irrelevant.

Two of those are worth pausing on. Delete-only compactions and block-property filters are the kind of work you would otherwise write yourself on top of an engine, and both are exposed through the same Options struct you already pass to Open. The range keys API is the other one: it stores key-value pairs defined over a range of keyspace with user-defined semantics, interleaved during iteration. That is a real extension beyond the RocksDB format, not a rename of an existing feature.

Installing Pebble as a Go module and opening a database

There is no binary to install and no server to start. Pebble is a library, and the module path is github.com/cockroachdb/pebble. Add it to a Go module the usual way, then open a database directory.

The README's usage model is to construct an Options value and pass it to Open. The package documentation referenced from the README is at pkg.go.dev/github.com/cockroachdb/pebble, and Open, Options, and DB are the entry points named there. The README does not print a full program, so the shape of a first program is: build an Options, call Open with a directory name, check the returned error, and defer Close on the returned DB. The first run creates the directory and its files. The second run opens the same directory; if the on-disk format is older than the current version supports, Open fails rather than migrating silently, which is the behaviour described under format major versions.

Building from source is a different path. The repository has a Makefile whose default target prints usage rather than building anything, and the test targets are the documented entry points. The Makefile defines TAGS as invariants, and testrace adds -race with a 20m timeout:

TAGS := invariants

.PHONY: test
test:
	${GO} test -tags '$(TAGS)' ${testflags} -run ${TESTS} ${PKG}

.PHONY: testrace
testrace: testflags += -race -timeout 20m
testrace: test

Running make with no target prints the usage list rather than building, so make test is the first thing to type. For the metamorphic suite, stressmeta overrides PKG to ./internal/metamorphic and TESTS to TestMeta.

Format major versions are a one-way door

This is the part of Pebble that will bite a team that skims the README. Physical file formats change over time, and Pebble gates backwards-incompatible changes behind format major versions. When Pebble opens a database it defaults to the lowest supported version, which in v1 is FormatMostCompatible. Newer Pebble versions refuse to open databases in formats they no longer support.

You opt into a newer format by setting FormatMajorVersion on the Options passed to Open, or by calling DB.RatchetFormatMajorVersion at runtime. The README states plainly that format major version upgrades are permanent and that there is no option to return to an earlier format. The version table in the README lists names such as FormatMostCompatible, FormatVersioned, FormatSetWithDelete, FormatBlockPropertyCollector, and FormatSplitUserKeysMarked, along with the Pebble versions that support each.

So the operational question is not whether to ratchet, but when. Ratcheting early in a deployment is cheap. Ratcheting after you have written terabytes means every node in your fleet has to be upgraded in a coordinated way, and you cannot roll back by downgrading the binary. Treat the format major version as a schema migration for the storage layer, and schedule it like one.

RocksDB compatibility: forward only, and narrower than it sounds

Pebble v1 aims for forward compatibility with RocksDB 6.2.1, which the README identifies as the latest version of RocksDB used by CockroachDB. Forward compatibility means a database generated by RocksDB 6.2.1 can be upgraded for use by Pebble. Pebble v2 and newer do not support opening RocksDB databases at all. To move a RocksDB database to recent Pebble you must migrate it through format major version upgrades using previous Pebble versions.

Even inside v1, the compatibility is scoped to the subset of functionality and configuration CockroachDB uses. The README says the scope of RocksDB functionality is too large to test and document all incompatibilities, and then lists the known ones: WAL recycling is only compatible with RocksDB's kTolerateCorruptedTailRecords recovery mode; column families are unsupported and not even detected when opening a database that may contain them; the hash table and plain table SSTable formats are unsupported; and SSTable format versions 3 and 4 are unsupported, with the format version controlled by BlockBasedTableOptions::format_version.

The README carries a warning in bold: Pebble may silently corrupt data or behave incorrectly if used with a RocksDB database that uses a feature Pebble does not support. That is the honest framing, and it is the reason you should not treat Pebble as a drop-in replacement for an existing RocksDB deployment. If you are starting fresh, none of this applies. If you are migrating, the migration is the project.

When Pebble is the wrong engine

The clearest wrong-tool case is a workload that needs transactions. Pebble has none. If your application requires multi-key atomic commits with isolation guarantees, you either build that layer above Pebble or you choose a different engine. CockroachDB itself is the example of the first path: it is a distributed SQL database that uses Pebble as its storage engine, and the transaction layer lives above the storage layer.

The second case is column families. Pebble does not support them, and the README notes that it does not attempt to detect their usage when opening a database that may contain them. If your data model depends on RocksDB column families, Pebble is not a candidate.

The third case is anything that needs the RocksDB features on the exclusion list: backups, delete files in range, FIFO or universal compaction, forward and tailing iterators, memtable bloom filters, persistent cache, pinning iterator keys and values, sub-compactions, or SSTable ingest-behind. Some of these have workarounds. Backups, for instance, can be built from checkpoints, which Pebble does support. Others do not.

Finally, consider whether you need an embedded engine at all. Pebble is a library that lives in your process and owns a directory on local disk. If you need a network service with replication and failover out of the box, an embedded LSM is the wrong shape, regardless of how good the implementation is.

Licence, maintenance, and the cost of the upgrade path

Pebble is BSD-3-Clause. That is a permissive licence, and it does not carry the copyleft obligations of the GPL family, but this is not legal advice and the LICENSE file in the repository is the authoritative text.

The repository is not archived, and the last push was on 2026-09-22. Recent releases are v2.1.7 on 2026-08-24, v2.1.6 on 2026-05-27, and v2.0.9 on 2026-05-27. The v2.1.x and v2.0.x lines are being released in parallel, which means a team on v2.0.x has a decision to make about whether to move to v2.1.x. The README does not document a rollback procedure for a format major version upgrade, and it states that upgrades are permanent, so the upgrade path is the main maintenance cost you are signing up for.

There is a second cost that is easy to miss. Pebble's dependency list in go.mod is long for a storage library. It includes compression libraries (github.com/DataDog/zstd, github.com/golang/snappy, github.com/klauspost/compress, github.com/minio/minlz), hashing (github.com/cespare/xxhash/v2, github.com/zeebo/xxh3), data structures (github.com/google/btree, github.com/puzpuzpuz/xsync/v3, github.com/cockroachdb/swiss), and observability (github.com/prometheus/client_golang). Several are CockroachDB's own modules. That is normal for a database, but it means your dependency graph grows when you adopt it, and you inherit the update cadence of all of those.

The good news for testing is that the repository ships a deep test harness. The Makefile exposes race, address sanitizer, memory sanitizer, no-cgo, and metamorphic test targets, plus a lint target that runs as a Go test under ./internal/lint. You can run the same suite against your own fork, which is more than many embedded storage projects offer.

Editorial conclusion

Adopt Pebble when you are writing a Go service that needs an embedded LSM engine and you can live inside its feature set. Do not adopt it if you need transactions, column families, or a drop-in replacement for an existing RocksDB database, because v2 cannot open RocksDB files and the README states that unsupported RocksDB features can cause silent corruption. Before committing, verify your required features against the two lists in the README and check which format major version your data would sit at, since upgrades through RatchetFormatMajorVersion are permanent.

Frequently asked questions

Official sources

1. cockroachdb/pebble on GitHub
2. Issues
3. License: BSD-3-Clause
4. README
5. Releases