facebook/zstd: the reference Zstandard implementation and its dictionary mode

What Zstandard solves, and who is actually the audience

The README states the target plainly: a fast lossless compression algorithm aimed at real-time scenarios, at zlib-level ratios or better. That sentence defines the audience. If you are choosing a codec for log shipping, database page compression, network payloads, or an archive format where decompression happens on the request path, this is the class of tool you are shopping in. The repository is the reference implementation, written in C, and it also ships a command line utility that produces and decodes .zst files and can handle .gz, .xz and .lz4 as well.

The format itself is the more important artifact. RFC8878 documents it, and the README notes that multiple independent implementations already exist. That matters for adoption: you are not betting on one vendor's binary format, and a Java, Go or Rust service can interoperate with a C producer as long as both follow the RFC. The README points to a list of known ports and bindings on the Zstandard homepage for projects that need another language.

Who is this not for? Someone who wants a compression library with a small, self-contained API surface and no interest in tuning. Zstd exposes a large number of parameters, and the README spends real space on the speed-versus-ratio curve. That flexibility is the point, but it is also work.

The speed and ratio curve, and why decompression barely moves

The README publishes a comparison table run with lzbench on a Core i7-9700K under Ubuntu 24.04, on the Silesia corpus. At level -1, zstd 1.5.7 reports a ratio of 2.896 at 510 MB/s compression and 1550 MB/s decompression. For scale, zlib 1.3.1 -1 in the same table reports 2.743 at 105 MB/s and 390 MB/s. brotli 1.1.0 -1 reports 2.883 at 290 MB/s and 425 MB/s. Those are the project's own numbers from its own README, not an independent audit.

The more interesting column is the negative levels. The README explains that --fast=# trades ratio for speed: --fast=1 reports 2.439 at 545 MB/s and 1850 MB/s, and --fast=4 reports 2.146 at 665 MB/s and 2050 MB/s. Compare that with lz4 1.10.0 at 2.101, 675 MB/s and 3850 MB/s. The overlap is narrow and worth understanding. At --fast=4, zstd is slightly better on ratio and slightly slower on compression than lz4, but lz4 decompresses roughly twice as fast in this table.

The README makes a structural claim that explains a lot: decompression speed is preserved and stays roughly the same at all settings, a property it attributes to most LZ compression algorithms such as zlib or lzma. If that holds for your data, the practical consequence is that you can raise the compression level for storage without paying for it on the read path. That is the real argument for zstd over a fixed-level codec, and it is testable on your own corpus in an afternoon.

Building the reference implementation with make

The README is explicit that make is the main build system and the reference one, and that other build systems are periodically updated to stay compatible. It warns that small drifts and feature differences can be present because perfect synchronization is difficult, and asks you to prefer make when your build system allows it. That is an unusually candid note, and it tells you where to look first if a CMake build behaves differently from a make build.

From the repository root, the default target builds a release library and a release binary. The Makefile defines default as lib-release followed by zstd-release, and the subdirectories involved are lib, programs, tests and zlibWrapper.

make

After that finishes you have a zstd binary and a library. The Makefile also defines a broader all target that additionally builds examples, manual and contrib, and an allmost target that skips zlibWrapper. The comment in the Makefile notes that zlibWrapper is skipped in that path because it cannot be built on alternate architectures without the proper zlib installed, which is a useful hint if a cross-compile fails.

The README's dictionary how-to is the shortest path to seeing the CLI in action, and it is reproduced in the next section. If you would rather read code than run a binary, the examples directory contains simple_compression.c, simple_decompression.c, streaming_compression.c, streaming_decompression.c and streaming_memory_usage.c, which map onto the common integration shapes.

Dictionary training is the part that changes the small-data math

The README devotes a section to what it calls the case for small data compression, and the reasoning is stated directly: compression algorithms learn from past data, and at the beginning of a new data set there is no past to build on. The smaller the input, the worse this gets. That is a general property of the technique, not a zstd quirk.

Zstd's answer is training mode. You supply a few samples, one file per sample, and get back a dictionary file that must be loaded before both compression and decompression. The README's worked example uses a github-users sample set of roughly 10K records at about 1KB each, and reports that the dictionary improves ratio while also making compression and decompression faster.

zstd --train FullPathToTrainingSet/* -o dictionaryName
zstd -D dictionaryName FILE
zstd -D dictionaryName --decompress FILE.zst

The three commands above are copied from the README's dictionary how-to: train, compress with -D, decompress with -D. The constraint the README states is the one people miss: there is no universal dictionary. Training works when the samples share some correlation, and the more data-specific the dictionary, the more efficient it is. Deploying one dictionary per type of data is described as the way to get the greatest benefit. Dictionary gains are mostly effective in the first few KB, after which the algorithm falls back on previously decoded content.

So the honest framing is this: dictionary mode is a schema-specific optimization, not a general speedup. If your records are heterogeneous, or you cannot assemble a representative sample set, training will not save you, and you should evaluate plain zstd on the actual data before designing a pipeline around dictionaries.

Where zstd is the wrong tool

The clearest wrong-tool case comes from the same small-data section. If your payloads are small and you have no correlated sample set, you are in the regime where compression is hardest and dictionary training has nothing to learn from. The README says this outright rather than hiding it, and it should shape your evaluation.

The second case is raw decompression throughput. In the README's own table, lz4 1.10.0 decompresses at 3850 MB/s against zstd's 1550 MB/s at level -1 and 2050 MB/s at --fast=4. If your bottleneck is decompression cycles and your ratio requirement is modest, lz4 is the better fit and zstd's tuning range will not close that gap. The README does not argue otherwise.

The third case is maximum ratio at any cost. The README acknowledges that a few other algorithms produce higher ratios at slower speeds and fall outside its graph, pointing to a separate image for slow modes. If you are archiving cold data where throughput is irrelevant, that is a different design point, and the README treats it as out of scope rather than claiming victory everywhere.

Finally, note the licence shape. The repository is dual licensed under BSD or GPLv2, and the Makefile header states that you may select, at your option, one of the two. That choice is yours to make with your own counsel, but the file layout means you should read LICENSE and COPYING together rather than assuming a single permissive grant.

Zstd against xz and lz4: three different design points

The related searches around this project pair it with xz and with lz4, and the README's own table is enough to separate them without guessing.

LZ4 is the speed-first option. In the same lzbench run, lz4 1.10.0 reports a ratio of 2.101 at 675 MB/s compression and 3850 MB/s decompression. Zstd at --fast=4 reports 2.146 at 665 MB/s and 2050 MB/s. So at the fast end the two are close on compression and ratio, and lz4 wins decisively on decompression. The difference in approach is that lz4 essentially offers one operating point, while zstd offers a dial from negative levels through level -1 and far beyond.

xz sits at the other end. The README places the slow, high-ratio algorithms outside its main graph and links to a separate chart for them, which is a fair summary of the split: xz is built for ratio, zstd is built for real-time scenarios. The README also notes that zstd's decompression speed stays roughly flat across settings, which is not the trade you get from a ratio-maximizing codec.

The practical test is not which algorithm wins a corpus benchmark. It is whether your workload is dominated by write cost, read cost, or storage size. Zstd's value is that you can move along that axis with a flag rather than swapping libraries, and the format stays the same across the whole range.

Maintenance, upgrades and the licence decision

The repository is not archived, and its last push was on 2026-09-18. The most recent release listed is v1.5.7 from 2025-02-19, preceded by v1.5.6 in March 2024 and v1.5.5 in April 2023. The gap between v1.5.6 and v1.5.7 is roughly eleven months, so release cadence is not rapid, and the README's benchmark table is labelled zstd 1.5.7, meaning the published numbers track the current release rather than the development branch.

The format is the reason this cadence is tolerable. Because RFC8878 fixes the wire format and independent implementations exist, a slow release cycle does not strand you: a decoder written against the RFC keeps working across library upgrades. The upgrade risk is concentrated in the library API and the CLI flags, not in the bytes on disk.

The build-system note from the README is also an upgrade cost. Make is the reference, and other build systems drift. If your project builds zstd through CMake or another path, a version bump may require checking whether the non-make build has caught up, and the README explicitly declines to promise synchronization.

On licensing, the repository is dual licensed under BSD or GPLv2, with LICENSE and COPYING as the two files. The Makefile header repeats that you may select one of the two at your option. Which one you take depends on how you link and distribute, and that is a question for your own legal review, not something the README resolves for you. What the README does make clear is that the choice is deliberate and documented.

Editorial conclusion

Adopt facebook/zstd when you need a stable, RFC8878-documented format with a C reference implementation and a CLI that already reads .zst, .gz, .xz and .lz4. Skip it when your data is a few hundred bytes per record and you have no representative training set, because a generic dictionary does not exist and the small-data gains depend on correlation in your own samples. Before committing, build with make from the repository root, confirm whether your build system tracks the Makefile, and read the LICENSE and COPYING pair to decide which of the two licences you are taking.

Frequently asked questions

Official sources

1. facebook/zstd on GitHub
2. Issues
3. Project website
4. README
5. Releases