zfp: compressed floating-point arrays that you can still index

The unusual part is that the compressed array stays indexable

zfp is a compressed format for representing multidimensional floating-point and integer arrays, and it provides compressed-array classes that support high throughput read and write random access to individual array elements.

That is the claim that separates it from an ordinary compressor. A gzip or zstd stream has to be decompressed from the start, so reading one value out of the middle of a hundred gigabyte file means reading the file. zfp is arranged so that the compressed data is partitioned into blocks that can be located and decompressed independently, which is what makes random access into a compressed array possible.

The second capability is whole-array compression, serial and parallel using OpenMP and CUDA, and the documentation gives the use case directly: applications that read and write large data sets to and from disk. So the same format serves two access patterns, the read a few elements pattern and the read everything pattern, without switching format.

The data model is 4D arrays. The documentation is specific that zfp works best for 2D, 3D and 4D arrays that exhibit spatial correlation, and it names the cases: continuous fields from physics simulations, natural images, and regularly sampled terrain surfaces. zfp compression of 1D arrays is possible but generally discouraged.

That last sentence is the most useful single line in the README for an evaluator. Spatial correlation is what the compression exploits, and a 1D array has none to exploit. Anyone with a time series wondering whether to apply zfp to it has the answer before they start.

The project is primarily written in C and C++ but also includes Python and Fortran bindings, conforms to a long list of language standards from C89 through C++14, and is supported on Linux, macOS and Windows.

Lossy by default, with an error bound you choose

The compression mode is the first decision a user makes, and the documentation is clear about the default.

zfp uses lossy but optionally error-bounded compression to achieve high compression ratios. Bit-for-bit lossless compression is also possible through one of zfp's compression modes.

The error bound is the point. A general purpose compressor minimises bits while preserving the bit pattern, which for floating-point data throws away information in a way that is hard to reason about. zfp instead lets you state a tolerance, typically an absolute or relative error, and the compressor guarantees that the decompressed value differs from the original by no more than that. That turns compression into a controlled numerical operation, and it means a downstream computation can be sized to tolerate it.

The consequence is that the mode is not a runtime choice. It is recorded in the compressed header, and a reader has to know which mode to decompress with. For a data pipeline where a file is written once and read many times by different tools, that is a real constraint, and it is the kind of thing that is easier to settle before you have a petabyte of files than afterwards.

There is also an intermediate option between the two extremes, which the Python packaging description names: it supports serial and parallel compression of whole arrays using both lossless and lossy compression with error tolerances. So the choice is not binary between perfect and approximate, it is a bound you pick.

For scientific work this is the selling point. A field solver that produced its output at a given tolerance, a visualisation pipeline that only needs the data to look right, and an archive that has to fit on a tape all want the same thing, a smaller file with a known error, and zfp is one of the few formats that offers it as a first class mode rather than as a lossy image codec that happens to work on numbers.

Building it: CMake with tests, or GNU make without CUDA

The quick start is short and there are two supported build systems, and the documentation is honest that they are not equivalent.

To download and build with CMake:

git clone https://github.com/LLNL/zfp.git
cd zfp
mkdir build
cd build
cmake ..
cmake --build . --config Release
ctest

That builds the zfp library in the build/lib directory and the zfp command-line executable in the build/bin directory, and then runs the regression tests. The full test suite can be run by enabling the BUILD_TESTING_FULL CMake option during the build step.

The GNU make route is shorter:

cd zfp
make
make test

But the documentation attaches a caveat to it: GNU builds are less flexible and do not support all available features, with CUDA support named as the example. So if you need the GPU path, the CMake route is the one to use, and that is a decision to make before the first build rather than after.

The Makefile itself explains how the build is organised, and the shape of the tree is the map. The top level includes a Config file that holds the compile-time settings, and the Makefile includes it, then dispatches into subdirectories per component: src for the C library, cfp for the C++ wrapper, fortran for the Fortran bindings, utils for the command-line utilities, tests for the test suite and examples for the sample programs. The all target walks each of them, and the test target runs the regression tests in the tests directory.

For a first build, the examples are the fastest way to see what the library does. The examples directory contains a simple.c program, an array.cpp for the C++ array classes, array2d.hpp, a chunk.c example, inplace.c, iterator.cpp and iteratorC.c for the iterator interfaces, diffusion.cpp and diffusionC.c for a small simulation, pgm.c and ppm.c for image formats, and speed.c for timing. The image examples are the interesting ones for a first look, because a PGM or PPM file is something you can open and look at.

One container, four language bindings, and a large set of packaging routes

The distribution story is unusually wide, and the README lists it explicitly. zfp may be installed using package managers such as apt, brew, conda, dnf, pacman, pip, port, spack, vcpkg and yum, and the documentation is linked for further installation instructions.

That list is itself informative. The presence of spack and vcpkg tells you the HPC and Windows C++ worlds, conda tells you the scientific Python world, and port tells you the older MacPorts route. This is a library that has been packaged for wherever its users happen to build.

The Python side is a compiled extension rather than a pure Python binding, and the setup.py shows how it is built. The package is named zfpy, the version is 1.0.1, and the extension is built from python/zfpy.pyx, which means it is a Cython module. The build requires numpy and cython, the include path comes from numpy's own include directory, and the extension links against the zfp library with library directories pointing at build/lib64 and build/lib/Release, so the C library has to be built first. The pyproject.toml build system requires setuptools, wheel, cython and numpy, with the numpy version pinned to the oldest supported one for Python versions below 3.9.

The Fortran bindings live in their own fortran directory, built as a separate library when BUILD_ZFORP is enabled in the Config file, and the C++ wrapper in cfp when BUILD_CFP is enabled. The C++ classes are the ones the documentation leads with, and the Fortran and Python bindings are the ones that make the library usable from an existing scientific code base without rewriting it in C.

The authors are named as Peter Lindstrom and Danielle Asher, and the project originated with Peter Lindstrom at Lawrence Livermore National Laboratory, with the project home at zfp.llnl.gov.

The right way to cite zfp, and what the papers cover

The documentation asks scholarly users to cite a specific paper, and this is worth reading because the two papers describe different things about the same library.

The first is Peter Lindstrom's Fixed-Rate Compressed Floating-Point Arrays in IEEE Transactions on Visualization and Computer Graphics, volume 20 number 12, December 2014. The title is the design. Fixed-rate is the mode where the compressed size is a known function of the uncompressed size and the tolerance, so you can compute the storage cost of your dataset before you compress it and know exactly what you are getting. That is a different contract from a general purpose compressor, which promises a ratio and delivers whatever it delivers.

The second is the current algorithm, described in Error Analysis of ZFP Compression for Floating-Point Data by James Diffenderfer, Alyson Fox, Jeffrey Hittinger, Geoffrey Sanders and Peter Lindstrom, in SIAM Journal on Scientific Computing, volume 41 number 3, June 2019. This one is about the error behaviour rather than the format, which is the paper to read if you need to know what the bound means in practice for your data.

The distinction matters because the first paper describes the design of the format and the second describes the algorithm as implemented in the current version, and the documentation says so directly. A project with a decade of continuous development and two papers in the venues it did publish in is a project with a research history rather than a product history, and the citations are the record of that.

The documentation itself is extensive and published on Read the Docs in HTML and PDF form, with a separate algorithm page describing the implementation in the current version, an installation page, and an examples page for calling the C library and using the C++ array classes. The repository has the same documents in a docs directory along with a CHANGELOG.md, a CITATION.cff file, a SUPPORT.md, a CONTRIBUTING.md and a SECURITY.md.

The release record is where the caution goes, and it is worth noticing. The recent releases are 1.0.1 on 2023-12-16, 1.0.0 on 2022-08-01 and 0.5.5 on 2019-05-06. The last push to the repository was on 2026-10-13, so work continues, but the last tagged release is from December 2023 and the repository's default branch is develop rather than the release branch, which the documentation badges point at as release1.0.1.

Where zfp is the wrong tool

Three situations rule it out, and one of them is the library's own guidance.

The first is 1D data. The documentation says compression of 1D arrays is possible but generally discouraged, and gives the reason in the surrounding sentence: zfp works best for arrays that exhibit spatial correlation. A time series, a log file or a table of measurements has no spatial neighbourhood to predict from, so the compression ratio collapses towards what a general purpose codec would give you while you also take on the error bound and the non-portability of the format. For that data, a standard codec is the right answer.

The second is data that must round-trip exactly and is integer. The default mode is lossy, and the lossless mode exists but is a different mode with a different header. If your data is a lookup table, a financial record or anything where a single bit matters, the sensible default is lossless, and at that point zfp's advantage over a conventional compressor is much less clear for 1D data and comes mainly from the random access property.

The third is a workflow that cannot accept a custom file format. A zfp file is not readable by anything except zfp, by design, because the whole point is a compressed representation that no general purpose decoder understands. If your data ends up in a tool that expects NetCDF, HDF5 or a plain array, then zfp is a layer you would have to put in front of that tool rather than replace it with it.

There is also a version question to settle early. The last tagged release is 1.0.1 from 2023, the default branch is develop, and the release documentation is versioned as release1.0.1. That is a library whose format is stable, which is good, but it also means the release line and the development line are separate and you need to know which one you are building against. The CHANGELOG.md in the repository is where that difference is recorded.

Editorial conclusion

zfp fits a scientific computing team that writes large simulation or field data to disk and later wants to read scattered elements out of it without paying for a full decompression, and it fits anyone who needs a controlled error bound rather than a best-effort ratio. It is the wrong choice for 1D data, where the documentation says compression is possible but generally discouraged, and it is the wrong choice for integer data that must round-trip exactly unless you use the lossless mode, because the default mode is lossy. Before adopting it, decide which mode you need, since the mode is fixed in the compressed header, and check your toolchain, because the GNU make build is documented as less flexible and does not support CUDA.

Frequently asked questions

Official sources

1. License: BSD-3-Clause
2. llnl/zfp on GitHub
3. Project website
4. README
5. Releases