TurboPFor: an integer compression library built around SIMD codecs

A codec library, not a general purpose compressor

TurboPFor is a library of integer compression codecs, written in C with C++ headers, and the README describes the goal bluntly: it is the synonym for integer compression.

The scope is large. There is a family of PFor style codecs, including a novel TurboPFor scheme with direct access and SIMD or AVX2 support, plus RLE. There is bit packing, with a SIMD implementation the README claims runs at more than 20 billions integers per second, and with direct or random access that lets a single packed entry be read with zero decompression. There is variable byte encoding, with a scalar implementation and a SIMD group varint called TurboByte, and a hybrid called TurboBitByte that the README says combines the two fastest codecs. There is Variable Simple, Elias Fano, TurboVLC for large integers with an exponent and a variable bit mantissa, and binary interpolative coding.

Underneath those sit the transforms: delta, zigzag, zigzag of delta and XOR, in scalar and SIMD forms, plus transpose and shuffle operations in 2D, 3D and 4D, and general purpose codecs layered on top including lz4, zstd, turborc with a range coder, and bwt.

The point of a codec library rather than a single format is that you pick the codec per data type. Sorted integers, random identifiers, timestamps and float series behave differently, and the library treats that as a choice the caller makes.

Floating point and time series get their own codecs

Two domains get dedicated treatment rather than being bolted onto the integer path.

For floating point, the README lists delta and zigzag, an improved gorilla style compression, and the Finite Context Method in both FCM and DFCM variants. On top of that sit TurboFloat, described as a novel floating point compression using TurboPFor, and TurboFloat LzXor, a lempel-ziv variant. Float16 support is there, and the README mentions float 16, 32 and 64 bit quantization with a variable quantization bit size, plus a point wise relative error bound for lossy compression.

The transform list also includes a lossy floating point route that uses TurboPFor or TurboTranspose with lz77 and bwt, which is a different way to trade accuracy for size than dropping mantissa bits.

For time series, the library offers what the README calls the fastest Gorilla implementation in 16, 32 and 64 bit styles, using zigzag of delta plus RLE. The claim attached to it is that it can compress timestamps to only 0.01%, with compression speed above 10 GB/s and decompression above 13 GB/s. The 0.01% figure refers to timestamp columns specifically, and is a statement about that data rather than about a general compression ratio.

The practical reading is that the library's centre of gravity is high volume numeric data in a columnar or time oriented store, where throughput matters as much as the ratio.

Direct access into compressed inverted indexes

The inverted index section is where the design decisions have consequences beyond compression.

The library provides direct access to compressed frequency and position data with zero decompression. It introduces an intersection mechanism with skip intervals, which the README says decompresses only the minimum necessary blocks, around 10 to 15% of them, and implicit skips with zero extra overhead. It also describes an efficient bidirectional inverted index architecture allowing forward and backwards traversal, including the integer compression itself.

The performance claims here are specific enough to be useful as targets rather than as facts: more than 2000 queries per second on the GOV2 dataset of 25 million documents on a single core, and more than 7000 queries per second on a simple quad core PC with parallel query processing. The README even strikes out the Map Reduce, Hadoop and multi-node cluster comparison in favour of the single machine figure.

Whether that holds for your index depends on things the README cannot know: your posting list lengths, your selectivity, your memory bandwidth and your query mix. Zero decompression random access matters because a search engine touches a small number of postings per query, so a codec that must decode a whole block to read one value wastes the work that vectorisation bought.

icapp is the benchmark, and the README shows its numbers

The README does not describe an installation procedure for the library. What it does describe is a benchmark application, IcApp, available from the project's releases, presented as a new benchmark for TurboPFor that tests almost all integer and floating point file types.

The synthetic data test is the one with a reproducible command line:

./icapp -a1.5 -m0 -M255 -n100M ZIPF

The flags correspond to a zipfian skewed distribution, and the README explains why this shape of test is chosen: unlike general purpose compression, integer compression generally uses a small fixed block size such as 128 integers, while large blocks are involved in the queries that inverted indexes, search engines, databases, graphs and in memory computing need to decode entirely.

The results table in the README is a single thread comparison with columns for compressed size, ratio, bits per integer, compression throughput in MB/s and decompression throughput in MB/s. The TurboPFor256 row is shown at 5.04 bits per integer with 2369 MB/s compression and 10950 MB/s decompression, and TurboPFor128 at 5.07 bits per integer with 1359 and 7803. The table also lists PFor variants, Simple16, VSimple, QMX, Simple-8b and the TurboByte plus TurboPack combination.

These are the project's own numbers on one machine, dated 2019.11 in the table header, and the README links a long list of benchmark issues in the repository for other comparisons, including against streamvbyte, Quantile Compression, zfp and blosc.

Rust and Java bindings, and a hole in the platform list

The library is C, but the README advertises access from other languages as a first class feature. There are Rust bindings, described as giving access to TurboPFor including the SSE, AVX2 and Neon paths from Rust. There are Java bindings through critical natives and JNI, described as letting Java call into TurboPFor as fast as calling from C.

The platform list is broad: AMD and Intel, 64 bit ARMv8 NEON on Linux and macOS including M1, and Power9 Altivec, on Linux amd64, arm64, Power9 and macOS. The README also lists a full range of 8, 16, 32 and 64 bit scalar functions plus 16, 32 and 64 bit SIMD functions.

One platform is explicitly carved out, and it is worth reading carefully if you are porting. The README states that riscv64 currently uses the existing generic or scalar path and does not enable x86 SIMD or AVX2 code paths unless a dedicated RISC-V backend is added. So on RISC-V the library still works, but the vectorised codecs, which are the reason to use it, are not the ones you get.

The repository layout supports the multi-language story. The top level holds include/ for the headers, lib/ for the implementation, java/ and rust/ for the bindings, a makefile and makefile.vs for the two build systems, a vs/ directory for Visual Studio, and a .gitmodules file. There is also a build workflow badge for Ubuntu in the GitHub Actions directory.

Licence, release tags and how to treat the performance claims

The licence is GPL-2.0, with a LICENSE file at the repository root. That is the most consequential practical detail in this project for a commercial product, because GPL-2.0 obligations attach to a derivative work distributed as part of a larger program, and the library is intended to be linked into other software. Read the licence text and take your own view, or ask whoever owns that decision.

The release history is unusual. The tags are named by year and month, and the listed releases are 2023.06 dated 2023-06-07, 2023.05 dated 2023-05-16 and 2023.04 dated 2023-04-09, with the 2023.06 release described as TurboPFor-Integer-Compression plus icapp for Windows and Linux. Meanwhile the repository is not archived and the last push was on 2026-10-08. So the code keeps moving while the published tags, by the names visible here, have not advanced past mid 2023. If you need a specific point in time, pin a commit rather than a tag.

Finally, the tone of the README deserves a warning. It contains claims such as no other integer compression compresses or decompresses faster and more efficient than any other fast integer compression scheme. Those are the author's own comparative statements, and the benchmark list they point to is the right place to look if you need evidence, since the numbers are reproducible with the bundled tool rather than asserted in prose. The library's actual quality is best judged by running icapp on your own data, which is exactly what the project gives you to do.

Editorial conclusion

TurboPFor fits a systems programmer who needs to compress integer columns quickly and is willing to work at the codec level, especially for inverted index payloads where random access into compressed data matters more than the ratio. It is the wrong choice if you need a general purpose compressor, if you cannot accept the GPL-2.0 licence in your product, or if you are on RISC-V, since the README states riscv64 currently uses the generic scalar path. Before adopting it, build it with the included makefile on your own target and measure it with the icapp benchmark rather than relying on the numbers in the README.

Frequently asked questions

Official sources

1. Issues
2. License: GPL-2.0
3. powturbo/TurboPFor-Integer-Compression on GitHub
4. README
5. Releases