JavaFastPFOR compresses integer arrays fast when the values are not random

The assumption the library is built on

JavaFastPFOR is a library to compress and uncompress arrays of integers very fast, and the second sentence of the README is the whole design brief.

The assumption is that most, but not all, values in your array use much less than 32 bits, or that the gaps between the integers use much less than 32 bits. These sort of arrays often come up when using differential coding in databases and information retrieval, for example in inverted indexes or column stores.

That is a narrower claim than a general compression library makes, and it is the right one. A frame of reference encoder stores the minimum value of a block and then packs each value as the difference from that minimum, using only as many bits as the largest difference needs. A set of monotonically increasing document identifiers, or a column of small enum-like values, compresses by a large factor. A set of hashes, a set of random keys or a set of measurements does not compress at all, and no library can make it.

The README makes the theoretical point explicitly: random integers are not compressible, by this library or by any other means. If you ever had the means of systematically compressing random integers, you could compress any data source to nothing, by recursive application of your technique.

That paragraph is worth reading as a selection criterion rather than as a disclaimer. If your integers are genuinely high entropy, this library will not help you and neither will anything else, and the honest response is to store the data rather than to spend engineering time on it.

The library is a Java port of the fastpfor C++ library by the same author, and the topics list is simply compression and java.

The compress and uncompress interface

The API is deliberately low level, and the shape of it is worth understanding before you write any code against it.

A codec instance holds scratch state, and the two operations take offset wrappers rather than returning lengths directly:

        IntWrapper inputoffset1 = new IntWrapper(0);
        IntWrapper outputoffset1 = new IntWrapper(0);

        fastpfor.compress(data, inputoffset1, N, compressedoutput1, outputoffset1);
        int compressedsize1 = outputoffset1.get();

        int[] recovered1 = new int[N];
        inputoffset1 = new IntWrapper(0);
        outputoffset1 = new IntWrapper(0);
        fastpfor.uncompress(compressedoutput1, outputoffset1, compressedsize1, recovered1, inputoffset1);

The offsets are mutable wrappers, so the input offset advances as the codec reads and the output offset advances as it writes, and the caller reads the final position from the output wrapper. The uncompress call takes the compressed size as an argument, which is why the compressed size has to be captured from the output wrapper right after the compress call.

The caller allocates the output array. The README's example allocates int[N + 1024] for the compressed output of N values, which gives headroom for the codec's own framing rather than assuming the compressed form is smaller than the input.

The library supports compressing and uncompressing data in chunks, and the README points at the advancedExample in example.java for that pattern. Chunking matters for large arrays, since it lets you compress a stream in bounded pieces instead of holding everything in memory at once.

There is also a family of integrated codecs that assume the integers are in sorted orders and use differential coding, so they compress deltas. Those live in the package me.lemire.integercompression.differential, and most of the other codecs do not assume sorted input.

Where the codec comes from, and the vector path

The README is unusually forthcoming about the performance claim and about who has put the library into production, so both are worth reading rather than paraphrasing.

It states the library can decompress integers at a rate of over 1.2 billions per second, which is 4.5 GB/s, and that it is significantly faster than generic codecs such as Snappy and LZ4 when compressing arrays of integers. That is a claim about a narrow case, and the narrow case is stated: arrays of integers whose values or gaps fit in fewer than 32 bits. A generic codec is not competing on the same ground, since it has to work on arbitrary bytes.

The adoption list is more informative than the benchmark number. The library is used in LinkedIn Pinot, a realtime distributed OLAP datastore, part of it has been integrated in Parquet, a modified version is included in the search engine Terrier, it is used by ClueWeb Tools, and it is also used by Apache NiFi. It also inspired a compression scheme used in Apache Lucene and Apache Lucene.NET.

That lineage explains something about the design. This is not a general purpose compression library with a fast path; it is a specific codec family that has been embedded in columnar storage and search indexes, where the values stored are exactly the sorted identifiers and low-cardinality columns that it compresses well.

The Vector API path is the recent addition and it comes with a hardware caveat. The Java Team at Intel introduced the vector implementation for FastPFOR based on the Java Vector API that showed significant gains over the non-vectorized implementation, and an example usage is at examples/vector/Example.java. On aarch64, for example on Graviton3, you should use JDK 24 or later, because earlier releases lack the Vector API SVE intrinsics and run a fallback that is slower than the scalar codec.

That last point is the kind of detail that saves a benchmark from being misleading. Running the same library on Graviton3 under JDK 21 gives you the scalar path, so a comparison against a non-vectorised baseline would show no gain and look like the vector work had failed.

Adding it as a dependency

JavaFastPFOR is available both on Maven Central and JitPack, and the README gives both sets of coordinates.

From Maven Central, no extra repository configuration is needed:

<dependency>
    <groupId>me.lemire.integercompression</groupId>
    <artifactId>JavaFastPFOR</artifactId>
    <version>0.3.8</version>
</dependency>

In Gradle with the Groovy DSL:

dependencies {
    implementation 'me.lemire.integercompression:JavaFastPFOR:0.3.8'
}

From JitPack you need the repository block as well, and the version string takes the release tag form rather than a plain number:

<repositories>
    <repository>
        <id>jitpack.io</id>
        <url>https://jitpack.io</url>
    </repository>
</repositories>

<dependency>
    <groupId>com.github.fast-pack</groupId>
    <artifactId>JavaFastPFOR</artifactId>
    <version>JavaFastPFOR-0.3.8</version>
</dependency>

The README notes that you should replace version by the version you desire, and the versions in the examples are 0.3.8 while the releases listed for the repository are 0.3.12, 0.3.13 and 0.3.14 from June 2026, so the documented snippets are behind the latest tags.

There is also a demo project that uses JavaFastPFOR as a dependency through both Maven and Gradle, at the JavaFastPFORDemo repository, which is the quickest way to get a build working before you wire it into your own project.

The build itself is a plain Maven pom.xml at the root, alongside build.xml for Ant and a CHANGELOG. The Java CI badge points at the basic.yml workflow, and the example files include a directory of examples by Axelbrooke plus the vector examples.

Thread safety, requirements and the Kamikaze comparison

The thread safety section is short and the advice is unambiguous.

Some codecs are thread-safe while others are not. For this reason, it is best to use one codec per thread. The memory usage of a codec instance is small in any case.

There is a convention stated for the cases where you want to share instances anyway: unless the documentation of a codec specifies that it is not thread-safe, then it can be assumed to be thread-safe. So the safe default is that a codec is shareable, and the exception is the documented one. The per-thread advice is a fallback for the codecs where the documentation says otherwise, and the memory cost of one instance per thread is negligible because the instances are small.

The Java requirements are split by version. Releases up to 0.1.12 require Java 7 or better, and the current development versions assume JDK 21 or later. That second statement is the one to plan around, since it is about the development line rather than a historical note, and it pairs with the JDK 24 guidance for the aarch64 vector path.

The comparison section names a competitor directly. In the README's tests, Kamikaze PForDelta is slower than these implementations, with results in the benchmarkresults directory, including a file named benchmarkresults_icore7_10may2013.txt, and a reference link to the Kamikaze project. The benchmark file name dates that particular comparison to 2013 on an icore7 machine, which is old hardware by current standards, so the number is better read as a historical data point than as a current ranking.

The README also points to the benchmark tool for measuring this yourself, in the package me.lemire.integercompression.benchmarktools.

Where JavaFastPFOR fits, and where it stops

The comparison worth making is with a general purpose codec such as Snappy or LZ4, and with the built-in Java compression.

Against a general purpose codec, the difference is that this library operates on the integers themselves rather than on the bytes they occupy. A block of small values costs almost nothing once the values are bit-packed, because 4 bytes of zeros become 4 bits. A byte-level codec has to compress the pattern of zero bytes and cannot do it as cheaply, and it also has to spend time on headers and framing that the specialised codec does not need.

Against the built-in java.util.zip, the difference is the same one expressed as a loss of generality. Deflate works on any bytes and is fine for text, but it has no idea that your array is sorted identifiers with small gaps.

The limits are clear. The output is not a standard format, so anything you write with JavaFastPFOR can only be read by something using a compatible codec, and it is not a substitute for a storage format like Parquet or ORC. The codecs assume structured integer data, and the random case is explicitly not compressible. The API is low level, with manual buffer allocation and mutable offset wrappers, so it is a library for a storage engine rather than for application code.

The sorting assumption is a further constraint worth noting, because it applies only to the integrated codecs in the differential package. Those compress deltas and therefore need sorted input, while most of the other codecs do not.

And the memory model is the practical detail. Because the codec keeps internal scratch state, the idiomatic usage is one codec per thread, and the caller has to size the output array itself with headroom for framing.

For a columnar or index-building system where the integers are small or sorted and the throughput matters, this library is a well-proven component with production use in systems such as LinkedIn Pinot and Parquet behind it and a vector path for modern hardware. For a general purpose application that just needs to store some bytes, it is the wrong level of machinery.

The project is Apache-2.0 licensed, and the last push to the repository was on 2026-06-24, alongside the 0.3.14 release.

Editorial conclusion

JavaFastPFOR suits a Java system that stores integer columns or document identifiers and needs to compress them without a general purpose codec, since the codec family is designed around bit-packed frame of reference encoding and the library has a Vector API path that the README says showed significant gains. It is the wrong tool if your values are random, and the README is explicit that random integers are not compressible by this library or any other, and it is the wrong tool if you need a format other systems can read, since the output is an internal representation rather than a standard codec. Before adopting it, check the thread safety note in the README and use one codec per thread, and on aarch64 make sure you are on JDK 24 or later, because earlier releases lack the Vector API SVE intrinsics and fall back to a slower scalar path.

Frequently asked questions

Official sources

1. fast-pack/JavaFastPFOR on GitHub
2. Issues
3. License: Apache-2.0
4. README
5. Releases