# FastLanes: a columnar file format designed around decoding speed

> FastLanes is a C++ big data file format from the CWI that replaces generic block compression with data parallel encodings, so that decoding stays fast and SIMD friendly. It ships Python, C++ and Rust surfaces, a PDF format specification, and no tagged releases, with the default branch named dev.

**cwida/FastLanes** — Next-Gen Big Data File Format

- Repository: https://github.com/cwida/FastLanes
- Website: https://fastlanes.io/
- Stars: 718 · Forks: 41
- Language: C++
- License: MIT
- Published: 2026-10-21 · Updated: 2026-10-21 · Language: en
- Canonical page: https://hysenlabs.com/projects/cwida-fastlanes

## Encodings instead of generic block compression

FastLanes is presented as a next generation big data file format, and the README leads with a claim of being like Parquet but with 40 percent better compression and 40 times faster decoding. The details of how it gets there are in the 2025 paper The FastLanes File Format, whose abstract is quoted in the README and which is the better source for the design.

The core idea is to avoid generic compression methods such as Snappy in favour of lightweight encodings that are fully data parallel. Compression ratio is improved by cascading encodings through a flexible expression encoding mechanism, and the same mechanism enables multi-column compression, which exploits correlations between columns and is described as a long time weakness of columnar storage. A two phase algorithm finds those encoding expressions during compression.

Two other design points matter for anyone building on top of it. The API supports partial decompression, so an engine can execute queries on compressed data without expanding the whole file, and the format is designed for fine-grained access at the level of small batches rather than row groups, with the stated aim of keeping the decompression memory footprint within CPU and GPU caches.

The implementation is portable, auto-vectorising C++. The README describes the format as fully SIMD friendly with zero explicit SIMD instructions, and the project has zero external library dependencies, which is a meaningful constraint when you are deciding whether to vendor it into a data engine.

## A format born in database research, not in a product team

The publication list is the clearest signal of what kind of project this is. The 2023 PVLDB paper on the FastLanes compression layout describes decoding more than 100 billion integers per second with scalar code, followed by a DaMoN 2024 paper on accelerating GPU data processing with FastLanes compression, the SIGMOD 2024 ALP paper on adaptive lossless floating point compression, and a DaMoN 2025 paper rethinking lightweight encodings for GPUs.

The through-line is the CWI database systems group, and the same names appear across the papers and the package metadata, with the Python package author recorded as Azim Afroozeh at cwi.nl. The ALP work is significant because adaptive lossless floating point compression is the kind of encoding that matters for sensor and telemetry data, where the value distribution matters more than the byte pattern.

For an adopter this background cuts both ways. The plus is that the design decisions are documented in peer reviewed venues with reference implementations linked from the README, including the data parallelised encodings code, the separate FastLanesGPU repository, the ALP repository and the G-ALP repository. The minus is that this is a research project with an active publication programme, so the format may keep evolving, and the README itself asks users to cite the work, which is what research software usually does.

The site at fastlanes.io is the project home, and the repository itself carries a CHANGELOG.md, a CONTRIBUTING.md, a convention.md and a CLAUDE.md alongside the code.

## Three language surfaces over the same C++ core

The API is a pipeline of builder style calls, and it is exposed through Python, C++ and Rust. The Python binding, pyfastlanes, starts with a connection and a call chain that reads a CSV directory and writes a FastLanes file:

```python
import pyfastlanes

# Connect to FastLanes
conn = pyfastlanes.connect()

# Convert a CSV directory to FastLanes format
conn.inline_footer().read_csv("path/to/csv_dir").to_fls("data.fls")

# Read back and write to CSV
reader = conn.read_fls("data.fls")
reader.to_csv("decoded.csv")
```

The C++ surface mirrors it, and the C++ example in the README is the shortest form of the whole API:

```cpp
#include "fastlanes.hpp"

int main() {
    fastlanes::Connection conn;
    conn.read_csv("data/csv_dir").to_fls("data.fls");

    auto reader = fastlanes::Connection().read_fls("data.fls");
    reader->to_csv("decoded.csv");
    return EXIT_SUCCESS;
}
```

The Rust binding follows the same shape, and it is added to a Cargo manifest as a path dependency:

```toml
[dependencies]
fls-rs = { path = "./rust" }
```

```rust
use anyhow::Result;
use fls_rs::connect;

fn main() -> Result<()> {
    let mut conn = connect();
    conn.inline_footer()
        .read_csv("data/csv_dir")
        .to_fls("data.fls");

    conn.read_fls("data.fls")
        .to_csv("decoded.csv");
    Ok(())
}
```

The inline_footer call in the Python and Rust examples is the interesting asymmetry. It appears in one path and not the other, which suggests the footer handling is configurable rather than fixed, and the format specification is where that question is answered. The examples directory holds a C API example, a C++ example, a Python example, a Rust example and a fetch content example, so the C API exists alongside the three language bindings.

## Building it: FetchContent, CMake and a Makefile with many includes

For C++ users the README recommends pulling the project in with CMake FetchContent:

```cmake
include(FetchContent)
FetchContent_Declare(
        fastlanes
        GIT_REPOSITORY https://github.com/cwida/FastLanes.git
        GIT_TAG dev
)
FetchContent_MakeAvailable(fastlanes)

add_executable(example example.cpp)
target_link_libraries(example PRIVATE FastLanes)
```

The GIT_TAG is dev, the same default branch the repository uses, which is a detail worth pausing on if you were planning to vendor this into a build.

The Python package is a native build. The pyproject.toml uses scikit-build-core with a CMake and Ninja toolchain, pybind11 for the bindings and setuptools_scm for versioning, and it sets two CMake arguments, FLS_BUILD_PYTHON=ON and FLS_ENABLE_VERBOSE_OUTPUT=ON. The project name is pyfastlanes, the version in the file is 0.1.3.post9, and the required Python is 3.8 or later. Build wheels are configured through cibuildwheel with a pytest command as the test command.

The Makefile at the root is a convenience layer over a set of includes, with mk/python.mk, mk/cpp.mk, mk/rust.mk, mk/data.mk, mk/embeddings.mk, mk/flatbuffers.mk, mk/quick_fuzz.mk and mk/header_check.mk among them, plus a default goal of help. The presence of a fuzzing target and a header check target is a good sign for a C++ library that other people will compile.

## GPUs are the destination, but the CUDA reader is still coming

The GPU story is the reason to care about this format, and it is also the least finished part. The README lists CUDA support for the FastLanes CUDA reader under a Coming Soon heading, which means the GPU path described in the DaMoN 2024 and 2025 papers is not yet in the repository as a released component. The related source code lives in separate repositories instead, with FastLanesGPU and the G-ALP work, both linked from the publication list.

The design intent is clear even so. The format is described as suitable for modern data-parallel execution, SIMD or GPU, and the paper on accelerating GPU data processing with FastLanes compression and the G-ALP paper on rethinking lightweight encodings for GPUs are separate pieces of work on the same problem. ALP, the adaptive lossless floating point compression scheme, is a sibling project with its own repository, and it is the kind of codec that a GPU can apply without a serialising dependency chain, which is exactly what a data parallel format needs.

The mechanism that ties it together is the expression encoding mechanism. Because the choice of encoding is described by an expression that can cascade, the same file can be decoded with different strategies depending on the hardware available, and multi-column compression is just another expression that spans columns. That is a more flexible design than a fixed per-column codec, and it is the reason the format claims to be both more compact and faster to decode.

For a team without GPU work in scope, the practical story is simpler: a CPU reader that is fast because the encodings are data parallel, and a partial decompression API that lets a query engine touch only the data it needs.

## No tagged releases, a dev default branch and a PDF specification

The distribution story is where this project differs from a product. There are no GitHub releases, the default branch is dev, and the last push to the repository was on 2026-04-22. The version numbers in the tree are development versions, with the Python package at 0.1.3.post9 and the Rust binding added as a path dependency rather than fetched from a registry, which tells you the same thing from the other direction.

Against that, the project is unusually well specified. There is a typing system document at docs/typing_system.md and the file format specification as a PDF at docs/specification.pdf, and both are linked from the README as the documentation. The 2025 paper in the citation block describes the format in detail, including the two phase encoding expression search and the fine-grained access model.

The source tree shows the machinery behind that. There are flatbuffers_schemas for the metadata layer, an fsst directory and a galp directory, alongside a publications directory with the paper code, a fls_gen generator directory, a benchmark directory, a test directory, a data directory, a cuda directory, a log directory, a publications directory with the paper code, and separate python, rust, pyfastlanes and toolchains directories. A.clang-format and a.clang-tidy at the root and a CMakeLists.txt complete the layout.

So the honest summary is a research implementation with a real specification, a language binding story and a paper trail, but without the release discipline that would make it safe to depend on without pinning to a commit hash.

## Conclusion

Adopt FastLanes if you are building an analytical engine or a data pipeline where decompression throughput, not just compression ratio, is the bottleneck, and you are willing to work with a format that has no tagged releases and a dev branch as its default. Do not adopt it as a drop-in Parquet replacement in a stack that expects mature ecosystem tooling, because the project publishes no GitHub releases, and the CUDA reader is still listed as coming soon. Verify first that your workload benefits from the fine-grained access model, since the design avoids row groups in favour of small batches, and read docs/specification.pdf and docs/typing_system.md before committing, because the file format is a research project with a specification rather than a stable standard.

## FAQ

### What is the FastLanes file format?

FastLanes is a big data file format written in portable, auto-vectorising C++. Instead of generic compression such as Snappy it uses lightweight data parallel encodings, cascaded through an expression encoding mechanism that also enables multi-column compression, and it is designed for fine-grained access on small batches rather than row groups.

### How do I use FastLanes from Python?

Import pyfastlanes, create a connection, then chain read_csv on a CSV directory and to_fls to write a FastLanes file, and use read_fls followed by to_csv to decode it back. The package is built with scikit-build-core, CMake, Ninja and pybind11, and the project name in pyproject.toml is pyfastlanes.

### How do I build FastLanes in a C++ project?

The README recommends CMake FetchContent, declaring the repository with GIT_TAG dev and calling FetchContent_MakeAvailable, then linking the FastLanes target. There is also a Makefile with build, install and test targets that pulls in separate include files for the C++, Python, Rust, data and fuzzing steps.

### Does FastLanes support GPU reading?

Not in the main repository yet. The README lists CUDA support for the FastLanes CUDA reader under Coming Soon, and the related GPU work lives in separate repositories, including FastLanesGPU and the ALP and G-ALP projects described in the SIGMOD 2024 and DaMoN 2025 papers.

## Sources

- [cwida/FastLanes on GitHub](https://github.com/cwida/FastLanes)
- [Issues](https://github.com/cwida/FastLanes/issues)
- [License: MIT](https://github.com/cwida/FastLanes/blob/dev/LICENSE)
- [Project website](https://fastlanes.io/)
- [README](https://github.com/cwida/FastLanes/blob/dev/README.md)

---

Hysen Labs editorial analysis, written from the project's own repository and release notes. Cite the canonical page: https://hysenlabs.com/projects/cwida-fastlanes