Nimble: Meta's Columnar Format for Tables With Thousands of Columns

The Problem Nimble Targets: Wide Tables, Not Just Large Ones

Most columnar format discussions are about volume. Nimble's README frames the problem differently: width. It states the format is better suited to tables with thousands of columns or streams, and names feature engineering workloads and machine learning training tables as the typical case. That distinction matters because a table with 5,000 narrow columns stresses metadata handling in ways a table with 50 wide columns does not. Parquet and ORC store per-column metadata, and at that column count the metadata section itself becomes a scanning cost. Nimble's stated response is lighter metadata organization plus Flatbuffers instead of thrift or protobuf, on the argument that Flatbuffers gives more efficient access to large metadata sections. The README explicitly positions Nimble as a replacement for Parquet and ORC, so this is a format-level bet, not a library wrapper. The audience is narrow by design: engineers building storage layers for ML training pipelines or feature stores, working in C++, and able to tolerate an unstable format. Anyone storing a few hundred columns of business data is not the target.

Decoupling Encoding From Layout, and Why Cascading Encodings Matter

The central design claim in the README is that state-of-the-art data encoding evolves faster than file layout, so Nimble separates stream encoding from the underlying physical layout. Encodings can be extended by library users and applied recursively, which the README calls cascading or composite encoding. This is a different architectural choice from a fixed encoding list baked into a specification. In a format where the layout and the encoding are one thing, adding a new compression scheme means a new format version and coordination across every reader. Nimble's answer is to make the encoding a pluggable layer over a stable physical stream. The README also lists pluggable encoding selection policies, meaning the choice of which encoding to apply is itself an extension point rather than a hardcoded heuristic. Related to this, the format keeps a clear separation between logical and physical encoded types. One more stated design point is block encoding instead of stream encoding, which the README justifies as predictable memory usage during decode and read. That is a real trade-off, not a free win: block-oriented decoding bounds peak memory but can change how much work is done per read, and the README does not quantify the cost. The parallel design goal is aspirational. The README says SIMD- and GPU-friendly encodings are intended, and that exposing metadata to let developers plan decoding trees and schedule kernels is not implemented yet.

Metadata in Flatbuffers and the Single-Library Policy

Two decisions in the README are as much governance as engineering. The first is the metadata layer: Flatbuffers rather than thrift or protobuf, chosen for efficient access to large metadata sections. Flatbuffers allows reading fields without a full parse, which is the property that matters when metadata is spread across tens of thousands of column entries. The second is the unified policy. The README states that Nimble is more than a specification, that it is a product, and that developers are strongly discouraged from re-implementing the spec, to avoid what it calls environmental fragmentation issues observed in similar projects. It encourages bindings to other languages instead. This is a deliberate departure from how Parquet and ORC were adopted, where independent implementations in Java, C++, and Rust were the norm. Nimble's maintainers are trading ecosystem breadth for consistency. The practical consequence for an adopter is that there is one C++ library, and any language you use has to go through a binding that someone has to write. The README does not list existing bindings, and no releases were retrieved, so the maturity of that binding story cannot be confirmed from the material.

Getting It to Build: make, Bundle Overrides, and System Packages

The README gives a short path. Clone the repository and run make:

git clone git@github.com:facebookincubator/nimble.git cd nimble make

The CMake build is described as self-sufficient, able to locate its main dependencies or compile them locally. To force a dependency to be built locally, the README shows a variable override:

folly_SOURCE=BUNDLED make

The dependencies it says it will compile automatically are gtest, glog, folly, abseil, and velox. On Ubuntu 22.04 the README lists system packages needed first, including flatbuffers-compiler, protobuf-compiler, libflatbuffers-dev, libgflags-dev, libunwind-dev, libgoogle-glog-dev, libdouble-conversion-dev, libevent-dev, liblz4-dev, liblzo2-dev, libelf-dev, libdwarf-dev, libsnappy-dev, libssl-dev, bison, flex, libfl-dev, pkg-config, clang, and clang-format. Builds have been tested with clang 15 and 16. Note the weight of that list: the build pulls in folly, Velox, and their transitive system requirements before you can write a single Nimble file. There is no pip install, no single-header drop-in, and no released binary mentioned in the material.

The Velox Submodule Is a Hard Dependency Today

Nimble integrates Velox as a Git submodule pinned to a specific commit. The README states the codebase is closely coupled with Velox today and that the intent is to decouple them in the future, which is a clear admission that the coupling is a current constraint rather than a design goal. The README's badge shows the current commit and how far behind it is from Velox main, and a pre-commit hook validates that the SHA in the README matches the submodule. The documented upgrade procedure is manual:

git -C velox checkout main git -C velox pull git add velox

You advance Velox when your changes depend on code not in the current commit, or when the submodule falls too far behind. Then you build and run tests, and the pre-commit hook rewrites the compare link in the README. This is the maintenance cost in concrete form. Your build is tied to a specific Velox commit, and moving it forward means moving the whole submodule and re-validating. For a team already inside the Velox ecosystem this is acceptable. For a team that just wants to write and read columnar files, it means adopting Velox's build surface as well.

Where Nimble Is the Wrong Tool

The README is unusually direct about readiness: Nimble is a work in progress, many listed features are still under design or active development, and it provides no stability or versioning guarantees yet, with the instruction to use it at your own risk. That single sentence disqualifies it for any use where the file is a durable artifact. If you write Nimble files today and the layout or encoding set changes before a stable release, the README offers no compatibility promise to fall back on. There is also no retrieved release, so there is no tagged version to pin against. Beyond stability, the format is C++-first and the unified policy discourages independent implementations, so a Python or Java reader depends on a binding that the material does not enumerate. The wide-table optimization is also a specialization: if your data is a few dozen columns, the metadata advantage Nimble claims is not the constraint you have, and you would be paying the build and stability cost for a benefit that does not apply. Finally, the parallel and GPU story is explicitly not implemented. If kernel scheduling and SIMD-friendly decode are why you were interested, the README says that metadata exposure is still an intention.

Parquet as the Alternative, and the Actual Difference

The README names Apache Parquet as the format Nimble is meant to replace, so the comparison is fair to make. Parquet has a published specification and multiple independent implementations across languages, which is exactly the property Nimble's unified policy rejects. That means Parquet files written by one engine are generally readable by another, and you are not dependent on a single C++ library or its pinned submodules. Parquet also has a stable format with versioning, which is the guarantee Nimble explicitly withholds. The difference in approach is where each format puts its extensibility. Parquet extends through the specification and community implementations, so a new encoding is a spec change coordinated across projects. Nimble decouples encoding from physical layout and lets library users add and cascade encodings without touching the layout, at the cost of a single implementation. Parquet's per-column metadata is the thing Nimble targets with lighter metadata and Flatbuffers, though the README does not provide a measurement showing the gap. If your table has fifty columns and you need a format your Go service can read, Parquet is the straightforward answer. If you have thousands of streams and control both writer and reader in C++, Nimble's encoding model is the reason to look.

Licence and What to Verify Before Adopting

Nimble is licensed under Apache-2.0, and the README points to the LICENSE file in the repository. Apache-2.0 is a permissive licence with an explicit patent grant, which matters if you are embedding the library in a product. The README does not discuss the licences of the bundled dependencies (folly, Velox, abseil, and the rest), so if you build with BUNDLED sources you should check each one yourself. This is not legal advice. On verification, the material supports a short list. Confirm which encodings actually exist in the source tree, since the README says additional encodings can be added and that many features are still under design. Confirm the current Velox submodule commit builds against clang 15 or 16 on your distribution, because the system package list in the README is an Ubuntu 22.04 example and the coupling to Velox is acknowledged as temporary. Confirm whether any language binding exists, since the unified policy means you cannot rely on a third-party reader appearing. And confirm your own column count, because the width argument is the only reason the README gives for choosing Nimble over Parquet.

Editorial conclusion

Adopt Nimble only if you are working on wide tables (feature engineering or ML training data) and can accept an unstable format with no versioning guarantees, a C++-only library, and a build that pulls in Velox and folly. Do not adopt it if you need a stable on-disk format, multi-language readers, or a format with a published specification you can reimplement. Before committing, verify what the repository actually supports today: the README states many features are still under design, so check which encodings exist in the source tree and whether the current Velox submodule commit builds on your toolchain (clang 15 or 16).

Official sources

1. facebookincubator/nimble on GitHub
2. Issues
3. License: Apache-2.0
4. README