Pixels: a columnar storage engine for data lakes, plus its own query stack

A columnar format that targets the lakehouse storage layer directly

Pixels is described as the core of a columnar storage engine designed for data lakes and warehouses, optimized for analytical tables stored in on-premises and cloud-native storage systems. The storage backends named are S3, GCS, HDFS, Redis, HTTP through Netty, and local file systems, which is already a wider spread than most columnar formats claim. The primary language is Java, with JNI hooks into system calls and C and C++ libraries, and there is a separate C++ implementation of the format alongside the DuckDB integration in the cpp folder. The tree at the root shows how the work is divided, with modules for storage, cache, index, parser, planner, executor, daemon, server and cli, plus a proto directory for the wire and storage definitions and a flatbuffers directory. The design intent is that Pixels is not a query engine in itself but a format and storage layer that other engines read.

Connectors put Pixels inside engines you may already run

The integration story is the strongest part of the project and the clearest reason to look at it. Pixels integrates with DuckDB at 1.3.0, Trino at 405 and 466, StarRocks at 3.3.5, PrestoDB at 0.279 and Hive at 2.3 or above. The DuckDB integration and the C++ implementation live in this repository under the cpp folder, while the others are separate repositories: a Pixels connector for Trino, a Pixels connector for PrestoDB, a StarRocks branch with Pixels integration, and a Pixels SerDe for Hive. That structure means you can adopt the format incrementally, using the engine you already operate rather than replacing it. For a Trino shop, the path is to run the Pixels storage layer and add the connector. For a team that wants to query Pixels from DuckDB without running a cluster at all, the in-repository C++ path is the shorter route, and it is also the one that avoids the version pinning that comes with a full Trino deployment.

Pixels-Turbo and Pixels-Rover treat query urgency as a scheduling dimension

Pixels has its own query engine, Pixels-Turbo, which prioritises processing queries in an autoscaling MPP cluster, currently based on Trino, and exploits serverless functions such as AWS Lambda, vHive running on Knative, and Spike to absorb workload spikes. The stated goal is better performance and cost-efficiency for continuous workloads without giving up elasticity when a spike arrives. On top of that sits Pixels-Rover, a web-based query interface giving users serverless query processing, natural-language-to-SQL translation, and what the README calls flexible service levels in query urgency. A user can choose to execute a query immediately, within a grace period, or eventually, and Pixels-Turbo applies different resource scheduling and execution policies per urgency level, which in turn produce different monetary costs. That last point is the design's real content: latency is treated as a scheduling input with a price tag, rather than a fixed property of the queue.

Pixels-Retina replays CDC logs into the columnar tables at 10 ms freshness

Pixels-Retina is the real-time data synchronisation framework in the project. It replays data-change operations from log-based CDC sources as mirror transactions on the columnar table data, using a lightweight MVCC mechanism so that concurrent analytical queries keep running while new data lands. The README claims 10-ms-level data freshness and positions this as beating the batch-granular merge-on-read approach used by lakehouses such as Apache Iceberg and Paimon. The mechanism is worth understanding as well as the claim, because the two designs differ in where the cost lands. Merge-on-read keeps late data in its own files and reconciles at query time, which costs read amplification. Replaying change operations as transactions against the columnar data with MVCC moves the cost to the write path instead, which is a trade you accept when you need the data fresh rather than eventually consistent. This is the component to evaluate against your own ingestion pipeline, because that is where the freshness claim will be won or lost.

Building the Java tree needs JDK 8, Maven 3.8 and a correctly pointed PIXELS_HOME

The build instructions are concrete and there is one trap in them. JDK 8 or above and Maven 3.8 or above are required, with earlier Maven versions possibly working but not tested. You clone the repository into an SRC_BASE directory and build it, and the important detail is that the PIXELS_HOME environment variable must point at the installation directory of pixels, not at SRC_BASE. The clone and build sequence is:

git clone https://github.com/pixelsdb/pixels.git
cd pixels
# ensure PIXELS_HOME environment variable is set to the installation directory of pixels (not SRC_BASE).
export PIXELS_HOME=[pixels-install-dir]
mvn clean install

The build takes a couple of minutes and installs the library jars into the local Maven repository. Two executable jars are then produced and are what you actually deploy: pixels-daemon-*-full.jar in pixels-daemon/target, which runs the Pixels daemons, and pixels-cli-*-full.jar in pixels-cli/target, which is the command line tool. The integration modules build with Maven too, for example cloning pixels-trino and running mvn package in it.

Daily builds are the release story, and they are marked not for production

There is no stable release to point at. The release list holds a daily-latest tag, described as Pixels Daily Build 2026-10-19, and a proper v0.1.0 dated 2023-08-04. The daily build is described in the README as including the latest changes from the repository, suitable for testing and early feedback, and explicitly not recommended for production use. It does contain pre-built pixels-daemon and pixels-cli jars, so you can try the format without a source build. That is a reasonable evaluation path: pull the daily artifact, point a connector at it, and see whether the format behaves for your workload. It is not a production deployment story, and the gap between a 2023 tagged release and a daily build in 2026 means the version you evaluate is not the version anyone has pinned for years. The repository carries a CHANGELOG and a set of skills directory, and the licence field on the repository says AGPL-3.0, which is a significant constraint to read before you embed the engine in a product, since AGPL obligations reach network use, not merely distribution of the binaries.

IntelliJ development and the generated Protobuf sources

The project documents an IntelliJ workflow, which tells you something about how contributors are expected to work. You open SRC_BASE/pixels as a Maven project, wait for indexing and dependency download, then use the Maven plugin as an alternative to mvn package, run and debug unit tests, and debug Pixels itself through a Remote JVM Debug configuration. PIXELS_HOME must be set for the Maven plugin and the run and debug targets, and the README points at configuring it under Run, Edit Configurations, Edit Configuration Templetes for Maven, Junit and Application. One practical snag is documented in detail: in some IntelliJ versions the default idea.max.intellisense.filesize is too small for the source files generated by Protobuf, so the large generated file is treated as plain text in the interface. The fix is to set that property to 4096 or larger in Help, Edit Custom Properties, and restart IntelliJ. The README also notes that some JUnit tests access low-level JDK packages such as sun.nio and java.nio, which need an older JDK such as 1.8 to compile and run, though those tests are not required for the build itself.

Editorial conclusion

Pixels suits a team that already runs Trino, Presto, StarRocks or DuckDB and wants a columnar format of its own on top of S3, GCS, HDFS or local disk, and the Pixels-Retina freshness argument is the one to test first. It does not suit anyone who needs a stable, versioned release, because the newest tagged artifact is a daily build explicitly marked as not for production, and the last proper release was v0.1.0 in 2023. Before you commit, build the Java tree with mvn clean install on JDK 8 or above and Maven 3.8 or above, set PIXELS_HOME to the installation directory rather than the source directory, and benchmark the format against Parquet on your own queries rather than relying on the project's own comparison.

Frequently asked questions

Official sources

1. Issues
2. License: AGPL-3.0
3. pixelsdb/pixels on GitHub
4. README
5. Releases