Icechunk gives Zarr on object storage database semantics

Why Zarr on object storage needs a transaction layer

Icechunk is an open-source, Apache 2.0, transactional storage engine for tensor and ND-array data designed for use on cloud object storage, and it works together with Zarr rather than replacing it. Zarr is an open source specification for storing multidimensional array data, and it defines the metadata that describes arrays, their shape and dtype, plus the way arrays are chunked and compressed into raw bytes. Zarr can store its data in any key-value store, and there are many implementations of Zarr in different languages. That generality is exactly the gap Icechunk fills. Icechunk exposes a key-value interface to Zarr and manages all of the actual I/O for getting, setting and updating metadata and chunk data in cloud object storage, so the Zarr libraries never need to know how it works underneath. The README is explicit that Icechunk only supports Zarr Python right now, and it asks anyone interested in implementing support for another implementation to open an issue. That single sentence tells you where the project is in its lifecycle.

Serializable isolation is the feature everything else hangs on

The key improvement Icechunk brings over regular Zarr is consistent serializable isolation between transactions, and the consequence stated in the README is that Icechunk data is safe to read and write in parallel from multiple uncoordinated processes, which allows Zarr to be used more like a database. Read that carefully, because it is a strong claim and it is the one that justifies the whole project. Plain Zarr on object storage is a key-value store with a chunk layout, so two writers touching the same array can interleave their updates and produce a state neither intended. Icechunk makes writes atomic and never partially visible, and reads always use a committed snapshot, so no locks are needed for reading. Alongside that sit time travel, since previous snapshots remain accessible after new ones have been written, and data version control with tags as immutable references to snapshots and branches as mutable ones. The remaining core requirements are chunk sharding, where chunk storage is decoupled from specific file names and multiple chunks can be packed into a single object, chunk references, where Zarr-compatible chunks inside other formats such as HDF5 or NetCDF can be referenced, and schema evolution, where arrays and groups can be added, renamed and removed with minimal overhead.

A repo is the unit of consistency, not the whole bucket

The core entity in Icechunk is a repository, or repo, and this is where the design guidance matters for anyone deploying it. A repo is a Zarr hierarchy containing one or more arrays and groups, and it functions as a self-contained Zarr store. Formally a repo can be any valid Zarr hierarchy, from a single array to a deeply nested structure of groups and arrays, and the most common scenario is a single Zarr group with multiple arrays, each corresponding to a different physical variable but sharing common spatiotemporal coordinates. The guidance to users is to scope repos only to related arrays and groups that require consistent transactional updates. That is a smaller unit than a dataset, and it is the unit of atomicity. A commit inside a repo is atomic, and two repos are two separate worlds. The practical consequence is that a team with a 40-variable model should think hard about which variables must move together, because putting everything in one repo buys atomicity across all of it at the cost of larger commits and a wider contention surface, while splitting it up means a reader that needs two repos can observe them at different points in time.

Nine crates, layered from types up to the Python bindings

The Rust workspace is organised into layered crates, and the dependency order is visible in the workspace manifest.

[workspace.package]
rust-version = "1.95.0"
edition = "2024"

The crate list in the README describes the stack. icechunk-types holds the shared foundational types, Path, ETag, Move and error wrappers, used across all crates. icechunk-format is the binary format types and serialization, covering snapshots, manifests, transaction logs and repo info. icechunk-storage defines the storage traits and common storage utilities. icechunk-arrow-object-store is a backend built on Apache Arrow's object_store, covering in-memory, local, GCS and Azure. icechunk-s3 is a native AWS S3 backend behind an optional feature. icechunk is the core storage engine holding transactions, version control and repositories, and it is the default member of the workspace. icechunk-python exposes the engine to Python through PyO3 bindings, and icechunk-macros provides procedural macro helpers for tests and internal use. The layering means you can depend on the format crate alone if you are writing a different implementation of the storage contract, and the repository also carries an icechunk-js/ directory even though the README's crate table does not yet list a JavaScript crate. The workspace pins a Rust version of 1.95.0 and edition 2024, so this is not a codebase that builds on an old toolchain.

Object storage with no catalog beside it

One of the core requirements is that the format is designed around the consistency features and performance characteristics available in modern cloud object storage, and that no external database or catalog is required to maintain a repo, though it also works with file storage. That single line is the architectural commitment, and it cuts both ways. On the good side there is no second system to operate, no database to back up, and no catalog to fall out of sync with the data, which removes an entire class of failure that plagues metadata-driven stores. On the other side, everything the engine needs to know has to live in the object store itself, which is why the format crate has as much to carry as it does, with snapshots, manifests, transaction logs and repo info all serialized. The README's own tip banner points out that Icechunk 2.0 is released, with better consistency, performance and reliability for tensor storage, which is the kind of change that suggests the pre-2.0 behaviour was not what you would want to run in production. The release history supports the reading that this is still moving quickly, with v2.2.1 and v2.2.2 both on 2026-09-17 and v2.3.0 on 2026-10-06.

The test environment is a local S3, and it is in the repo

The compose file at the repository root shows how the project tests itself, and it is a nice illustration of what it means to take object storage seriously in a test. It brings up a RustFS container as an S3 compatible server, with a healthcheck on the health endpoint, an entrypoint that creates a test bucket, and two published ports, 4200 for the API and 4201 for the console.

  rustfs:
    container_name: icechunk_rustfs
    image: rustfs/rustfs:1.0.0-rc.6

A second container then configures that server, waits for the healthcheck, creates aliases and users, and attaches a read-only policy that allows s3:GetObject on the test bucket, plus a second user with modify rights. The reason to point this out is that the test setup exercises the permission model, not just the happy path. A read-only user and a modify user with different policies is how you check that the engine respects the credentials the caller supplies, which is the kind of bug that only appears once real users share a bucket. The image pins a release candidate version of RustFS, and the compose file is a development aid rather than a deployment recipe, since the engine itself is what talks to your object store in production.

Where Icechunk is the wrong choice

The limits are clear and they are worth stating before anyone builds a platform on this. The first is the Zarr Python limitation. The README says outright that Icechunk only supports Zarr Python, and anyone using Zarr in Rust, C, Julia or JavaScript is out of luck until the project says otherwise, which is why the request to open an issue if you want to implement support is there. The second is that the specification is still a specification. There is an Icechunk Spec in the documentation, and a spec is a contract that can still change, so anyone depending on on-disk format stability across versions should read that document before committing to it. The third is the cost of the consistency guarantee itself. Serializable isolation means commits, versions, manifests and transaction logs in the object store, and a workload that writes a single small chunk and never collaborates is paying for machinery it does not need, when a plain Zarr store on the same bucket would be simpler and cheaper. The fourth is organisational rather than technical. This is a project from Earthmover, with its own community Slack, its own contributor guide and a single mainline, so a team that needs long-term support guarantees should read the contributor guide and community channels the project provides rather than assume more support exists than does.

Editorial conclusion

Adopt Icechunk when several uncoordinated processes need to write the same set of Zarr arrays on object storage and you want commit semantics, history and branching without standing up a database, especially in Python where the PyO3 bindings are the supported path. Skip it if your stack is not Zarr Python, because the project states that as its only current implementation, and skip it if you need a fully specified storage contract, since the format is still published as a spec rather than a finished standard. Before your first commit, decide what a repo should contain, because the guidance is to scope repos only to arrays and groups that need consistent transactional updates, and read the FAQ on the site for the concurrency and conflict cases the README does not cover.

Frequently asked questions

Official sources

1. earth-mover/icechunk on GitHub
2. License: Apache-2.0
3. Project website
4. README
5. Releases