ZeroFS: a log-structured filesystem that serves S3 buckets over NFS, 9P and NBD

The problem ZeroFS targets: POSIX semantics on top of an object store

Object storage gives you durability and a bill that scales with usage, but it gives you an HTTP API, not a filesystem. Applications that expect open, read, write, rename, fsync and directory listings need something in between. ZeroFS is that layer. It presents an S3-compatible bucket as a POSIX filesystem over NFS and 9P, and as a raw block device over NBD, with all three servers running in one userspace process.

The project's own framing is that it differentiates itself by being POSIX-conformant, reaching near-raw-S3 throughput on large files, scaling to hundreds of millions of small files, handling tens of thousands of requests per second, and requiring no external database service because everything is stored on S3. That last point is the one that changes operational shape. Many similar systems pair an object store with a separate metadata service you have to run, back up and monitor. ZeroFS puts the metadata in the bucket too.

The audience follows from that: platform teams running Kubernetes who want a CSI-backed volume, people who need a block device for a filesystem like ZFS, and anyone who wants to mount a bucket on a Linux box without writing an S3 client. The README lists Amazon S3, Google Cloud Storage, Azure Blob, any S3-compatible store, and local disk as backends, so the local-disk option also makes it usable as a test harness for the same code path.

Log-structured storage, segment reclamation and the metadata path

The name is literal. ZeroFS is a log-structured filesystem: writes are appended as segments rather than updated in place, which is the natural fit for an object store where you cannot rewrite a byte range of an existing object cheaply. The repository layout backs this up. There is a deterministic simulation test tree at zerofs/tests/dst whose README description covers data, namespace, segment-reclamation and repack paths, which tells you the storage engine has to reclaim space from old segments and rewrite live data forward.

That reclamation is the interesting part and also the part with the most failure modes. A log-structured design on top of S3 has to decide when a segment is garbage, how to compact it without losing acknowledged writes, and how to keep the accounting consistent if the process dies mid-repack. The project's CI description says the simulation runs recovery against byte-level and namespace reference models, a full metadata consistency scan, segment-accounting reconciliation, and an authoritative footprint scan, with crashes injected at arbitrary await points or narrow failpoint windows. One seed is one exact schedule, so a failure reproduces identically. That is a stronger testing story than most storage projects publish, and it is the part I would read before trusting the write path.

On the data path, extents are encrypted with XChaCha20-Poly1305 and the data key is wrapped via Argon2id. Compression is zstd or lz4 and happens before encryption, and the README states the codec is changeable at any time without migration. Caching has memory and disk tiers. For block access, NBD devices support TRIM, and the README says FLUSH and FUA replies return only after data is durable, which is the property that makes a filesystem like ZFS safe on top of it.

Installing ZeroFS from apt, dnf, the install script or Docker

There are three documented installation routes. The package repositories set up a systemd service and a config skeleton, which is the route to take on a long-lived host. The install script fetches a release tarball, verifies the published SHA-256 checksum and drops a prebuilt binary; it supports Linux (amd64, arm64), macOS (x86_64, aarch64) and FreeBSD (amd64). Docker is the shortest path to a running server.

On Debian or Ubuntu, the README gives these commands. They add the signing key, add the repository, and install the package.

curl -fsSL https://pkgs.zerofs.net/zerofs.gpg | sudo gpg --dearmor -o /usr/share/keyrings/zerofs.gpg
echo "deb [signed-by=/usr/share/keyrings/zerofs.gpg] https://pkgs.zerofs.net/deb stable main" | sudo tee /etc/apt/sources.list.d/zerofs.list
sudo apt update && sudo apt install zerofs

Fedora, RHEL and Rocky use the same repository through dnf.

curl -fsSL https://pkgs.zerofs.net/zerofs.repo | sudo tee /etc/yum.repos.d/zerofs.repo
sudo dnf install zerofs

After installation the package places a config skeleton under /etc/zerofs/ and a disabled systemd unit. You set ZEROFS_PASSWORD and credentials in /etc/zerofs/zerofs.env, set the [storage] url in /etc/zerofs/config.toml, then enable the service. The README points at packaging/README.md for the details, and that file is where I would look before putting this on a machine that matters.

If you would rather not install anything, generate a starter config with the container and then run it. The init subcommand writes the config to stdout when given a dash, so redirecting it produces a file you can edit.

docker pull ghcr.io/barre/zerofs:latest
docker run --rm ghcr.io/barre/zerofs:latest init - > zerofs.toml
$EDITOR zerofs.toml
docker run --rm -v "$PWD/zerofs.toml:/zerofs.toml" \
  ghcr.io/barre/zerofs:latest run -c /zerofs.toml

The container runs as UID 1001, and the README warns that a bind-mounted cache directory must be writable by that UID. To reach the servers from the host you bind addresses to 0.0.0.0 and map a port per enabled server: 2049 for NFS, 5564 for 9P and 10809 for NBD. Those three numbers also appear as EXPOSE lines in the repository Dockerfile, and the Dockerfile builds the binary from rust:1.92.0-slim-trixie before copying it into a debian:trixie-slim runtime image.

For a binary install, the flow is three commands: generate a config, edit in your S3 credentials, run it.

zerofs init            # Generate zerofs.toml
$EDITOR zerofs.toml    # Set S3 credentials
zerofs run -c zerofs.toml

The web UI is configured as another server block. The README shows uid and gid as required keys, since the browser's file operations need a POSIX identity to act as.

[servers.webui]
addresses = ["127.0.0.1:8080"]
uid = 1000  # POSIX identity for file operations from the browser; required
gid = 1000  # Required

The native 9P2000.L.Z kernel client and the mount fallback

The README is explicit that file access has two paths: the native kernel client when DKMS can install it for the running kernel, or zerofs mount as a fallback. The kernel/ directory holds an out-of-tree VFS module that speaks a private 9P2000.L.Z dialect over TCP or AF_UNIX, without FUSE or Linux v9fs, and it supports x86-64 and little-endian arm64.

This is where the deployment risk sits. An out-of-tree module has to be built against the running kernel, which means DKMS, matching headers, and a rebuild after every kernel upgrade. On a distribution kernel that works fine until it does not. The README does not document what happens to an existing mount when the module fails to rebuild, and it does not describe rollback. If you are on a platform where DKMS is not available, you are on the fallback path, and the fallback is a userspace mount, which is a different performance profile from the kernel client.

The testing claims are worth reading alongside this. CI runs pjdfstest, described as 8,662 POSIX cases per protocol (NFS, 9P and FUSE, with per-protocol exclude lists in .github/), plus xfstests over NFS, 9P and FUSE, a full Linux kernel build with make -j$(nproc) on each of those mounts, stress-ng file-handling stressors against live mounts, and a ZFS pool on ZeroFS block devices with a kernel source extraction followed by a scrub. Those are the right tests for this kind of system, and the exclude lists are the honest part: a POSIX conformance claim with per-protocol exceptions is a claim with known edges.

Where ZeroFS is the wrong tool

The first limitation is the licence. AGPL-3.0 is a strong copyleft licence, and the network clause means running a modified ZeroFS as a service triggers source distribution obligations. If you are embedding storage in a closed product, this is a conversation with your legal team before it is a conversation about throughput. Nothing in the README suggests a commercial exception, so treat the licence as the constraint it is.

The second is the failure model. A log-structured filesystem on object storage has to compact segments, and compaction competes with foreground writes for the same bucket. The deterministic simulation exists precisely because crash recovery during reclamation is hard. The README states that Jepsen local-fs kills the server mid-run and verifies recovery matches the last fsync, and that Jepsen HA runs a leader/standby pair under a nemesis that kills or pauses nodes with the requirement that no acknowledged write is lost, resurrected or corrupted across the tested failovers. Tested failovers is the operative phrase. High availability is optional and runs over the same bucket, so you are still depending on that bucket's consistency and availability for failover to work.

The third is workload fit. If you have one writer and no need for a network filesystem, an S3 client library or a local filesystem plus periodic sync is less machinery. ZeroFS earns its place when several clients need the same namespace, or when something needs a block device. For a single container writing logs to a bucket, it is a large dependency to take on.

Finally, the README does not document rollback or downgrade between releases. Releases are frequent, with v2.3.5 on 2026-09-18, v2.3.4 on 2026-09-17 and v2.3.3 on 2026-09-08, and the repository publishes no on-disk format versioning policy that would tell you what a newer binary does to an older bucket. Before upgrading a production bucket, confirm that a newer binary can read it and whether an older binary can read what the newer one wrote.

ZeroFS compared with JuiceFS and SeaweedFS

JuiceFS is the comparison people search for, and the difference is architectural. JuiceFS keeps metadata in a separate engine (Redis, TiKV, MySQL and others) and stores data chunks in the object store. ZeroFS stores everything in the bucket and requires no external database service, which removes a service to run and back up but puts metadata operations on the object store's latency profile. If your metadata engine is already operated and fast, JuiceFS gives you a place to tune it; ZeroFS gives you fewer moving parts and less to tune.

SeaweedFS appears in the search data and in the project's own testing, where Jepsen HA runs a leader/standby pair over SeaweedFS. That is a different relationship: SeaweedFS is an S3-compatible store in its own right, so it can be the backend ZeroFS sits on rather than a replacement for it.

SlateDB is the other name that comes up. It is a storage engine rather than a filesystem, and the connection here is conceptual: both are log-structured designs that put an LSM-style engine on object storage. ZeroFS is the layer above that turns the result into something mountable.

Against all of them, the specific ZeroFS claim to check is the one about small files. Scaling to hundreds of millions of small files is a metadata claim, and metadata is exactly what a bucket-backed design makes expensive. The CI description says the simulation checks namespace reference models and runs a full metadata consistency scan, which is the evidence the README offers. It is not a benchmark, and the README does not publish one for that case.

Maintenance, release cadence and licence cost

The repository is not archived, and the last push was on 2026-09-23. The release history shows three releases in September 2026, so the project is moving quickly. Fast releases are a maintenance cost as much as a benefit: you have to decide how often to upgrade a storage layer, and the README does not document a downgrade path.

Upgrade cost also depends on which access path you chose. The kernel module has to be rebuilt for each kernel, so hosts on a rolling kernel update cycle will exercise DKMS regularly. The userspace fallback avoids that but changes performance. The compression codec is the one piece the README says can be changed at any time without migration, which is a rare property and worth noting.

On licensing, AGPL-3.0 applies to the code in the repository. The README and the repository layout do not describe a separate commercial licence or a contributor licence agreement, and there is a CITATION.cff and a SECURITY.md at the top level but nothing that reads as a dual-licensing offer. If you distribute the binary or offer it as a network service, read the licence text itself rather than a summary. This is not legal advice; it is a pointer to where the constraint lives.

Editorial conclusion

Adopt ZeroFS when you already pay for object storage and want a POSIX or block interface in front of it without operating a separate metadata service; the packaging, Docker image and systemd unit make a first trial short. Skip it if you need a permissive licence in a closed product, or if your access pattern is a single writer with no HA requirement, where a plain mount or a database is simpler. Verify first that your kernel can load the out-of-tree 9P2000.L.Z module or that you are content with the zerofs mount fallback, and check the AGPL-3.0 obligations against how you ship the binary.

Frequently asked questions

Official sources

1. Barre/ZeroFS on GitHub
2. License: AGPL-3.0
3. Project website
4. README
5. Releases