TernFS is a distributed file system built for immutable training data

A file system shaped by immutable training data

TernFS was built for one workload, and the design is easier to judge once you know which one. The stated target is the kind of machine learning XTX does, and the two words that define the data are large and immutable. A file that does not need modifying after it is first created, and a storage estate where most of the space is taken by files bigger than a few megabytes. That combination removes most of the awkward cases a general purpose filesystem has to handle, and the architecture is a direct consequence of it.

The project is published by XTX Markets Technologies Limited under a 2025 copyright, and a high level description lives on the XTX Markets tech blog with the readme pointing to it for context before going into the codebase. There is also a project site at ternfs.com.

Two more workload assumptions shape the whole system. New directories are not created often, and files or directories are not moved between directories often. Every design decision about where state lives, how transactions are ordered and what the block layer does follows from those two sentences. A system that assumed frequent mutation would need a different metadata engine, a different write path and a different recovery story, and it would be a much larger piece of software.

The readme is written by engineers for engineers, with concrete numbers rather than adjectives, which is unusual and useful. It states the goals in the imperative, in terms of what must not happen rather than what should happen, and that framing is the clearest signal of how the system is meant to be operated.

The numbers the design is sized against

The readme gives the upper bound the system is expected to reach, and it is roughly the bound for the data planned for a single data centre:

- 10EB of logical file storage, counting the sum of all file sizes - 1 trillion files, with an average file size of about 10MB - 100 billion directories, averaging about 10 files per directory - 1 million clients

The intended hardware is commodity hardware and Ethernet networking, and the reason for that constraint is scale. At a million clients and a trillion files, any component that has to be consulted for every operation becomes the bottleneck, so the design pushes work outward. Metadata is split across many shards, data is spread across many block services, and the block layer is deliberately described as extremely dumb, with the blobs themselves as its only state.

The failure requirements are just as specific. Witnessing half-written files should be impossible, in the sense that a file is either fully written by its writer or not readable by other clients. A power loss or similar failure of storage or metadata nodes should not corrupt the filesystem, whether that means the metadata or the data. Corrupted reads from bit rot should be exceedingly unlikely, and so should data loss, unless there is a datacenter-wide catastrophic event such as fire, flooding or datacenter-wide vibration. The filesystem should keep working through the maintenance or failure of metadata or storage nodes. Deleted files and directories should be restorable, governed by a configurable permanent deletion policy.

The last goal is the one that pushes into architecture rather than engineering. TernFS has to offer the option of replicating to multiple regions, so compute can scale across data centres and no single data centre is a point of failure. That is a different system from a cluster file system, and it is why the metadata layer has a consensus component at all.

Registry, shards, a coordinator and a million disks

The component diagram in the readme is the fastest way to understand the topology, and the request arrows run from clients to every server:

  A ──► B means "A sends requests to B"

  ┌────────────────┐
  │ Metadata Shard ◄─────────┐
  └─┬────▲─────────┘         │
    │    │                   │
    │ ┌──┴──┐                │
    │ │ CDC ◄──────────┐     │
    │ └──┬──┘          │     │
    │    │             │ ┌───┴────┐
  ┌─▼────▼────┐          │ Client │
  │ Registry  ◄──────────┤        │
  └──────▲────┘          └─┬──────┘
         │                 │
  ┌──────┴────────┐        │
  │ Block Service ◄────────┘
  └───────────────┘

The registry is a single logical instance, the ternregistry C++ binary, and it stores metadata about one deployment: the addresses of the shards and the coordinator, the addresses of the block services and their storage statistics. Its own state lives in RocksDB, with five-node distributed consensus handled through LogsDB, and it replicates over UDP while serving TCP bincode request and response traffic.

The metadata shards are 256 logical instances of ternshard, also C++, each holding the file attributes of size, mtime and atime, the directory attributes and listings including names, the mapping from files to blocks and from blocks to block services. The cross directory coordinator is a single logical instance of terncdc, also C++, and it exists to sequence actions that span directories, such as creating a directory, removing one, or moving a file or a directory from one directory to another. It keeps very little state, mainly which transactions are running and which are queued, with transactions currently executed serially, plus a directory to parent directory mapping used for no-loops checks. It talks to the shards to carry the actions out and to the registry to register itself, fetch the shard list and report statistics.

The block services are where the volume is. There can be up to a million logical instances, one logical instance being one disk, with one physical instance handling every disk in the server it runs on. The binary is ternblocks, written in Go, with the readme noting it might become C++ later if performance requires it. Blocks are the blobs that hold file contents, a file is split into many blocks stored on many block services, and the only state is the blobs themselves: the job is to stream blobs efficiently from disks into TCP connections, with the registry used to register the instance and report free space and block counts. The two supporting documents named in the readme cover space accounting and XFS metadata reserves, and disk I/O concurrency limits.

Redundancy comes from spreading, not from replicating

The block service design is the part most likely to surprise someone coming from a RAID or replicated-disk background, so it is worth being explicit. A single block service instance is not redundant. The redundancy is handled by spreading a file over many instances, so that if up to four block services fail, the file can still be recovered.

That distinction has real consequences. There is no per-disk mirroring to reason about, and no rebuild to wait for, but the file is only recoverable as long as enough of its blocks survive somewhere. The readme points at the space accounting document for the XFS metadata reserves, which is the sort of detail that a storage system has to get right when the filesystem is managing the disks itself and the raw device is no longer simply a filesystem to the operator.

The metadata side takes the opposite approach to scale. Each shard is one of 256 logical instances with its own RocksDB store and a five-node LogsDB consensus group, so the metadata is replicated within a shard while the shard count spreads the load. The registry is a single logical instance with the same five-node consensus arrangement, and it is the component that knows where everything is, which makes it both the simplest part of the system to reason about and the one whose failure the readme is most careful to address.

The client side is deliberately thin. The web client, ternweb, is a Go binary running a TCP HTTP server with no state of its own, serving a web interface. The command line client, terncli, is a Go toolkit for the operational chores, most notably migrating the contents of dead block services with terncli migrate, and moving blocks around so that files are stored efficiently. That last task is a hint about what the system expects: storage is not automatically balanced, and a human or a script has to move blocks.

Version 0.X.Y means read the changelog every time

The versioning policy is documented as carefully as the architecture, and it is short enough to quote. TernFS is used in production, but the project is still moving quickly. Every release is in the 0.X.Y range and should be treated as pre-1.0. A change in X, for example from 0.4.5 to 0.5.0, may remove or change internal APIs, and the readme says plainly that upgrading across minor versions should not be skipped and that the changelog should be read. A change in Y, for example from 0.5.3 to 0.5.4, adds functionality or small configuration changes and may carry bug fixes.

In practice this means there is no version you can upgrade to and ignore. Every minor bump is a migration with reading attached, and a deployment that pins a version is pinning an API that the project reserves the right to change in the next minor release.

The release record so far is short. The single tagged release is v0.1.0, the initial release, dated 2026-02-09, while the last push to the repository was on 2026-10-14. That gap between a first tag and eight months of further commits is a fair description of where the project sits: the code is being worked on continuously, and the tagging has not kept pace with it.

The repository layout backs that reading. The tree separates cpp, go and kmod directories, with the C++ servers, the Go block service and clients, and kernel module work each in their own place, alongside docs, a build.sh, and the Python driven CI scripts ci.py, ci_nfs.sh and integration.py. There is a githooks directory and a LICENSES directory. A separate ci_nfs.sh script suggests interoperability with NFS is part of the test picture, which is the sort of check that matters for a system intended to sit behind existing tooling.

The licence question and the missing third party notices

Two things about licensing deserve attention, and they point in different directions. The readme carries a copyright line for XTX Markets Technologies Limited and an SPDX identifier of GPL-2.0-or-later, and the repository holds a LICENSES directory, which is where the actual text and any bundled third party notices live.

Against that, the licence the repository declares is not one GitHub can classify, so the platform's own licence field is empty. That is not necessarily a contradiction, since a project that puts its own licence file in a nonstandard place often leaves the classifier unset, but it does mean a legal review cannot rely on the badge. Read the LICENSES directory and the readme header together before you ship anything built on it.

There is a second question that GPL raises for a service of this kind, and it is worth raising even though the readme does not address it. GPL covers the distribution of the software. A company running TernFS internally, modifying it and operating the resulting service does not distribute it in the ordinary sense, and the obligations that attach to modification and internal use are different from the obligations that attach to shipping a binary to a customer. This is a question for whoever handles licence obligations in your organisation, not one to answer from a readme, and the point to take away is that the answer depends on whether you distribute the service or only run it.

The operational caveat is simpler. With a 0.1.0 release and a policy that reserves the right to change internal APIs in a minor version, the upgrade path is a reading task rather than a version bump, and that effort is part of the cost of running TernFS as your storage layer.

Where an ordinary filesystem fits better

The alternative is not another distributed file system, it is the local filesystem you already have, and the difference in approach is worth stating plainly because TernFS is a specialist tool.

A local or network attached filesystem gives you POSIX semantics that every tool expects, mutable files that can be edited in place, small files that are cheap, and no consensus layer to reason about. If your data set is a few terabytes of files you will modify tomorrow, a local filesystem plus a replication tool is simpler and faster than anything described here.

TernFS earns its complexity in a specific shape of problem. Large immutable files, up to ten exabytes of them, read constantly and written once, across commodity disks and ordinary Ethernet, with a million clients. There, the blocking of a file across many block services, the split of metadata across 256 shards and the serialised cross directory transactions are not overhead, they are the mechanism. The same properties are liabilities elsewhere. There is no story in the readme about small files, there is no mutation of file contents, and the no-loops check and the directory move path are designed for a hierarchy that changes rarely.

There is a third option worth naming for teams that do not want to run the system themselves: a managed object store with an API, where durability, replication and capacity are someone else's problem and your code speaks HTTP instead of a filesystem protocol. The trade is the opposite of TernFS's. You get a well known interface and an operational burden you do not carry, in exchange for the POSIX-like behaviour and the performance characteristics of a filesystem that keeps data on local disks.

Editorial conclusion

TernFS fits an organisation that stores up to ten exabytes of large, write-once data and wants the storage layer to spread across commodity disks and ordinary Ethernet, with replication across regions as a later step. It does not fit a workload of small mutable files, and it does not fit anyone who needs a stable API before version 1.0. Verify first that the version you pin is one you can upgrade from, that the block service and shard counts match the hardware you intend to run, and that the licence position is clear, because the repository carries a GPL-2.0-or-later header in the readme while GitHub cannot classify the licence itself.

Frequently asked questions

Official sources

1. Issues
2. README
3. Releases
4. XTXMarkets/ternfs on GitHub