# DatenLord: cloud native storage that hides where the data lives

> DatenLord is a Rust distributed storage system built for the case where applications and storage have been separated into different clusters, so the cost of moving data over the network dominates. It hides data location from applications, caches hot data across local, neighbour and remote layers, and targets multi cloud and multi data centre availability.

**datenlord/datenlord** — DatenLord, Computing Defined Storage, an application-orientated, cloud-native distributed storage system

- Repository: https://github.com/datenlord/datenlord
- Stars: 924 · Forks: 97
- Language: Rust
- License: MIT
- Published: 2026-10-20 · Updated: 2026-10-20 · Language: en
- Canonical page: https://hysenlabs.com/projects/datenlord-datenlord

## The argument against separating compute from storage

The central claim of the project is that the modern separation of computing and storage into dedicated clusters has become a performance liability. The reasoning is presented in two parts.

The first part is about hardware. The README notes that storage related functionality in the Linux kernel has not changed much in the last ten years, a period in which the hard disk drive was the main storage device. Now solid state drives are mainstream, and the latency numbers it quotes run from roughly 1 to 10 ms for a hard disk, to 50 to 150 microseconds for a solid state drive, to about 25 microseconds for NVMe, to 350 nanoseconds for non-volatile memory. Traditional blocking synchronous IO in the kernel becomes very inefficient against those figures, and context switch overhead stops being negligible next to a 25 microsecond device.

The second part is architectural. Isolating computing and storage does simplify maintenance, because you can upgrade, expand or migrate each cluster independently. But data sets have grown, and the example given is an AI training job that takes one hour to load data while the training itself finishes in 45 minutes. With storage behind a network, applications must also know where the data is and what it costs to reach.

DatenLord's answer is to abstract the heterogeneous storage details away so that applications can assume all data is local, with the system accessing it on their behalf.

## Caching hot data in local, neighbour and remote layers

The high availability story is the reason the system exists, and it is built on data locality. The scenario is the online business that cannot afford downtime and therefore deploys across multi cloud, hybrid cloud and multiple data centres. Duplicating all data everywhere in a timely manner is not feasible because of the data volume, so a single cloud or data centre failure can still take the business down through data unavailability rather than compute unavailability.

The mechanism is a set of large scale distributed and automatic cache layers that buffer hot data. The data is large overall, but the hot portion involved in online business is usually of limited size. DatenLord uses that locality to build local cache, neighbour cache and remote cache layers, and applies different caching strategies depending on the target application, with least recently used caching named for some kinds of random access and most recently used caching for some kinds of sequential access.

Two properties are called out as the benefit. The system is transparent to applications, meaning no modification to application code is needed, and it is automatic, caching by data hotness rather than by explicit user instruction.

There is a second consequence beyond the cache. Because DatenLord knows the exact location of all cached data, it can help Kubernetes schedule jobs close to that data, which turns the cache index into a placement hint for the scheduler.

## Building from source with protobuf, CMake and an etcd client

The crate is datenlord at version 0.1.0, edition 2021, with the MIT licence and keywords that name the target environment directly.

```toml
edition = "2021"
license = "MIT"
keywords = ["Storage", "CSI", "K8S"]
```

The dependency list shows the shape of the system. There is an async runtime in tokio with the full feature set, and a smaller smol runtime pinned to version 1.2.4 with a comment stating that its version should match the etcd client used, alongside etcd-client at 0.11. The gRPC stack is grpcio with protobuf-codec and default features disabled, and protobuf and serde_xml_rs handle serialisation in both binary and XML forms.

The kernel facing code uses nix with the fs, ioctl, signal, user, mount and socket features, so mount management and namespace work are part of the design rather than an afterthought. Observability comes from prometheus and from the tracing and tracing-subscriber pair, and hyper with server, http1 and tcp features serves HTTP.

One dependency is not from crates.io at all. The ring-io crate is pulled from a Git repository at a pinned revision, which means a build needs network access to that repository or a vendored copy.

The repository also carries a rust-toolchain.toml, a rustfmt.toml, a .typos.toml, a config.toml at the root, and a Dockerfile_with_mirror alongside the plain Dockerfile.

## The container image and how the binary is laid out inside it

The Dockerfile shows a two stage build. The builder stage uses a rust image whose version is an argument defaulting to latest, installs cmake, g++, libprotobuf-dev and protobuf-compiler with apt, and runs a release cargo build.

```dockerfile
ARG RUST_IMAGE_VERSION=latest
FROM rust:${RUST_IMAGE_VERSION} as builder
WORKDIR /tmp/build
COPY . .
RUN apt-get update && apt-get install -y cmake g++ libprotobuf-dev protobuf-compiler
RUN cargo build --release
```

The runtime stage is ubuntu:22.04 rather than a distroless or scratch base, which is worth noticing if you care about image size or attack surface. Three artefacts are copied in: the release binary at /usr/local/bin/datenlord, a umount-in-container.sh helper, and the datenlord-entrypoint.sh script, and the entrypoint is set to the latter with an empty command.

```dockerfile
FROM ubuntu:22.04 as datenlord
LABEL maintainers="DatenLord Authors"
LABEL description="DatenLord Distributed Storage"

COPY --from=builder /tmp/build/target/release/datenlord /usr/local/bin/datenlord
COPY --from=builder /tmp/build/scripts/setup/umount-in-container.sh /usr/local/bin/umount-in-container.sh
COPY --from=builder /tmp/build/scripts/build/datenlord-entrypoint.sh /usr/local/bin/datenlord-entrypoint.sh
ENTRYPOINT ["/usr/local/bin/datenlord-entrypoint.sh"]
CMD []
```

The separate umount helper and the entrypoint wrapper tell you that container lifecycle handling, including cleaning up mounts, is a first class concern in this project rather than something bolted on later.

## Where DatenLord is the wrong choice

The design has a clear cost profile, and it is worth being honest about who does not benefit.

On a single machine with local disks, the cache layering buys nothing. The whole argument rests on data being far away and network latency dominating. Running the daemon there adds a daemon, an etcd dependency and a mount layer for no gain, and a local filesystem or a plain block device is the better answer.

The project also assumes a substrate it does not provide. Kubernetes appears throughout the description, from scheduling jobs near cached data to the CSI keyword in the crate metadata and the k8s-openapi dependency pinned to the v1_19 feature set, and etcd is used for coordination. On a cluster without etcd, or with a Kubernetes version far from that era, the deployment assumptions need checking before anything else.

Finally, maturity. The crate version is 0.1.0, the repository publishes no GitHub releases, so there is no published version history to pin against, and the last push was on 2026-04-13, which means the code has been sitting without a push for the better part of six months. The README is written in the present tense and describes an intended system rather than documenting measured behaviour, and the architecture section with its single data centre and multi data centre diagrams is cut off partway through, so the on-disk design details are not available here.

Anyone evaluating this should read the source and the docs directory rather than the project description.

## Project layout and what the tests directory holds

The tree is organised for a system of this size. src/ holds the main implementation, tests/ the test suite, docs/ the documentation including the architecture diagrams referenced by the README, scripts/ the build and setup scripts that the Dockerfile copies, hooks/ for git hooks, and macro-utils/ is a local path dependency listed in the Cargo.toml for shared macros. A .devcontainer/ directory is present, which tells you there is a prepared development container for contributors.

The build tooling is visible in the Cargo.toml scripts, with entries for the various cargo and lint operations that the project runs.

```toml
[package]
name = "datenlord"
version = "0.1.0"
authors = ["DatenLord <dev@datenlord.io>"]
```

The author contact is an address at the datenlord.io domain, and the repository field points at the GitHub project. The README also links a Gitter chat room, and a codecov badge points at coverage reporting on the master branch, so the project does track test coverage.

Nothing in the repository listing indicates a published container image registry or an operator chart, which tells you deployment is currently something you assemble yourself from the Dockerfile, the config.toml and the documentation.

## Conclusion

Adopt DatenLord if you are running stateful workloads on Kubernetes across multiple clouds or data centres and the dominant cost is fetching data over the network, since its whole design is about turning remote data into local data and choosing caching strategy per access pattern. Do not adopt it for a single node application with a local disk, where the cache layers add cost with no benefit, and do not expect a production ready system, since the crate version is 0.1.0, the repository publishes no GitHub releases, and the last push was on 2026-04-13. Verify first that your kernel and storage combination fits the async IO path it depends on, that you can accept the etcd dependency for coordination, and that you read the source rather than the project page, because the architecture section of the documentation is cut off in the documentation available to me.

## FAQ

### What problem is DatenLord designed to solve?

It addresses the cost of separating computing from storage into dedicated clusters, where moving data over the network dominates application time. DatenLord abstracts the storage details so applications can assume data is local, and it caches hot data across local, neighbour and remote layers.

### How does DatenLord keep applications running when a cloud or data centre fails?

It caches data to multiple layers, such as local cache, neighbour cache and remote cache, exploiting the fact that the hot data in an online business is of limited size even when the total data set is large. The approach is transparent to applications and needs no modification to them.

### What does DatenLord require to build and run?

The Cargo.toml lists tokio with the full feature set, an etcd client at 0.11, grpcio with protobuf-codec, nix with mount and namespace features, and a ring-io crate pulled from a Git repository at a pinned revision. The Dockerfile builds with cmake, g++ and protobuf-compiler installed, and runs on an ubuntu:22.04 base image.

### Is DatenLord production ready?

The crate version is 0.1.0 and the repository publishes no GitHub releases, so there is no release history to pin against. The last push was on 2026-04-13, and the README describes the intended design in the present tense rather than documenting measured behaviour.

## Sources

- [datenlord/datenlord on GitHub](https://github.com/datenlord/datenlord)
- [Issues](https://github.com/datenlord/datenlord/issues)
- [License: MIT](https://github.com/datenlord/datenlord/blob/master/LICENSE)
- [README](https://github.com/datenlord/datenlord/blob/master/README.md)

---

Hysen Labs editorial analysis, written from the project's own repository and release notes. Cite the canonical page: https://hysenlabs.com/projects/datenlord-datenlord