OpenEBS Mayastor, a Rust data plane over cluster-wide NVMe

Users supply the what, the data plane works out the how

Mayastor is described as a cloud-native declarative data plane written in Rust, and the goal statement is the clearest summary of the project: to abstract storage resources and their differences through the data plane such that users only need to supply the what and do not have to worry about the how, so that individual teams stay in control.

The second half of the overview is the constraint that makes it interesting. The project tries to be as unopinionated as possible, which means it tries to work with the storage systems you might already have and unify them as abstract resources instead of swapping them out whenever the resources are local or remote.

That is a deliberate refusal to be a storage system. Most storage products for Kubernetes propose a new block device or a new file system and ask you to move your data onto it. Mayastor's position is that the interesting part of a storage system is not the medium but the coordination: which devices hold which replica, which node serves the I/O, and what happens when one of them goes away. That part can be provided as a data plane while the bytes stay wherever the hardware already put them.

The terms matter here, because they are used consistently throughout. A disk pool is an abstraction for the devices supplying the cluster with persistent backing storage. A nexus is the volume target. A replica is a logical volume carved out of a local storage pool. The data plane is the part that does the I/O, and the control plane is the part that decides what should exist.

The project is under the OpenEBS umbrella, and the CNCF status badge shows it as a sandbox project, which is the CNCF's entry-level maturity stage. For an adopter, that is a signal about the level of support and stability to expect.

Four reasons, and three of them are about hardware

The section asking why Mayastor exists lists four motivations, and they describe the deployment it is designed for.

The first is low latency workloads for converged and segregated storage, by leveraging NVMe and NVMe over Fabrics, the fabric version of NVMe. The second is micro-VM based containers like Firecracker microVMs and Kata Containers, by providing storage over vhost-user. The third is programmatic storage access, that is writing to block devices from within your application instead of making system calls. And the fourth is storage unification, to lift barriers so that you can start deploying cloud native apps on your existing storage without painful data gravity barriers.

Three of the four are about the hardware and the runtime, and that tells you the target environment. This is a system for clusters with local NVMe devices and for runtimes that need a block device with a predictable interface rather than a kernel file system. The vhost-user detail is the one that ties it to Firecracker and Kata, since that is the interface the project offers them storage over, which is exactly the shape a nexus provides.

The third motivation, programmatic access, is the one that surprises people. The readme shows a Rust example in which an application opens a descriptor on the nexus and issues reads and writes to it directly, with DMA buffers, without going through the operating system's block device layer. That is not a convenience API, it is a different integration model, and it is what lets a data-intensive service talk to the storage as a device.

The fourth, data gravity, is the commercial argument. If the storage system that holds your persistent volume reaches end of life, the usual response is a migration project, which is painful and complicated. The readme's point is that the individual volumes per team or application are relatively small, but that today it is not possible for individual teams to handle their own storage needs, and the nexus provides the abstraction so that they can.

The control plane lives in three repositories, not one

The architecture section splits the system into a control plane and a data plane, and the control plane is spread across repositories in a way that is worth understanding before you clone anything.

The control plane is microservices patterned, centred around a core agent and a RESTful API. It is extended by a dedicated operator responsible for managing the life cycle of Disk Pools, which is the abstraction for the devices supplying the cluster with persistent backing storage, and by a CSI compliant external provisioner, the controller.

The source code for the control plane components is in the mayastor-control-plane repository, and the Helm chart along with other Kubernetes specific extensions, an example being the kubectl plugin, are in the mayastor-extensions repository. So the repository you are reading is the data plane, and the chart that deploys the control plane lives somewhere else entirely. That is a common shape for a project that has grown, and it is a genuine papercut for a first-time user who expects one clone to be the whole product.

The CSI side is a split of two plugins with different deployment models. There is a csi-node plugin, deployed as a daemonset, implementing the identity and node services, and a csi-controller plugin, deployed as a deployment, implementing the identity and controller services. That is the standard CSI decomposition, and it is worth naming because the two pieces have very different lifecycles: the node plugin is per node, the controller is a singleton.

The project is registered in Artifact Hub with a Helm chart badge, and the release badge points at the Mayastor releases. The project is also signed off by FOSSA for licence compliance, and carries the OpenSSF best practices badge.

The data plane: an I/O engine per node, pools, and replicas

Each node you want to use for storage or for storage services has to run an I/O Engine instance, and that instance has two major components.

The first is the local storage pools, which can be carved out into logical volumes, the replicas, which in turn can be shared to other I/O engines over NVMe over Fabrics. That last clause is the mechanism that makes a volume cluster-wide: a replica living on one node is exported over the fabric so that the I/O engine on another node can write to it.

The second is the volume target, the nexus, which is the subject of the next section. So the shape of the data plane is: local devices, pooled on a node, carved into replicas, exported over NVMe-oF, and gathered by a nexus into something that looks like one device.

That is a significant amount of machinery to run on every node that participates in storage, and it is the operational cost that comes with the cluster-wide model. A cluster where some nodes provide storage and others do not has to make that distinction somewhere, and the readme puts the I/O Engine requirement plainly: each node you wish to use for storage or storage services will have to run one.

The workspace in the repository reflects the same split. There is an io-engine/ directory, an io-engine-bench/ directory for the benchmarks, an io-engine-tests/ directory for the tests, a sysfs/ directory for the Linux sysfs interactions, a jsonrpc/ directory for the JSON-RPC layer, and a spdk-rs/ directory for the Rust bindings to SPDK, which is the userspace storage stack that sits underneath the data plane.

The nexus turns storage URIs into one device

The nexus is responsible for attaching to your storage resources and making them available to the host that is selected to run your Kubernetes workload. From the nexus's point of view those resources are its children.

The readme then shows what a nexus looks like in code, and the example is short enough to be worth reading in full because it explains the semantics. A nexus is created from a vector of child devices, each named by a storage URI, and the URIs in the example are an aio URI pointing at a local image file with a block size, and two nvmf URIs pointing at exported NVMe targets on other hosts, with a comment noting that it is assumed those hosts are reachable over the network.

let children = vec![
      "aio:////disk1.img?blk_size=512".to_string(),
      "nvmf://fooo/nqn.2019-05.io-openebs:disk0".into(),
      "nvmf://barr/nqn.2019-05.io-openebs:disk0".into()
];

The creation call takes a name, a block size, a capacity, an optional UUID, and the children, and the comment on the UUID says that if none is given one will be generated. A descriptor is then opened on the nexus, a DMA buffer is allocated with the alignment handled implicitly, and the buffer is filled, written and read back.

The key line is the comment on the write: all child devices will receive the same IO, and a single IO becomes three IOs. That is the replication model in one sentence, and it also explains the cost. Every write to a replicated volume is a write to every child, so a three-way replica is three writes and the failure of a child is a failed write, which is the trade the design accepts in exchange for durability and for a volume that can move.

The URIs are not created by the application. Based on storage URIs the nexus knows how to connect to the resources and makes them available as a single device to a protocol standard, and those URIs are managed by the control plane, which keeps track of what resources belong to what nexus instance and subsequently to what PVC.

Rust, SPDK and a Nix shell for the build

The repository is a Cargo workspace, and the member list is a map of the data plane's concerns: jsonrpc, libnvme-rs, io-engine, io-engine-bench, io-engine-tests, sysfs, spdk-rs, and a utils/dependencies/ tree with separate crates for the io-engine API, the prost extensions, version information, the composer, device information and the nvmeadm utility.

The dependency list explains the transport and the tooling. tonic and prost for gRPC with the protobuf build tooling, tokio and futures for the async runtime, tracing with an env-filter subscriber for logging, inotify for file watching, which the comment says is for cpuset affinity and TLS certificate reload and is Linux only. Then the Linux-specific crates, nix, udev and libc, plus bindgen and cc for binding generation. There is bollard for talking to a container runtime, async-nats for messaging, snafu and thiserror for error handling, serde with yaml and json support, and a set of smaller utilities including run_script, semver, derive_builder and regex.

The development environment is the most unusual part of the setup. The Dockerfile at the root is not an application image; it creates a Nix build environment for developing Mayastor, and it is also used in the CI pipeline because it comes with all the Mayastor dependencies downloaded and prebuilt. All that is needed is to check out the sources and enter the Nix shell. The image adds the unstable Nix channel, installs a few basic tools, copies the shell.nix and the nix/ directory into a temporary expression directory, and then runs the nightly and stable channel shells once to fetch the dependencies.

The rest of the tree is conventional and disciplined: a .cargo/ directory, a Dockerfile, a default.nix, an ide.nix and a shell.nix for the environment, a terraform/ directory, a commitlint.config.js for commit message rules, a .pre-commit-config.yaml, and governance files including RELEASE.md, SECURITY_CONTACTS.md, MAINTAINERS.md and GOVERNANCE.md.

Where Mayastor is the wrong tool

The first limit is the hardware. The reasons the project exists are about NVMe and NVMe over Fabrics, and a cluster with managed cloud block volumes, a network file system or an object store has nothing for it to abstract. This is a system for clusters with local fast storage and a fabric between them.

The second is the operational surface. Running an I/O Engine on every node that provides storage, plus a control plane with its own operator and CSI plugins, plus the data plane's own dependencies, is a substantial platform commitment. A team that wants shared storage between pods is better served by the storage system it already has, and a team that genuinely needs the cluster-wide volume model should budget for operating the stack rather than just deploying it.

The third is the honesty of the unopinionated stance. Working with the storage you already have means you still have to operate that storage, and Mayastor does not remove the vendor, the warranty or the end-of-life date of the device under the floor. It changes who has to coordinate the move when the device reaches end of life, not whether the move happens.

The fourth is the maturity signal. A CNCF sandbox project is explicitly one that is still finding its users and proving its operational model, and the version numbers, 2.12.x, are the version of the data plane rather than a promise of a stable interface. A team that has to justify the choice to a risk committee should treat the sandbox status as part of the answer.

The fifth is the direct device access model, which is the most interesting feature and the most demanding one. Letting an application open a descriptor on a replicated volume and issue its own DMA writes bypasses the usual kernel interfaces, and the readme is clear that alignment is handled implicitly and that a single write becomes several. Code written against that model is tied to it, and moving later means rewriting the I/O path, not swapping a driver.

Editorial conclusion

Mayastor fits a Kubernetes platform team with NVMe hardware that needs cluster-wide replicated volumes, that runs micro-VM runtimes such as Firecracker or Kata, and that has a storage end-of-life problem to solve without a migration project. It does not fit a team whose storage is a managed cloud block device, a file system or an object store, since the abstraction is built around local NVMe devices and NVMe over Fabrics. Before adopting it, read the parts of the readme that are promises rather than instructions, particularly the end-of-life migration story, which is the actual reason to try it, and check the CNCF status, which is sandbox, meaning the project is still looking for adopters. The licence is Apache-2.0, version 2.12.2 was published on 2026-10-06, and the last push was on 2026-10-13.

Frequently asked questions

Official sources

1. Issues
2. License: Apache-2.0
3. openebs/mayastor on GitHub
4. README
5. Releases