Irmin applies the Git model to application data

The Git workflow, applied to data instead of source

Irmin is described as an OCaml library for building mergeable, branchable distributed data stores, and its tagline is that it is a distributed database built on the same principles as Git.

The lineage is stated in the README. Irmin is based on distributed version-control systems, which are extensively used in software development to track data provenance and show modifications in the source code, and it applies those principles to large-scale distributed data. The vocabulary is deliberately borrowed. There are functions similar to Git's clone, push, pull, branch and rebase.

The reasoning is in the README too. The Git workflow was initially designed for humans to manage changes within source code, and Irmin scales this to handle automatic programs performing a very high number of operations per second, with fully-automated conflict handling. That is the gap being filled. Git is a human-scale collaboration tool, and a fleet of agents writing to shared state needs something with the same conflict semantics but no human arbitrating every merge.

The origin explains the shape of the project too. Irmin was created at the University of Cambridge in 2013 to be the default storage layer for MirageOS applications, both to store and orchestrate unikernel binaries and the data those unikernels use.

The project is primarily developed and maintained by Tarides, with involvement by contributors from various organisations, and external maintainers and contributors are welcome. The licence is ISC, and the home page is irmin.org.

A collection of libraries rather than a database engine

The README is explicit about what Irmin is not. It says that, similarly to other MirageOS components, Irmin is a collection of libraries designed to solve different flavours of the challenges raised by the CAP Theorem, and that each application can select the right combination of libraries to solve its particular distributed problem. It is not, strictly speaking, a complete database engine.

That framing is the key to the whole design, and it is worth unpacking. The CAP Theorem says you have to choose between consistency and availability under partition, so a library that refuses to choose for you has to expose the choice. Irmin's answer is a core of well-defined, low-level data structures that dictate how data should be persisted and shared across nodes, a set of algorithms for efficient synchronisation of those distributed low-level constructs, and a collection of higher-level data structures that developers can use without knowing precisely how Irmin works underneath.

Some of those components have formal semantics, including the Conflict-free Replicated Data Type implementations, and the README links the relevant paper on CRDTs and the formal semantics work. That is the sort of thing you can point at in a design review.

The last point in that list is the one that surprises people. Because Irmin is part of MirageOS, it does not make strong assumptions about the operating system environment, which makes the system very portable. It works well for in-memory databases and for slower persistent serialisation, such as SSDs, hard drives, web browser local storage, or even the Git file format.

That is a real constraint, not a feature list. Irmin is a distributed data structure library with a Git-like synchronisation model, and the applications it was built for are unikernels and edge services, not warehouse-scale analytics.

Storage backends and data types are both pluggable

The feature list names six capabilities, and two of them are about the shape of the data rather than about synchronisation.

Storage agnostic means you can use Irmin on top of your own storage layer, and the README names the options it ships with: in memory, on disk, in a remote Redis database, and in the browser. Custom datatypes means the library will serialise and deserialise your own types, with the derivation handled by a PPX, `ppx_irmin`, whose own README is linked. Highly portable covers running anywhere from Linux to web browsers and Xen unikernels. Git compatibility means `irmin-git` uses an on-disk format that can be inspected and modified using Git itself, which is a remarkable property to have and the reason a lot of people look at Irmin at all. Built-in snapshotting covers backup and restore. And dynamic behaviour allows users to define custom merge functions, to use in-memory transactions that keep track of reads as well as writes, and to define event-driven workflows using a notification mechanism.

That last group is where the distributed dataflow idea lives. The README describes the API as event-driven, for defining programmable dynamic behaviours and for programming distributed dataflow pipelines, and the table of contents in the project documentation lists three framings of what Irmin can be used as: a portable and efficient structured key-value store, a distributed store, and a dataflow scheduler.

The same three framings show up in the examples directory, which has separate files for the custom storage layer, the custom merge function, a custom GraphQL server, synchronisation, pushing, trees, folds, processes, deployment and a plugin. That is a good map of the design space, and the examples are the fastest way to understand what the library expects of you.

Twelve opam packages, and a minimal default

The install story is a family of opam packages rather than one library, and the choice matters because the base package is deliberately small.

The full set with the command line tool and all Unix backends is one command.

  opam install irmin-cli

A minimal installation with the reference in-memory backend is another.

  opam install irmin

The packages are `irmin` as the base plus an in-memory storage implementation, `irmin-chunk` for chunked storage, `irmin-cli` for a simple command line tool, `irmin-fs` for filesystem-based storage using `bin_prot`, `irmin-git` for Git compatible storage, `irmin-graphql` for a GraphQL server, `irmin-mirage` for MirageOS compatibility, `irmin-mirage-git` and `irmin-mirage-graphql` for the MirageOS variants, `irmin-pack` for a compressed on-disk POSIX backend, `ppx_irmin` as the PPX deriver for Irmin content types, and `irmin-containers` as a collection of simple, ready-to-use mergeable data structures.

That last package deserves a note. It is the answer to the question a new user asks first, which is which data structure should I use, and if the ready-made mergeable structures cover the case, you do not have to design a merge function at all.

Any specific package installs by name.

  opam install <package-name>

And the development version is a clone and a local install.

  git clone https://github.com/mirage/irmin
  cd irmin/
  opam install .

The repository layout reflects the package structure, with an `.opam` file per package including `libirmin.opam`, `irmin-client.opam`, `irmin-server.opam`, `irmin-test.opam`, `irmin-tezos.opam` and `irmin-pack-tools.opam`, alongside `src/`, `test/`, `bench/`, `examples/` and the `dune` build files.

The benchmark replays a real commit trace

The build system is dune, and the Makefile is short enough to read in one go, which is useful because the targets say what the project considers important.

`all` is `dune build`, `test` is `dune runtest`, `doc` is `dune build @doc`, `examples` is `dune build @examples` and `clean` is `dune clean`. There is a `fuzz` target that builds the fuzz targets with the `--no-buffer` flag, which is a sign that the library is exercised with generated input rather than only with hand-written cases.

The interesting one is the benchmark, and it is not a microbenchmark. The `bench-pack-with-trace-replay` target runs a tree benchmark in trace mode over a file called `tezos_actions_1commit.repr`, with the number of commits set to 12000, producing artefacts, and a second binary then computes a statistics summary from those artefacts as JSON.

So the benchmark for the `irmin-pack` backend is a replay of twelve thousand real commits recorded from Tezos, rather than a synthetic write loop. That is a much better test of a mergeable store than a synthetic one, because the merge behaviour is only exercised by data that was actually produced by a running system with real conflicts.

The type of the data also explains why `irmin-tezos.opam` exists in the package list. The trace file is named after Tezos, and using it as the benchmark corpus means the library is tested against a workload with a similar shape to its intended use.

Where Irmin is the wrong tool

The limits of Irmin are the limits of the model it adopts, and they are worth stating before a team commits to it.

The first is that there is no query language. Irmin is a set of data structures with merge semantics, not a database you can ask questions of, and the closest thing to a query interface in the package list is `irmin-graphql`, a GraphQL server built on top. If what your application needs is to filter, aggregate and join over stored data, a conventional database is the tool for that and Irmin is not a substitute.

The second is that the performance envelope is the one Irmin describes. The documentation says it works well for in-memory databases and slower persistent serialisation, such as SSDs, hard drives, browser local storage or the Git file format. The Git file format in particular is a wonderful property to have and a poor choice for a hot write path, since every commit is an object write into a repository rather than a page write into a file.

The third is the OCaml and MirageOS specificity. The project grew out of unikernel storage, the CI runs on OCaml Labs, and the package list is a set of dune packages with their own opam files. Porting the ideas to another language is a research project, not an install.

The alternative approach, and the difference in approach is real, is to use a CRDT library for your language, or a database with native CRDT support, and get the same automatic conflict resolution with a mature storage engine underneath. The trade is that you give up the storage agnosticism, the Git compatible on-disk format and the ability to inspect and modify your data with Git, and in exchange you get something with a query language, a much wider ecosystem of tooling and a smaller surface for subtle bugs. Irmin is the more interesting design and the one that fits a unikernel or an edge service where the whole data set is small and the merge semantics are the hard part of the problem.

Editorial conclusion

Irmin suits an OCaml team that needs replicated, conflict-tolerant application state and wants the merge semantics to be explicit and inspectable, or a MirageOS unikernel that needs storage it can reason about. It is the wrong tool for a team that wants a database with a query language, and the wrong tool if you need a proven implementation rather than a set of primitives. Check first the package list, because the base `irmin` package is deliberately minimal and you will need to choose a storage backend, and read the `irmin-containers` and `irmin-git` descriptions before designing your own data structures, since the ready made ones may already be what you want.

Frequently asked questions

Official sources

1. License: ISC
2. mirage/irmin on GitHub
3. Project website
4. README
5. Releases