# Oxen: Git-Style Data Version Control for Terabyte Datasets

> Oxen-AI/Oxen is a Rust data version control system that mirrors git commands but indexes images, parquet files and model weights instead of source code. It is Apache-2.0, ships a CLI, a Python package and a self-hostable server, and the last push was on 2026-08-27.

**Oxen-AI/Oxen** — Lightning fast data version control system for large repositories of data. Feels like git, pushes and pulls like oxen.

- Repository: https://github.com/Oxen-AI/Oxen
- Website: https://oxen.ai
- Stars: 1,192 · Forks: 32
- Language: Rust
- License: Apache-2.0
- Published: 2026-08-08 · Updated: 2026-08-18 · Language: en
- Canonical page: https://hysenlabs.com/projects/oxen-ai-oxen

## The dataset problem Oxen is aimed at

Git tracks lines of text well and large binaries badly. Teams that version training data usually end up with a folder of dated copies, a spreadsheet of which run used which snapshot, or git-lfs, which stores pointers and moves the payload elsewhere. Oxen's README frames the target directly: versioning data should be as easy as versioning code, and the tool is optimised for repositories with millions of files and terabytes of data.

The intended user is not a solo developer with a CSV. It is a team holding image sets, audio, video, game assets or studio media, plus annotation files that must stay in step with the media. A typical commit in the README's own example adds 200k images and their corresponding annotations in one operation. If your repository is mostly source code with one model file attached, git plus git-lfs is a smaller dependency than a second version control system, and Oxen's own README concedes the interface mirrors git rather than replacing the mental model.

## How the merkle tree and metadata extractors work

The architecture visible in the repository is a Rust workspace. Cargo.toml declares members = ["crates/*"], and the README names liboxen as the crate that powers the CLI, the Python bindings and the server. The Python package is not a reimplementation: the README describes oxen-python as the library that wraps the core Rust codebase, with bindings provided by PyO3. So the same indexing path runs whether you call oxen from a shell or from Python.

What gets stored is the interesting part. Oxen stores any blob type, but the README states it has specialised metadata extractors for certain filetypes and caches that information in the merkle tree for fast access later. That is the mechanism behind the tabular features: csv, parquet and jsonl files are indexed and queryable rather than treated as opaque bytes. The trade-off is that extractor coverage is not uniform across formats. The README lists images, audio, video, text, parquet, json, model weights and csv as handled, but it does not publish a table of which formats get a dedicated extractor and which fall back to plain blob storage. That distinction decides whether a query on your data is cheap or impossible, and it is the first thing to check against your own file types.

Transfer is the other half. The README says the tool pushes and pulls to an oxen-server, which you can run on your own storage or use through Oxen.ai. There is also a Workspaces feature for interacting with data on the centralised server, which the README links to in the docs rather than describing inline. The self-hosting path is documented in the repository as SelfHosting.md, and docker-compose.yml wires an oxen service to a Traefik reverse proxy on port 80 with a volume at /var/oxen/data.

## Installing the CLI and committing a first dataset

The README gives two install routes plus the releases page. Homebrew for the command line tool:

```bash
brew install oxen
```

After that, oxen should be on your PATH and the subcommands mirror git. The Python route is a separate package name, oxenai, not oxen:

```bash
pip install oxenai
```

To see the workflow without creating anything, the README suggests cloning a public dataset from OxenHub:

```bash
oxen clone https://hub.oxen.ai/ox/CatDogBBox
```

That command should produce a working copy of the CatDogBBox repository on disk. For a repository of your own, the README's five-line example is the whole loop:

```bash
oxen init
oxen add images/
oxen add annotations/*.parquet
oxen commit "Adding 200k images and their corresponding annotations"
oxen push origin main
```

oxen init creates the repository, the two add calls stage a directory and a glob of parquet files, commit records the message, and push sends it to the origin remote. The README does not show the command that configures that remote, so check the CLI documentation before assuming push works against a fresh local repository. For self-hosting, docker-compose.yml defines the server as image oxen/server:0.1.0 with a data volume mounted at /var/oxen/data; note that this image tag does not match the workspace version in Cargo.toml, which is 0.56.1, so verify which server tag you actually intend to run.

## Where Oxen is the wrong tool

The clearest limitation is in the platform table. oxen-server is marked unsupported on Windows, and the README says to run it on Linux, macOS or in Docker. The CLI and Python client do work on Windows and connect to a server hosted elsewhere, so a Windows-only team can use Oxen but cannot host the central copy on Windows.

Versioning is the second issue. The workspace version is 0.56.1 and the recent releases are v0.53.3, v0.53.4 and v0.55.0, all in the 0.x range. Pre-1.0 data tooling means the on-disk format and the server API can move between releases, and the repository carries a RELEASING.md and a ReleaseNotes.md precisely because upgrades need reading. If you need a format you can freeze for a decade, this is not it yet.

The third case is a repository that is mostly text. Oxen's advantages come from indexing large blobs and extracting structure from tabular and media files. On a codebase of source files, git's ecosystem of hosting, review and CI integration is deeper, and adding a second version control system to the same directory buys you nothing. The README itself positions Oxen as shining where git and git-lfs fall short, which is an admission about where they do not.

## Oxen compared with git-lfs and DVC

Git-lfs keeps git as the source of truth and replaces large file contents with pointer files, storing the payload in a separate LFS endpoint. The history, branching and review model stays git's; the large files are an attachment. Oxen inverts that. The data is the repository, and the merkle tree carries extracted metadata about the contents, which is why the README can claim native tabular handling and querying of csv, parquet and jsonl. With git-lfs, a parquet file is an opaque blob and querying it means downloading it first.

DVC takes a third position: it keeps a git repository for the pipeline definitions and stores data in a cache with .dvc pointer files, so pipeline reproducibility is the organising idea. Oxen's README does not describe a pipeline or stage concept, and the interface it advertises is the git command set applied to data. If your problem is reproducing a multi-stage training pipeline, DVC's model maps to it more directly. If your problem is that a 200k-image directory with annotations is painful to snapshot, diff and share, Oxen's model is the closer fit.

One practical difference worth noting: Oxen ships a server you can run yourself. Dockerfile and docker-compose.yml are in the repository, and SelfHosting.md documents the path, so the central store can live on your own infrastructure rather than a vendor's.

## Licence, releases and the cost of upgrading

Oxen is Apache-2.0, with the workspace declaring license-file = "LICENSE". Apache-2.0 permits commercial use, modification and redistribution and includes an explicit patent grant, which matters if you embed liboxen in a product. It also requires that you keep the licence and notice files with any redistribution. That is a summary of the licence text, not legal advice; read LICENSE and your own counsel's view before shipping a derived binary.

The upgrade cost is real but bounded. Releases arrive frequently, with v0.53.3 on 2026-08-14, v0.53.4 on 2026-08-19 and v0.55.0 on 2026-08-27, and the last push to the default branch was on 2026-08-27. Each release note is the place to look for breaking changes, and ReleaseNotes.md is tracked in the repository. Because the CLI, the Python package and the server share liboxen, a version skew between a client and a server is the failure mode to watch: upgrading oxen without upgrading the server, or the reverse, is the situation where mismatched expectations surface. The README does not document a compatibility matrix between client and server versions, so pinning both from the same release is the safer habit.

## Building from source and what that requires

Contributors and anyone embedding liboxen build from the Rust workspace. The README points at bin/install-prereqs for an automatic setup that supports macOS and Debian-based Linux, and at rustup for the manual path:

```bash
curl --proto '=https' --tlsv1.2 -sSf https://sh.rustup.rs | sh
```

The developer tooling is a single cargo install line, covering bacon for reload-on-change, cargo-machete, cargo-llvm-cov and cargo-sort:

```bash
cargo install bacon cargo-machete cargo-llvm-cov cargo-sort
```

cmake is required, installed on macOS with brew install cmake. The workspace pins rust-version = "1.97.1" and edition = "2024", so an older toolchain will refuse the build rather than fail obscurely. The Dockerfile pulls in clang, openssl, libssl-dev, pkg-config, libjpeg-turbo-progs, libpng-dev and a pinned FFmpeg installed through bin/install-ffmpeg with versions read from tool-versions.env. That file is described in the Dockerfile comments as the single source of truth shared with Linux development and CI, which is a sensible arrangement: one place to bump a pinned dependency.

One comment in Cargo.toml is worth reading before you touch linker flags. It records that the macOS "__eh_frame section too large" note from the bundled DuckDB and RocksDB is expected and harmless, and warns that silencing it with -Wl,-no_compact_unwind previously turned every DuckDB error into an uncaught-exception abort of the whole server. That is a documented past failure, not a theoretical one.

## Conclusion

Adopt Oxen if your repository is mostly binary or tabular data and your team already thinks in git commands; the CLI and oxenai package install on Linux, macOS and Windows, and oxen-server runs on Linux, macOS or Docker. Do not adopt it if you need Windows server hosting, because the platform table marks oxen-server as unsupported on Windows, or if you need a stable on-disk format, since Cargo.toml still carries 0.x versions. Before committing a real dataset, run oxen init and oxen add on a representative sample and confirm what the specialised metadata extractors record for your file types, because the README lists extractors without enumerating which formats have one.

## FAQ

### Does Oxen run on Windows?

The oxen CLI and the oxenai Python package are supported on Windows, but oxen-server is not. The README says to run the server on Linux, macOS or in Docker, and Windows clients connect to a server hosted on one of those platforms.

### How do I install the Oxen command line tool?

The README gives Homebrew as the install route for the CLI, with brew install oxen, and points at the GitHub releases page as an alternative. The Python package installs separately with pip install oxenai.

### What kind of data can Oxen version?

The README says Oxen can store any blob type, and that it has specialised metadata extractors for certain filetypes which are cached in the merkle tree. It lists images, audio, video, text, parquet, json, model weights and csv among the handled formats.

### Can I host Oxen repositories on my own infrastructure?

Yes. The README says you can host repositories on Oxen.ai or self host the oxen-server, and the repository contains SelfHosting.md plus a docker-compose.yml that runs an oxen service behind a Traefik reverse proxy with a data volume at /var/oxen/data.

## Sources

- [Official documentation](https://oxen.ai)
- [Official README](https://github.com/Oxen-AI/Oxen#readme)
- [Project repository](https://github.com/Oxen-AI/Oxen)
- [Release notes](https://github.com/Oxen-AI/Oxen/releases)

---

Hysen Labs editorial analysis, written from the project's own repository and release notes. Cite the canonical page: https://hysenlabs.com/projects/oxen-ai-oxen