rudolfs: a caching Git LFS server with S3 and local disk backends

An LFS server with a cache in front of the storage

rudolfs is described as a high-performance, caching Git LFS server with an AWS S3 back-end, and the description in the repository metadata adds local storage as the second option.

The problem it addresses is the cost and the latency of LFS traffic. Every LFS clone of a repository with large assets pulls every object through the server, and if the objects live in S3 that means an API call and a bucket read per object, repeatedly, for every user and every clone. The cache in front changes that arithmetic. The README says a configurable local disk cache speeds up downloads and reduces your S3 bill, and the production notes add that a bigger cache is almost always better, suggesting around 85 percent of the available disk space.

That is the core idea, and the rest of the feature list is refinements on it. Encryption of LFS objects in both the cache and in permanent storage, so the cache is not a plaintext copy of your assets sitting on a disk. Corruption resilience in the cache, which is the interesting one and has its own section below. Separation of GitHub organizations and projects, which is about storage efficiency rather than permissions. And a tiny Docker image, under ten megabytes, built from scratch with a statically linked musl binary.

The tool is written in Rust with edition 2024, the crate version in the manifest is 0.3.8, and it is published on crates.io, in the AUR for Arch, and on Docker Hub as jasonwhite0/rudolfs. The repository publishes no GitHub releases, so the crates.io and Docker Hub versions are the ones to track.

The audience is an engineering team running a private Git server whose repositories contain large binaries, and who want the object store to be a cloud bucket while the hot path stays on local disk.

Two backends behind a composable storage layer

The README lists the backends as two numbered options. The first is an AWS S3 back-end with an optional local disk cache, and the second is a local disk back-end.

The two are selected by a positional argument on the command line, which is visible in both the development and the compose examples. The S3 form ends with s3 and --bucket foobar, and the local form ends with local and --path=/data.

Behind that choice, the design intent is stated explicitly: the back-end storage code is very modular and composable, and pull requests for implementing other storage back-ends are welcome, with the request that you open an issue before starting work. The Rust dependency list supports that story. The AWS side is the aws-config and aws-sdk-s3 crates with the rustls feature and default features turned off, so the S3 client is TLS through rustls rather than through the platform's OpenSSL. The HTTP server is hyper 0.14 with the server, http1, http2, tcp and stream features, the async runtime is tokio with the full feature set, and the command line is structopt, the older of the two popular Rust argument parsers.

Two dependencies point at features the README does not spell out. The linked-hash-map crate with a serde implementation suggests that insertion order is preserved somewhere the ordering matters, and the hybrid-array crate is the fixed capacity array type used for the LFS object identifiers, whose length is fixed by the protocol and therefore awkward to model with a general purpose Vec.

The AWS credential handling follows the standard chain, and the README documents the order precisely. The server searches environment variables first, AWS_ACCESS_KEY_ID and AWS_SECRET_ACCESS_KEY, then the AWS credentials file, usually at ~/.aws/credentials, and finally the IAM instance profile, which only works on an EC2 instance that has one. The region is read from AWS_DEFAULT_REGION or AWS_REGION, falls back to us-east-1 if the value is malformed, and otherwise falls back to the value associated with the current profile in ~/.aws/config or the file named by AWS_CONFIG_FILE.

The fallback chain is a nice piece of care, because a malformed region string that silently becomes us-east-1 is the kind of thing that produces a confusing permissions error rather than a clear one.

Encryption at rest, and the key change that destroys everything

Encryption is optional and the key handling has sharp edges that the README does not soften.

When it is configured, all LFS objects are encrypted with the xchacha20 symmetric stream cipher, and the key must be generated before the server starts:

openssl rand -hex 32

That is a 32-byte key, and the README asks you to keep it secret and save it in a password manager so you don't lose it, then pass it to the server with the --key option. If the --key option is not specified, the LFS objects are not encrypted.

The cipher choice is visible in the dependency list, the chacha crate at 0.4, which provides the xchacha20 construction, and the objects are encrypted in both the cache and in permanent storage.

Now the warnings, because they are the most important part of this section. If the key ever changes, or if encryption is disabled, all existing LFS objects will become garbage, and when the Git LFS client attempts to download them the SHA256 verification step will fail. Likewise, if encryption is later enabled after it has been disabled, all existing unencrypted LFS objects will be seen as garbage.

That is a sharp edge with no recovery path, and it is worth understanding why it is sharp. The LFS protocol has the client verify the SHA256 of what it downloaded against the pointer in the repository. rudolfs encrypts objects before storing them and decrypts them before serving them, so a client that has the wrong key receives a decrypted stream that does not hash to the expected digest, and the client rejects it. The object in storage is fine, but the key is gone, and the key is the only thing that can make it readable again. A backup of the bucket without a backup of the key is not a backup.

The final note is the one that matters for threat modelling. Objects in both the cache and in permanent storage are encrypted, however objects are decrypted before being sent to the LFS client, so take any necessary precautions to keep your intellectual property safe. In other words, encryption at rest protects the disk and the bucket, not the wire, and if that distinction was not already obvious it now is.

A cache that finds its own corruption

The corruption resilience feature is the most distinctive thing in the project, and the README describes it with a bit of humour that is still technically precise.

The claim is that even if the disk is getting blasted by cosmic rays, the server will find corrupted LFS objects and purge them from the cache transparently, and the client should never notice this happening.

The mechanism follows from how LFS works. Every object is content addressed, its identity is the SHA256 of its contents, so a cache entry that has been damaged on disk is detectable by hashing it and comparing against the digest in the object identifier. When the mismatch is found, the entry is unusable, and the correct response is to delete it and fetch the object again from permanent storage, which is where the authoritative copy lives. A cache that serves a corrupt object without checking would push the corruption to the client, and the client would fail its own verification step with a confusing error, and the user would blame their network or their repository.

So the cache is treated as untrusted storage, which is the correct posture for a cache, and the cost of that posture is a hash computation on cache reads.

The Cargo manifest also carries a testing tool for exactly this. There is a feature named faulty, which when enabled injects random failures into the byte stream, and it pulls in the rand crate for that purpose. A feature that deliberately corrupts data on demand is how you test that the healing path works, rather than discovering it works because a disk failed on a Tuesday afternoon.

The backoff dependency, with the tokio feature, is the retry behaviour, and the same feature flag is what lets the test suite exercise it. The human-size and humansize crates, plus humantime, are for formatting the cache size, which is a detail that matters when the cache is configured to take 85 percent of a disk and you need to see what that means in gigabytes.

Running it, and the loopback binding that catches people

The development instructions use cargo directly, with the AWS credentials exported and the key set in a variable:

cargo run -- \
  --cache-dir cache \
  --host localhost:8080 \
  --max-cache-size 10GiB \
  --key $KEY \
  s3 \
  --bucket foobar

The local disk variant is shorter:

cargo run -- --port 8080 local --path=/data

And the README attaches a warning to the first form that is easy to skim past. This server may not be accessible from other machines. Specifying --host localhost:8080 will often bind the server to an internal-only loopback network interface, if localhost resolves to 127.0.0.1 or [::1]. To make the server accessible from the outside world you must specify --host 0.0.0.0:8080 or just --port 8080, since the default IP the server binds to is 0.0.0.0.

That is a genuine footgun, and the README points at an issue comment for the background. The two spellings of the same thing, --host and --port, behave differently, and the one that looks most explicit is the one that restricts you.

Production uses docker compose, with a .env file holding the configuration:

AWS_ACCESS_KEY_ID=XXXXXXXXXXXXXXXXXXXQ
AWS_SECRET_ACCESS_KEY=XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX
AWS_DEFAULT_REGION=us-west-1
LFS_ENCRYPTION_KEY=xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx
LFS_S3_BUCKET=my-bucket
LFS_MAX_CACHE_SIZE=10GB

Then the compose file is started, and the README notes that there are alternative compose files for MinIO and for the local disk back-end:

docker-compose up -d

# use minio yml
docker-compose -f ./docker-compose.minio.yml up -d

# use local disk yml
docker-compose -f ./docker-compose.local.yml up -d

The MinIO variant matters more than it looks, because MinIO is an S3-compatible object store, and it lets you run the same production topology without an AWS bill. The compose file also carries a commented nginx service and a note that a real production server should use nginx for TLS, which is a sensible default to leave commented out rather than assume.

One more note from the README: always use a different S3 bucket, cache directory, and encryption key than what you use in your production environment.

URLs namespace projects, and the missing authentication

Client configuration is a file named .lfsconfig in the root of the Git repository, committed so that everyone uses the same LFS server, with the URL carrying the organisation and the project:

[lfs]
url = "http://gitlfs.example.com:8080/api/my-org/my-project"

The README explains what that buys: you specify the org and project names in the URL and they are automatically created, and if two projects share many LFS objects, have them use the same URL to save on storage space.

So the URL is a storage namespace, not a permission boundary. Two projects that share a URL share their objects, and the server deduplicates the storage. That is a real feature, because the same asset committed in two repositories is one object rather than two, but it also means the scheme is not a way to isolate tenants, and a reader should not mistake one for the other.

The last section of the README is the one that should shape any deployment decision, and it is under the heading Non-Features. There is no client authentication, and the server is meant to be run in an internal network with clients you trust, not on the internet with malicious actors.

That is an unusual level of honesty, and it is a deliberate design choice rather than an oversight. The server is a caching proxy in front of a bucket that has its own credentials, so anyone who can reach the server can read the objects, and adding authentication would mean adding a user model, a credential store and a session or token scheme, which is a different project. The consequence is that the network boundary is the security boundary, and the README's advice to put nginx in front for TLS is the first half of a defence that you have to complete yourself.

The packaging reinforces the small-footprint story. The Dockerfile builds with rust:1.98.1, compiles a static musl binary for either amd64 or arm64, adds Tini as PID 1 for correct signal handling, and finishes with a scratch base image containing the CA certificate bundle, the binary and nothing else, which is where the under ten megabyte figure comes from.

Editorial conclusion

Adopt rudolfs if you need to serve Git LFS objects from a private network and want S3 as the system of record with a local cache in front of it, since the cache both speeds up repeated downloads and cuts your storage bill. Do not adopt it on a public endpoint, because the README lists the absence of client authentication as a deliberate non-feature, and do not adopt it if you need multi tenant isolation beyond the URL scheme, since a project namespace is not an authorisation boundary. Verify first that you can protect the encryption key, because the README warns that changing it or toggling encryption turns every existing object into garbage that fails SHA256 verification on download, that the server binds to all interfaces by default and will therefore listen on your network unless you set --host, and that the cache is sized at roughly 85 percent of the free disk rather than filling it.

Frequently asked questions

Official sources

1. Issues
2. jasonwhite/rudolfs on GitHub
3. License: MIT
4. README