cachey: a read-through cache for object storage, in Rust

A cache for blobs that never change

Cachey is a high-performance read-through cache for object storage, written in Rust and published by the S2 team, with the project site at cachey.dev and the source under the s2-streamstore organisation.

The problem it addresses is the one every team hits when object storage becomes the system of record for large files. A bucket is cheap and durable, and a single object read over HTTP has a latency profile that is fine for a thumbnail and poor for a video segment or a parquet range. The application pays that latency on every request, and it pays the tail of it as well.

The design assumption is stated in the feature list: it is designed for caching immutable blobs. That single phrase rules out a lot. There is no write path, no invalidation problem to solve, and the cache can be trusted to keep a page once it has fetched it. The cost is that the model does not fit a bucket whose objects are rewritten, and that is a genuine constraint rather than a missing feature.

The backend is any S3-compatible store, but the front end is not the S3 API. The README is explicit that the service has its own /fetch API requiring a precise Range. That is the central trade: you give up transparent S3 compatibility and gain control over how a range read is served, which is where the latency work happens.

The remaining features explain the rest of the design. It has a simple HTTP API, a hybrid memory and disk cache powered by the foyer crate, a fixed page size of 16 MiB with requested byte ranges mapped to page-aligned lookups, coalescing of concurrent requests for the same page, hedged requests to manage the tail latency of object storage, and the ability to attempt redundant buckets for a given object.

The project also links a Reddit discussion on r/databasedevelopment that motivated it, which is a good place to find the problem statement in the author's own terms.

The 16 MiB page and the byte range arithmetic

The caching unit is a page of 16 MiB, and that single decision explains most of the behaviour.

A request arrives as a HEAD or GET to /fetch/{kind}/{object}. The kind and the object form the cache key, where kind identifies the bucket set, up to 64 characters, and the object is the S3 object key. The Range header is required, in the format bytes={first}-{last}, which means Cachey is not a general purpose proxy that can serve a whole object without knowing its size.

The service maps the requested range onto page-aligned ranges and fetches those pages, caching each one. The response keeps standard HTTP semantics, 206 Partial Content and 404 Not Found among them, and carries a small set of headers of its own:

HTTP/1.1 206 Partial Content
Content-Range: bytes 1048576-18874367/52428800
Content-Length: 17825792
Content-Type: application/octet-stream
C0-Status: 1048576-16777215; us-west-videos; 1704067200

Content-Range gives the range actually served, Content-Length the number of bytes, Last-Modified the timestamp from the first page, and Content-Type is always application/octet-stream regardless of what the object really is.

The interesting header is C0-Status, and its format is {first}-{last}; {bucket}; {cached_at}. It tells you the byte range of the first page, which bucket served it, and when it was cached, with a cached_at of zero meaning a cache miss. So the response tells you not just what you got but where it came from and whether it came from the cache at all.

The 16 MiB granularity is the main thing to weigh. For large media or dataset files it amortises well, and a range read that starts mid-page still fetches a whole page and serves from it. For a bucket of small objects, a page holds many of them and the effective hit rate drops, and a single small read costs a page fetch. The fixed page size is a deliberate simplification, and it trades efficiency for it.

Redundant buckets, hedged requests and per-page timeouts

Two optional request headers carry the resilience story, and they are the most interesting part of the API.

C0-Bucket names the bucket or buckets containing the object. Sending it several times specifies redundant buckets with the preferred one first, and omitting it means the kind segment is used as the singular bucket name. So a request can ask for a primary bucket and a backup, and the service will try the backup if the primary is slow or failing. The README points at a replica reads document for the selection, fallback, hedging and download limits.

C0-Config overrides the S3 request configuration per page miss, as space-separated key-value pairs. The keys are ct for the connect timeout, used when an existing connection could not be reused, rt for the read timeout, which the README defines as time to first byte, ot for the operation timeout across retries, oat for the operation attempt timeout, ma for the maximum attempts for each bucket's primary SDK operation, ib and mb for the initial and maximum backoff duration, and fps to force path-style addressing.

The scope of those overrides is bounded, and the README says so: SDK overrides are bounded by the server's download deadlines, and early hedges use one SDK attempt. So a client can tune the client-side behaviour but cannot exceed the server's own limits.

The hedging itself is the tail latency management. A hedged request is a second attempt at the same page sent before the first has given up, which converts a slow storage response into a fast one by racing two requests. The budget is a server-side percentage, --hedge-budget-percent, 5 by default, where zero disables early hedging. A client that wants fewer speculative requests can set it in C0-Config or the client can accept the default.

The example request in the README shows all of this in one line:

GET /fetch/prod-videos/movie-2024.mp4 HTTP/1.1
Range: bytes=1048576-18874367
C0-Bucket: us-west-videos
C0-Bucket: us-east-videos-backup
C0-Config: ct=1000 oat=1500 ma=5 ib=10 mb=100

Two buckets, a one second connect timeout, a 1.5 second attempt timeout, five attempts and a backoff window of 10 to 100 milliseconds.

Per-page status needs HTTP/2 trailers

A multi-page read has more than one page status, and the way Cachey reports the later ones is a detail worth understanding before you design a client around it.

The first page status is always sent as a header, over both HTTP/1.1 and HTTP/2. For a multi-page GET, the subsequent page statuses are included as trailers, and trailers are an HTTP/2 only mechanism here. The README is explicit: use HTTP/2 when you need per-page status for a multi-page read, and HTTP/1.1 responses include Content-Length and only the first page's status.

That is a genuine constraint on the client side. On HTTP/1.1 you learn whether the first page came from cache and which bucket served it, and you learn the total length of the response, but you do not learn whether page seven was a miss or which bucket produced it. On HTTP/2 you get the full picture at the end of the response, after the body has been streamed.

The order matters for observability. A header is available when the response starts, so a client can log the first page status immediately, while a trailer arrives only once the transfer is complete, which is useful for diagnosis after the fact and useless for reacting while the transfer is in progress.

The rest of the response contract is conventional. Content-Type is always application/octet-stream, which means a client cannot rely on the content type of the underlying object, and Last-Modified comes from the first page, so a multi-page response carries the modification time of the page that was fetched first rather than of the object as a whole.

Monitoring is part of the same design. GET /stats returns throughput statistics as JSON, described in the README as being for load balancing and health checking, and GET /metrics returns a larger set of metrics in Prometheus text format. The JSON endpoint is the one you point a load balancer at, and the Prometheus endpoint is the one you graph.

Memory, disk, io_uring and the TLS options on the command line

The server is configured entirely from the command line, and the options show what the design cares about.

Memory and disk are separate budgets. --memory sets the maximum memory to use for the cache, 4GiB by default, and --max-download-memory sets the body memory reserved by active downloads, 1GiB by default, explicitly described as separate from the cache. That separation is what stops a burst of in-flight downloads from evicting the cache that is meant to absorb them.

--disk-path points at the disk cache storage, which may be a directory or a block device, and --disk-kind chooses between a file system and a block device, with fs as the default. --disk-capacity sets the maximum capacity, for example 100GiB, and if it is not specified up to 80 percent of the available space is used. The io_uring option switches disk IO to io_uring where it is available.

Then there are the timeouts, which line up with the earlier hedging story. --bucket-timeout-ms caps the bucket download time through body validation at 5000 milliseconds, and --page-timeout-ms caps the page download time including fallback at 10000 milliseconds.

TLS is straightforward: --tls-self uses a self-signed certificate, and --tls-cert and --tls-key point at a certificate and key file, which the README says must be used together.

--port defaults to 443 when HTTPS is configured and 80 for plain HTTP, which is a sensible default for a service meant to sit in front of a bucket.

A few of the options are visible in the help text:

      --memory <MEMORY>
      --disk-kind <DISK_KIND>
      --hedge-budget-percent <HEDGE_BUDGET_PERCENT>

The hedge budget option is worth setting deliberately. Zero disables early hedging, which removes the speculative second request and the tail latency benefit that comes with it.

foyer underneath, and the coalescing of duplicate page reads

The hybrid cache is powered by foyer, and the dependency list in the Cargo manifest explains what each piece is for.

foyer at 0.22.6 is the cache engine itself, giving the memory and disk tiers. The AWS SDK for S3 at 1.152.0 with the Tokio runtime is what talks to the backend, and axum 0.8 with axum-server and rustls is the HTTP front end. dashmap provides the concurrent map that the in-flight page table is built on, which is what makes coalescing possible: when two requests arrive for the same page at the same time, the second one waits for the first fetch instead of starting its own.

The observability stack is unusually complete for a small service. prometheus 0.14 and mixtrics with the prometheus 0.14 feature are there for the metrics endpoint, and exponential-decay-histogram is a latency histogram implementation, which is the right structure for a service whose whole job is tail latency. tokio carries the full feature set plus test-util, tracing and tracing-subscriber provide the logging, and rcgen is there to generate the self-signed certificate for --tls-self.

The optional jemalloc feature pulls in tikv-jemallocator with the unprefixed allocator on supported platforms, along with jemalloc_pprof for flamegraph output and jemalloc-ctl for stats. That is a performance-oriented choice with a profiling story attached, and the Docker build enables it.

The remaining dependencies are small and tell you the shape of the work: http-range-header and http-content-range for the byte range arithmetic, httpdate for the Last-Modified header, bytesize for parsing the human-readable size arguments, compact_str for the small string types that a per-request path benefits from, and itertools, eyre, thiserror, serde, clap and futures for the rest.

Build discipline, container images and the release line

The engineering practice in the repository is stricter than most small services, and it is worth knowing before you fork it.

The justfile is the entry point. It builds the server binary with cargo build --locked --release --bin server, runs clippy across the workspace with all features and all targets and denies warnings, allows deprecated, runs cargo deny checks, formats with nightly rustfmt, runs the test suite with cargo nextest run --locked --all-features, and has a check-locked recipe that verifies Cargo.lock is up to date through cargo metadata --locked. Everything is locked, which is the point: the lockfile is treated as part of the build, not as a convenience.

The dependency update policy is unusual and stated plainly. For adding, updating and removing dependencies, and for generating a lockfile, the README tells contributors to use the nightly Cargo commands, so that the seven-day publication cooldown applies, and it explains why with the command list. The rationale is that a new crate version must sit in the ecosystem long enough before it reaches production, and a pull request dependency gate verifies each proposed lockfile change before the Rust build jobs start.

The container image has three stages. The builder uses the reg.mini.dev rust image with BuildKit cache mounts and builds with --locked --release --bin server --features jemalloc. A debug stage on ubuntu:latest gives you a shell inside the container. The production stage is a gcr.io distroless cc-debian13 image, which is the right choice for a service whose only job is to move bytes. Both runtime stages copy the binary in and set it as the entrypoint, and images are published to the GitHub container registry under s2-streamstore/cachey.

The release record shows a project shipping every couple of weeks: 0.11.2 on 2026-09-26, 0.11.3 on 2026-09-30, and 0.11.4 on 2026-10-17, with the last push to the repository on 2026-10-20. The licence is MIT, and the repository also carries a deny.toml for dependency policy, a release-plz.toml for release automation, an AGENTS.md and a GEMINI.md, and a replica simulation harness under docs/simulation that the README lists among the development resources.

Editorial conclusion

Cachey suits teams serving large immutable objects from S3-compatible storage where the bottleneck is range read latency and tail latency rather than throughput, and who want a cache that is a separate service they can put in front of the bucket rather than a library they link in. It does not suit you if your objects change, because the whole design assumes immutable blobs and a read-through cache of that kind cannot keep up with in-place mutation. Check three things before you adopt it: that your client can issue precise byte ranges, because the fetch API requires a Range header and a request without one is not served, that your object sizes justify a 16 MiB page, since a bucket of small objects will cache badly at that granularity, and that you can operate a second service, because the health and throughput endpoints exist so a load balancer can make decisions about it. The newest release is 0.11.4 from 2026-10-17 and the last push to the repository was on 2026-10-20.

Frequently asked questions

Official sources

1. License: MIT
2. Project website
3. README
4. Releases
5. s2-streamstore/cachey on GitHub