Hypura treats NVMe as a third memory tier for Apple Silicon inference

Three tiers, and a scheduler that picks between them

Hypura is described as a storage-tier-aware LLM inference scheduler for Apple Silicon, and the problem it addresses is specific. Consumer Macs ship with fast unified memory and fast NVMe, but the capacity is limited, and a 32 GB M1 Max cannot naively load a 40 GB model, because the operating system will swap until the OOM killer steps in. Hypura's answer is to stop treating memory as one pool. It reads the GGUF file, profiles the hardware including the GPU working set, RAM and NVMe bandwidth, and then solves a placement optimisation that assigns every tensor to a tier. The GPU tier, through Metal, holds attention layers, norms and embeddings and is limited by `recommendedMaxWorkingSetSize`. RAM holds overflow layers accessed through mmap. NVMe holds the rest, loaded on demand through direct I/O with `F_NOCACHE` and `pread`, prefetched ahead of the forward pass. The mechanism is a real scheduler rather than a wrapper, and the repository shows the tooling around it: a `hypura-sys` crate for the native side, a `vendor/` directory for the vendored llama.cpp, a `patches/` directory, benchmarks and benches, and a linear programming solver wired in through the `good_lp` crate with the HiGHS solver enabled.

Three inference modes chosen automatically

The mode is selected from the model size, architecture and available memory, with no manual tuning of pool sizes, prefetch depth or memory budgets, all of which the tool computes from the hardware profile. Full-resident applies when the model fits in GPU plus RAM, with no NVMe I/O and full Metal speed. Expert-streaming applies to MoE models such as Mixtral, where only the non-expert tensors, about 1 GB, stay on the GPU and the expert tensors stream from NVMe through a pool buffer on demand. Dense FFN-streaming covers dense models that are too large for the GPU, such as Llama 70B, where attention and norms stay resident at roughly 8 GB and the feed-forward tensors, around 32 GB, stream from NVMe. The third mode is the generalisation of the second, and it is the reason the project applies beyond mixture-of-experts models. The two commands that make the decision visible are `hypura inspect`, which prints the placement plan without loading the model, and `hypura bench`, which compares the scheduler against a naive baseline.

Expert streaming leans on sparsity and a neuron cache

The MoE path is where the interesting engineering sits, because the speed comes from not doing the work. Only 2 of 8 experts fire per token in the Mixtral case, and the router interception identifies the selected experts in the eval callback so that only the needed expert strides are loaded from NVMe, which the documentation puts at a 75% I/O reduction. A neuron cache then tracks loaded expert slices across tokens, and the documentation reports a 99.5% hit rate from temporal locality, which is what removes most of the I/O after warmup. Co-activation tracking predicts which experts will fire next, so the fetch can start speculatively. The same idea appears in the release notes, where v0.2.0 is titled around Gemma 4, sparse MoE mmap and M5 Pro support, and in the release history, where v0.1.0 is titled around storage-tier-aware LLM inference. The honest reading is that these numbers describe a specific model on a specific machine, and the second half of the design, the prediction, is what would need to hold up on a workload that is not a repeated question.

The published numbers come from one machine

The benchmark table is explicit about its conditions: an M1 Max with 32 GB of unified memory and roughly 5.1 GB/s of NVMe sequential read. On that setup, a Qwen 2.5 14B Q4_K_M of 8.4 GB runs at 21 tokens per second, the same as llama.cpp, because it fits in the GPU working set and the scheduler adds no overhead. A Mixtral 8x7B Q5_K_M of 30.9 GB runs at 2.2 tokens per second in expert-streaming mode with 1.1 GB on the GPU, where the documentation records llama.cpp as out of memory. A Llama 3.3 70B Q4_K_M of 39.6 GB runs at 0.3 tokens per second in dense FFN-streaming mode with 7.8 GB on the GPU, again where llama.cpp is recorded as out of memory. Two caveats belong next to those figures. The first is that a model that does not fit is not thereby fast, and 0.3 tokens per second is well below interactive. The second is that the comparison is against a baseline the same documentation calls naive, and the version of llama.cpp tested is not stated.

An HTTP server that speaks Ollama and OpenAI

Hypura exposes an HTTP server with two compatibility surfaces, an Ollama-compatible API and an OpenAI-compatible `/v1/*` API, and the documentation presents this as a drop-in replacement for tools including OpenClaw, Cursor, Cline, Continue.dev, LangChain and standard OpenAI SDKs. The endpoints are the usual ones: a health check on `GET /`, `GET /v1/models` and `POST /v1/chat/completions` and `POST /v1/completions` for the OpenAI side, with JSON responses or SSE streaming when `stream` is true, and `GET /api/tags`, `GET /api/version` and `GET /api/ps` for the Ollama side. The server is started with:

hypura serve ./model.gguf
# Hypura serving model
#   Endpoint: http://127.0.0.1:8080
#   Ollama API: /api/generate, /api/chat, /api/tags
#   OpenAI API: /v1/models, /v1/chat/completions, /v1/completions

Binding to loopback is the default in that output, which is the right default for a service that loads multi-gigabyte models. The server side of the implementation is an axum application on tokio with CORS support from tower-http, and the whole thing is a single Rust workspace whose manifest lists the package version as 0.2.0.

Everything builds from source, with a vendored llama.cpp

There is no packaged install to speak of. The project builds with Cargo, and you need Rust 1.75 or newer plus CMake for the vendored llama.cpp, with the submodules pulled in:

git clone --recurse-submodules https://github.com/t8/hypura.git
cd hypura
cargo build --release

The binary ends up at `target/release/hypura`. The documentation notes that a Homebrew tap is coming soon, which is the clearest signal of how early this is. The first thing to run is a hardware profile, which is cached after the first run, and the everyday commands are a plain prompt, an interactive chat, the benchmark against the naive baseline and the placement inspection:

hypura profile
hypura run ./model.gguf --prompt "Hello, world"
hypura run ./model.gguf --interactive
hypura bench ./model.gguf
hypura inspect ./model.gguf

The one piece of advice the documentation gives about running is to start with `--max-tokens 10` on untested models before scaling up, which is sensible advice for a tool that decides how to move tens of gigabytes around your disk.

The licence is declared in the manifest and nowhere else

The licence needs a note, because two sources disagree in a way that matters. The Cargo manifest declares `license = "MIT"` for the package, but the repository listing does not include a LICENSE file, and the licence field on the project listing is empty. The repository root does contain a CHANGELOG.md, a CLAUDE.md, a RESEARCH_INTEGRATION_PLAN.md, a `start_hypura_funnel.sh` script, a `docs/` directory including REAL_WORLD_TESTS.md with the detailed model evaluations and agent benchmarks, and a `tests/` directory. On maintenance, the two releases are v0.1.0 on 2026-03-17 and v0.2.0 on 2026-04-08, and the last push to the repository was on 2026-10-03 with the repository not archived, so work is landing on the default branch between tagged versions. The README also points at recorded agent workflows, including a live diagnostic check of remote weather station storage and a Windows client orchestrating multi-step agent actions over a LAN against Hypura on Apple Silicon, which is a different kind of evidence from a benchmark table.

Editorial conclusion

Mac users who want to run a model that does not fit in unified memory, and who understand that a 40 GB model on a 32 GB Mac will be slow rather than fast, will find the tiering design and the placement tooling in Hypura genuinely different from plain mmap. It is the wrong tool if you need a packaged install, since the Homebrew tap is described as coming soon and everything builds from source with a vendored llama.cpp, and the wrong tool if you need a model that fits in memory, since the documentation claims zero overhead in that case anyway. Before you rely on it, run `hypura profile` and `hypura inspect` on your own machine, and start with `--max-tokens 10` on a model you have not tested.

Frequently asked questions

Official sources

1. Issues
2. README
3. Releases
4. t8/hypura on GitHub