sqliteai/warp: running Kimi K3 from NVMe when RAM runs out

The problem WARP attacks: weights that do not fit in RAM

A 2.78-trillion-parameter mixture-of-experts model occupies 1.42 TB in its published form, and the converted WARP container for Kimi K3 is 982 GB. Neither number fits in a 64 GB laptop. WARP's answer is to treat RAM as a cache rather than as storage: the model trunk stays resident, and the experts a token actually routes to are read from disk. The README states that only about 4% of K3's parameters are active for each token, which is what makes the split viable at all. The audience is narrow on purpose. This is for someone who owns a machine with fast internal NVMe and wants to run the full model rather than a distilled or pruned version, and who is willing to accept 0.6 tokens per second to do it. The project describes itself as intentionally narrow and as an experiment in how far local inference can be pushed when weights live mostly on storage. It is not a serving stack for a team, and the README does not present it as one.

How the expert streaming and lookahead router work

The container is laid out so that a single expert requires one aligned read. Those reads overlap with computation, and unused RAM becomes a bounded expert cache. A lookahead router predicts which experts the next layer will need and starts reading them early; the README is explicit that the real router still makes the decision, so the prediction changes timing and not the result. Quantization is not uniform: experts use 3-bit residual vector quantization, while the more sensitive shared weights stay at 4 or 8 bits. The KV cache is the other half of the trick. K3 uses linear attention and a compressed latent KV cache, so at 4K context the cache is about 0.21 GB instead of 11.25 GB. The engine needs 29.19 GB to open K3, and the rest of available memory is used to avoid repeated disk reads. On the test machine the default budget is 46.39 GB, of which 17.56 GB is the expert cache. The design is documented in docs/ENGINE.md, docs/EFFICIENCY.md and docs/FORMAT.md, with docs/KDA.md covering Kimi Delta Attention. Since 0.7.2 the expert banks can also be spread across drives with WASTE_BANK_SHARDS, reading expert e from shard e % N so the k experts a token routes to do not queue behind one device; tools/split_banks.py writes the split.

Installing WARP and running a first prompt

The repository builds with make, and the Makefile's default goal is all, which produces the library and the CLI. Accelerators are build-time options rather than runtime flags: the Makefile documents WASTE_ENABLE_METAL=1 for a Metal build, and the release notes for 0.7.1 mention that builds on x86_64 Linux and Windows work again. The Makefile also carries a warning worth reading before you file a bug: the Python recipe interpreter is resolved by running a candidate rather than looking up a name, because on Windows the name python3 on PATH is usually the Microsoft Store App Execution Alias, a zero-byte reparse point that exits 49 and prints an advert instead of running anything. If no interpreter answers, the recipe stays python3 and fails loudly at first use.

make

Running a model needs a converted container. The README's example uses a .waste file and the CLI binary is still named waste:

waste run ~/models/k3.waste 'What is the capital of Italy?'

With no --budget flag the engine picks one itself. The README's sample output shows the line waste: no --budget, using 46.39 GB of 64.00 GB (expert cache 17.56 GB), then the answer, then a footer with tokens, elapsed time, tokens per second and the expert hit and miss counts. That footer is the number to watch on your own hardware, because it tells you whether the cache is doing anything. The examples directory holds small C programs against the embedding API, examples/api_text.c, examples/api_vision.c and examples/api_plan.c, plus JSON chat templates such as examples/chat-k3.json and examples/chat-glm53.json. The README does not document rollback or downgrade steps for a container.

The failure mode: a bigger expert cache can cut throughput eightfold

This is the most useful thing in the README, and it is counterintuitive. Measured across four cache sizes in one process, a 3.32 GB cache gives a 29.1% hit rate and 0.56 to 0.58 tok/s; 17.32 GB gives 36.2% and 0.63 tok/s; 23.32 GB gives 38.4% and 0.07 to 0.09 tok/s; 29.32 GB gives 41.3% and 0.07 to 0.08 tok/s. The hit rate keeps climbing and the bytes read keep falling while throughput drops roughly eightfold. The engine stays inside its budget and the machine does not, so a cache hit becomes a page fault. Giving the process more memory is not always faster, and anyone tuning this by chasing hit rate will make it slower. There is a second trade in num_experts_per_token, a manifest key that K3 ships at 16. Dropping to 8 measured 0.89 tok/s against 0.59 at 16, a 1.49x gain with a working set of 8.50 GiB instead of 17.01 GiB, at a KL divergence of 0.037 from top-16. The README says that reproduces top-16's greedy continuation on the prompts tested, while top-4 does not: its next-token distribution still looks close but it stops following the prompt within a few tokens, which is why the gate is a continuation and not a KL. The default stays 16. Storage is the other hard boundary. A cold K3 token reads about 17 GB of experts; the internal SSD sustains 12.78 GB/s and a tested USB enclosure managed 0.94 GB/s. On a 32 GB machine the README says the model can open but will page heavily.

Where WARP sits next to llama.cpp and vLLM

The obvious comparison is llama.cpp, which also runs quantized models locally in C and C++. The difference is in what each one assumes about memory. llama.cpp's quantization work is aimed at making a model fit in RAM or VRAM, with mmap as a fallback when it does not. WARP assumes from the start that the expert set will not fit and builds the paging, the aligned per-expert container layout, the lookahead router and the bounded cache around that assumption. That is why WARP can name a 2.78-trillion-parameter model and a 29.19 GB minimum RAM in the same table, and why its throughput on K3 is 0.6 tok/s rather than a number anyone would serve from. A server-oriented engine such as vLLM makes the opposite bet: keep the weights resident, batch aggressively, and get throughput from concurrency. WARP has no batching story in the README at all. If your model fits in RAM, WARP's disk path is overhead you are paying for nothing. The project is also explicit that the code is written by LLMs while the ideas, hypotheses, priorities, tests and decisions are human, which is a governance choice you should weigh for a component you embed.

Maintenance, licence and the cost of upgrading

The repository is not archived and the last push was on 2026-09-16, with v0.8.0 released on 2026-09-15, v0.7.2 on 2026-08-28 and v0.7.1 on 2026-08-27. The release titles are informative rather than cosmetic: 0.7.1 was about building on x86_64 Linux and Windows again, and 0.7.2 added bank striping, K2 tool calling and three oracles. That cadence means upgrades are real work, not a version bump, because the container format and the manifest keys are part of the interface. The README does not document a container migration path, so a format change in a future release is a conversion you would have to redo. Converting a 1.42 TB model into a 982 GB container is itself a large one-time cost, and the docs point to docs/FORMAT.md for the on-disk layout. The licence is Apache-2.0, with a NOTICE file in the repository root. Apache-2.0 permits commercial use and modification and includes a patent grant; it also requires that you keep the NOTICE and licence text with redistributions. Model weights are a separate question from the engine's licence, and the README says nothing about the terms attached to the Kimi K3, DeepSeek or GLM weights you would convert. Check those separately; this is not legal advice.

Editorial conclusion

WARP fits engineers who already have a fast internal NVMe drive and want to run a frontier MoE model locally with no runtime dependencies, and who can accept sub-1 tok/s on K3. It is the wrong tool on a 32 GB machine or with the container on a USB enclosure: the README reports 0.94 GB/s there against 12.78 GB/s internal, and a cold K3 token reads about 17 GB. Before committing, run the smallest container you care about and watch the printed hit rate and bytes read, then check whether num_experts_per_token in the manifest is worth moving off its default of 16.

Frequently asked questions

Official sources

1. License: Apache-2.0
2. Project website
3. README
4. Releases
5. sqliteai/warp on GitHub