KVMem: virtualized key value cache so long context fits in a bounded GPU set

The problem: the key value cache is what does not fit

KVMem manages long conversations and agent tool histories within a bounded GPU working set, and the mechanism is stated plainly: native key value history lives in host RAM, with optional SSD tiering, and retrieval brings selected history onto the GPU.

That is the whole idea, and it is aimed squarely at long context inference on hardware that cannot hold it. A transformer keeps a key and a value for every token at every layer, and that state grows with the conversation, not with the question. When a chat session runs long enough, the cache is what pushes the context out of video memory, and the usual responses are to shorten the conversation or to move the whole thing to the CPU, which is slow.

KVMem's answer is to treat the cache as a managed resource rather than as a by-product of the forward pass. The history stays in host memory, which is plentiful, and only the portion a query needs is brought onto the GPU.

The project is written in C++ and the repository is the multi-backend source for version 0.18.0. The recent releases are tightly paced, with the hardware trial release for older GPUs, the stable 0.18.0 and the release candidate build all landing in the same week in October 2026, and the last push to the repository was on 2026-10-19.

One housekeeping note that matters if you follow old links. The repository is now kvmem/kvmem, and GitHub redirects the former kvmem/kvmem-llama.cpp URL, including existing release and download links, and the README asks that the old name not be reused so that the redirect is not replaced.

Two backends and one memory contract

The 0.18.0 release is described as multi-backend source, and the structure is the interesting part.

There is shared KVMem memory, retrieval and storage contracts, and then separate native inference and serving implementations for llama.cpp and for NInfer. The upstream versions, the integration patches and the full provenance are pinned in a backend manifest, at backends/versions.json.

That manifest is what makes the multi-backend claim auditable rather than a promise. A system that patches an upstream inference engine has to record exactly which upstream it is based on and what was changed, otherwise two builds of the same project are indistinguishable. Pinning versions, patches and provenance in one file is the mechanism that lets you rebuild a given configuration.

The two backends are not interchangeable skins. NInfer is the one that gets the Windows packages, the embedded Web UI and the serving features described later, while llama.cpp appears in the earlier packages and in the source build path. The repository carries patches and scripts directories, which is where the integration work with each upstream lives.

The 0.18.0 binary release covers Windows with NInfer only, and the earlier packages retain their own feature and parameter scope, with mainline updates not adding 0.18.0 features to them. That is an honest statement about the state of the tree, and it means the source and the binaries are not always at the same level of capability.

RAM and SSD tiering, without requantization

The tiering model is the part of the design that makes it more than a cache with a different address space.

Active and idle histories share quotas, so a conversation that is being worked on and a set of older conversations that are merely retained compete for the same memory rather than each getting its own. Complete native key and value data, quantization metadata and MTP KV can move to SSD and are read back according to the selected history, without requantization.

That last clause is the significant one. A quantized cache that is written to slower storage and read back is normally either requantized, which loses accuracy, or stored in a form that has to be reconstructed. Keeping the quantization metadata alongside the data means the values come back in the same numerical form they went in, so a session that has been sitting on SSD for a week resumes with the same numbers it had.

The history selection itself is query based, and it preserves the original logical positions. That matters for a chat model, where the order of the turns is part of the meaning, so a retrieved history that is re-ordered or renumbered changes what the model sees even if the content is the same.

The defaults in the release are worth knowing since they are the numbers you will live with. The Windows launchers enable history weighting with a lambda of 0.5 and a top-frac of 0.9, and GQA max. Sink defaults to 12288 tokens, 12K, and recent to 1024 tokens, 1K. The NInfer minimum remains 64, and values from 0 to 63 are rejected.

State reuse across turns, tools and cancellations

The other half of the design is what happens when a request does not simply continue the conversation.

Multi-turn requests, tool continuations, edits and cancellations restore state according to each backend's exact-prefix and checkpoint rules. In other words, the KV cache is not a linear tape. A tool call that goes out and comes back, an edit to an earlier message, or a cancelled request all leave the cache in a state that has to be reconstructed, and the rules for how to reconstruct it are per backend, because the two engines checkpoint differently.

This is the part of a caching system that is easy to get wrong. A cache that assumes the conversation only ever grows forward will break the first time a user edits a message or an agent cancels a tool call, and it will break in a way that produces plausible but wrong output rather than an error.

The release notes are precise about the boundaries, which is a good sign for a project at this stage. Only complete tool calls are published, and unsupported schemas produce explicit errors rather than being silently dropped. Responses reasoning returns an empty summary array while the raw reasoning text is retained, so a client asking for a summary gets an empty summary and the text is still there for a client that wants it. Image detail accepts low, high or original and maps them to auto. And native tool constraints compose with thinking, JSON output and speculative decoding, so asking for a structured answer from a model that is also using tools is a supported combination rather than a conflict.

The serving side: NInfer, clients and execution options

NInfer is where the serving features live, and the list is long enough to be worth taking in two parts.

The client side covers Chat Completions, Responses and Anthropic Messages, with tool choice and strict schema constraints enabled by default. The execution options include nine key value formats, fixed or adaptive MTP, ngram, concurrent text requests and CPU or resident image requests, and a layer pipeline across up to eight GPUs of the same compute capability.

The layer pipeline is the part that is most constrained by hardware. Eight GPUs of the same compute capability, and not a mixed set, because the layers have to be split along a boundary where the tensors have the same shape and the same instruction set. It also means the feature is unavailable on the single consumer cards the packages are mostly aimed at.

The stable release retains the release candidate engine binaries and updates version metadata, launch defaults, documentation and checksums. A BUILD-INFO.json file records the binary provenance, and the launcher revisions ship through master and updated packages, with the 0.18.0 tag preserving the initial source snapshot.

There is also a documentation set in the repository, including a capability and qualification guide for the supported combinations, so the support matrix is written down rather than implied by the download table.

Packaging, hardware targets and the limits of the releases

The distribution section tells you what you can actually download today, and the answer is narrower than the source.

The stable 0.18.0 packages are Windows x64 builds for the RTX 30, 40 and 50 series, one per architecture, at sm86, sm89 and sm120a, and each release carries a SHA256 sums file. The packages include the NInfer server, the embedded Web UI, the launchers and the CUDA runtime dependencies. Models and the llama.cpp worker are supplied separately, which is the right split, since a model is a legal and practical question of its own.

Running a package needs a CUDA 13.2 compatible NVIDIA driver and the Microsoft Visual C++ 2015 to 2022 x64 runtime, and notably it does not require the CUDA Toolkit or Visual Studio, so the runtime requirements are much smaller than the build requirements. NVFP4 model weights require the RTX 50 package.

The RTX 20, TITAN RTX and Tesla V100 machines are served by a release candidate marked as intended for hardware trials. The README is careful about the status of those packages, saying their build targets, dependencies and archives have been checked, but that physical GPU qualification is still pending. That distinction between a checked build and a qualified one is the kind of wording a project should use and not the kind a reader should skim.

The 0.18.0 binary release covers Windows with NInfer only. Earlier packages cover Windows, Linux and WSL2 with llama.cpp and GGUF models on NVIDIA, a ROCm beta for AMD, and a dedicated build for Windows Bonsai 2, but they keep their own feature and parameter scope and do not receive the 0.18.0 features.

The final practical constraint is about paths. The README instructs you to use English paths for the package and the model, and the full path including the username, the parent directories and the filename must not contain Chinese characters.

Editorial conclusion

Adopt KVMem if you run local inference and the limit on your context length is the key value cache rather than the model, because the whole design is about keeping the history out of video memory and retrieving what a query needs. Do not adopt it if you are deploying on Linux with AMD hardware, since the current 0.18.0 binary release covers Windows with NInfer only, and the earlier ROCm packages do not gain the 0.18.0 features. Before you download, match the package to your GPU architecture, since the packages are built per compute capability, and check the SHA256 sums file that sits next to each download.

Frequently asked questions

Official sources

1. Issues
2. kvmem/kvmem on GitHub
3. README
4. Releases