kvcached: virtual-memory KV cache for shared GPUs
Virtualized Elastic KV Cache for Dynamic GPU Sharing and Beyond
At a glance
- What is it?
- kvcached is an Apache-2.0 Python library that decouples GPU virtual addressing from physical KV cache allocation so vLLM and SGLang can share one GPU elastically. The idea is sound and the engine integrations are real; the operational cost is a patched serving stack and a version matrix you have to keep current.
- Who is it for?
- Adopt kvcached if you run more than one model on the same GPU and your workload alternates between them, or if you are building a serverless or compound system where models must come and go on demand. Do not adopt it if you serve a single model at steady utilization, since the virtual-memory indirection buys nothing and you still carry a patched engine.
- Can I use it commercially?
- Yes. Apache-2.0 is a permissive licence: you can use, modify and sell software built on it, as long as you keep its copyright and licence notices.
- Is it still maintained?
- Yes. The repository received new commits within the last day.
- What is it written in?
- Mainly Python, according to GitHub's language statistics.
Answers come from the project's GitHub data, last synced on September 15, 2026, and from our analysis. They are not legal advice.
DEEP OPEN-SOURCE ANALYSIS
The problem: KV cache reserved for the peak, paid for at the idle
A serving engine sizes its KV cache pool at startup. That pool is a fixed reservation of GPU memory, and it stays reserved whether the engine is under load or not. Put two models on one GPU and you partition the memory between them by hand, picking a split that is wrong most of the day: too small for one model's morning burst, wasted for the other's quiet afternoon. The README frames this as the reason kvcached exists, describing rigid memory partitioning as the current default for concurrent deployment.
The audience is narrow and specific. It is whoever operates the GPU, not whoever writes the model code. If you run one model at steady saturation, the reservation is not waste, it is capacity you are using, and this project has nothing to offer you. The case kvcached targets is dynamic and mixed: several models behind one endpoint, a serverless pool where models spin up and down, or a compound system that chains models and cannot afford each stage to hold a private cache. The README lists exactly those three as its example use cases.
Virtual addressing for KV blocks, and what backs it
The mechanism is stated plainly in the README: kvcached decouples GPU virtual addressing from physical memory allocation for KV caches. A serving engine reserves virtual memory at startup and only later backs that reservation with physical GPU memory when the cache is actually used. Allocation becomes demand-driven rather than up-front.
The consequence is that two engines can hold large virtual reservations that overlap in address space without both paying physical memory at the same moment. When one goes idle, its physical pages are reclaimable; when it wakes, they are mapped back. The README calls this elastic and demand-driven allocation and ties it to better utilization under dynamic workloads.
What the supplied material does not give is the mapping layer itself. There is no description of the page table structure, the granularity of a mapping, the latency of a remap, or what happens when the physical pool is exhausted while virtual reservations still stand. The README links an arXiv paper on a GPU OS vision and a second on multi-LLM serving, so the design detail lives there rather than in the repository front page. Treat the README as an accurate statement of intent and a poor substitute for reading those papers before you depend on the failure semantics.
Engine integration, prefix caching and sleep mode
kvcached is not a server. It is a library that plugs into SGLang and vLLM, and the version table in the README is the most operationally useful thing in it. SGLang is supported from v0.4.9 and tested up to v0.5.15. vLLM is supported from v0.8.4 and tested up to v0.24.0. That is a wide tested band on the vLLM side, which suggests the integration is maintained against upstream churn rather than pinned to one release.
Attention coverage spans MHA, GQA, MLA, sliding window and hybrid layouts, with DeepSeek-V3, Qwen3-8B, GPT-OSS-20B and several Gemma variants named as examples. MLA support in vLLM arrived in the 2026-03 update, alongside pipeline parallelism. GPT-OSS support in SGLang was updated to v0.5.9 in the same window. The README points to issue #425 for per-model results on each engine and KV layout, which is where you should look before assuming your model is covered; a named example model is not the same as a verified layout for yours.
Prefix caching landed in 2026-04: automatic prefix caching for vLLM and RadixCache for SGLang, with cross-request prefix reuse under elastic memory management. The README describes a configurable memory bound for APC and points at examples/09_prefix_caching. That bound matters, because prefix reuse and elastic reclamation pull in opposite directions. A cache that must survive to be reused is a cache you are less willing to evict. The README does not explain how the two policies interact, and that interaction is the first thing I would probe in a test deployment.
Sleep mode and a frontend router are listed as features. The router sends requests to the target model; sleep mode puts an idle model down. Together they are what makes the multi-model story work, since a sleeping model is the clearest case of physical memory that can be handed to someone else. Again, the README names the capability without describing the wake latency or what a request in flight sees when its model is put to sleep.
Getting it running: engines, the CLI and the config surface
The README does not print a pip install line or a full launch command, so I will not invent one. What it does establish is the shape of the setup. You need Python 3.9 to 3.13, one of the two supported engines at a supported version, and the kvcached library installed into that engine's environment. The integration is per-engine, so a vLLM deployment and an SGLang deployment are separate configurations even on the same host.
The one concrete control surface named in the material is the memory control CLI, described as enforcing memory limits with the kvcached CLI. That is your lever for capping how much physical memory the elastic pool may claim, and it is the setting that decides whether co-located models coexist or starve each other. Beyond that, the README names a configurable memory bound for automatic prefix caching and refers to examples/09_prefix_caching for the prefix caching configuration.
That is thin, and it is the weakest part of the repository front page. For an operator-facing library whose entire value proposition is memory policy, the absence of a documented config key list on the README is a real gap. The material points to DeepWiki, kvcached.org and the examples directory as the places where the configuration is actually documented. Budget time to read those before you plan a rollout, and expect to read the engine-side integration code if you need to know what the defaults are.
Where it is the wrong tool, and what can go wrong
The clearest wrong-tool case is a single model at high steady utilization. Elastic allocation only helps when there is slack to reclaim. If your cache is full because your traffic is genuinely heavy, virtualizing the address space changes nothing about the physical bytes you need, and you have added a mapping layer between the engine and its memory for no return.
The second case is version drift. The support table is explicit about tested upper bounds, and both engines move fast. If you track vLLM or SGLang main rather than a release, you are outside the tested band by definition. The 2026-03 and 2026-04 updates show the integration being actively reworked for MLA, pipeline parallelism and prefix caching, which is a good sign for maintenance and a bad sign for anyone who wants a frozen target.
The third is the one the README does not address: what happens under exhaustion. With fixed reservation, an engine that runs out of KV memory has a known behavior, typically preemption or rejection. With elastic allocation, the failure depends on the reclamation and remapping path, and the README provides no description of it. If you are running latency-sensitive traffic, that is the behavior you need to characterize yourself before you put it in front of users. I would also treat the interaction between prefix caching and reclamation as an open question rather than a solved one, since the README presents both features without saying how the memory bound arbitrates between them.
Alternatives: fixed partitioning, and why the difference is structural
The default alternative is what you are already doing: give each model a fixed slice of GPU memory and run them side by side. The difference is structural, not a matter of tuning. Fixed partitioning decides the split at launch and cannot change it without a restart, so you size for the peak of each model and accept the sum. kvcached keeps the split undecided until pages are actually touched, so the sum of peaks no longer has to fit. That is the whole trade: you give up a static, easy-to-reason-about memory budget in exchange for a dynamic one whose worst case is less predictable.
A second alternative is running one model per GPU and routing between GPUs. That sidesteps the sharing problem entirely and is the right answer when you have spare GPUs, because it removes the mapping layer, the version constraints and the co-tenancy risk in one move. kvcached exists for the case where you do not have spare GPUs and cannot get them. Red Hat's Sardeenz is named in the README as building on kvcached for dynamic multi-model serving with Kubernetes and OpenShift support, which is a sign that the co-tenancy problem is being attacked at the orchestration layer rather than solved by adding hardware.
Maintenance, licence and the upgrade treadmill
The release cadence is roughly two months between v0.1.3 in January 2026, v0.1.4 in March and v0.1.5 in April, and the last push to the repository is dated 2026-08-23. The version numbers are still 0.1.x, which is honest about maturity. Nothing here is archived or abandoned.
The upgrade cost is the engine version matrix. kvcached sits between your engine and the GPU, so an engine upgrade is not a routine dependency bump; it is a change to the layer kvcached patches. The tested ranges in the README (vLLM up to v0.24.0, SGLang up to v0.5.15) are the boundary of what the maintainers have exercised. Plan on validating each engine upgrade against kvcached rather than assuming compatibility, and pin both sides.
The licence is Apache-2.0, which permits commercial use and modification and includes an explicit patent grant. That is the permissive end of the spectrum and is unlikely to be a blocker for internal deployment or for a product. Note only that Apache-2.0 requires you to preserve notices and state changes when you redistribute, so if you fork the integration for your own engine, that obligation follows the fork. This is a description of the licence text, not legal advice; your counsel should confirm how it applies to your distribution model.
Editorial conclusion
Adopt kvcached if you run more than one model on the same GPU and your workload alternates between them, or if you are building a serverless or compound system where models must come and go on demand. Do not adopt it if you serve a single model at steady utilization, since the virtual-memory indirection buys nothing and you still carry a patched engine. Before committing, verify three things: that your exact engine version appears in the support table (vLLM >= v0.8.4 tested to v0.24.0, SGLang >= v0.4.9 tested to v0.5.15), that your model's attention type and KV layout are covered in issue #425, and that the kvcached CLI memory limit you set leaves room for the sleep-mode wake path you plan to use.
Community notes