kvpress: A Test Bench for KV Cache Compression Methods
LLM KV cache compression made easy
At a glance
- What is it?
- NVIDIA's kvpress packages ten training-free KV cache compression methods behind a single transformers pipeline, so the real question is not whether it compresses but whether the method you pick survives your workload.
- Who is it for?
- Adopt kvpress if you are a researcher or inference engineer who wants to compare existing KV cache compression methods, or prototype a new one, without writing the pruning plumbing yourself. Do not adopt it as a drop-in production memory fix: several presses require flash-attention-2 and a CUDA device, decoding compression only accepts ScorerPress subclasses, and the README documents no quality guarantees per model or context length.
- Can I use it commercially?
- Yes. Apache-2.0 is a permissive licence: you can use, modify and sell software built on it, as long as you keep its copyright and licence notices.
- Is it still maintained?
- Yes. The repository last received commits 1 day ago.
- What is it written in?
- Mainly Python, according to GitHub's language statistics.
Answers come from the project's GitHub data, last synced on September 15, 2026, and from our analysis. They are not legal advice.
DEEP OPEN-SOURCE ANALYSIS
The memory bill that kvpress is trying to reduce
The README opens with an arithmetic argument rather than a promise. Handling 1M tokens with Llama 3.1-70B in float16 requires up to 330GB of memory, because the KV cache grows linearly with sequence length. That figure is the whole motivation: the weights are fixed, the cache is not. The audience is stated just as plainly. kvpress targets researchers and developers in the compression field, and its stated aim is to simplify the development of new methods. That framing matters when you evaluate it. A library built to make new methods easy to write will optimise for pluggability and comparability over the last few percent of inference latency. If you want a single tuned kernel to bolt into a serving stack, this is a different kind of project.
Presses, ScorerPress and the compression_ratio contract
Every press inherits from BasePress, and all current presses are training free. The main family is ScorerPress, which assigns each KV pair a score and prunes the lowest-importance pairs. The README lists ten of them, each tied to a paper: RandomPress (random score), KnormPress (inverse key norm), SnapKVPress (average attention weight of the last queries), ExpectedAttentionPress (expected attention weight during generation), StreamingLLMPress (keep initial and recent tokens), TOVAPress (last query's attention averaged across heads), ObservedAttentionPress (attention observed during prefill), QFilterPress (project keys onto the main SVD component of the query vectors), PyramidKVPress (more cache budget in lower layers, less in higher layers) and LagKVPress (KV lag-relative information). The unifying interface is the compression_ratio attribute, which measures how much of the cache is kept. That single knob is what makes the presses comparable, and it is also the abstraction's weak point: the ratio says nothing about which tokens survive. Two presses at 0.5 can behave very differently on the same prompt, so the ratio is a budget, not a quality setting.
How compression is wired into the transformers pipeline
kvpress registers a custom transformers pipeline under the name "kv-press-text-generation" when the package is imported. The pipeline handles chat templates and tokenization, so the call site stays small: build the pipeline, construct a press, pass it in. In the README example the model is Qwen/Qwen3-8B, the press is ExpectedAttentionPress(compression_ratio=0.5), and the answer comes back as pipe(context, question=question, press=press)["answer"]. Compression happens during the prefilling phase by default, and the README notes that it is applied only to the context tokens, which lets you reuse one compressed context across different questions. That is a deliberate evaluation affordance: it isolates the effect of compression from the effect of the question. The data flow is therefore prefill-time pruning, not a change to the attention kernel's arithmetic. The press scores KV pairs and drops the low-scoring ones before generation begins.
Decoding compression and its ScorerPress-only constraint
The experimental DecodingPress wrapper moves compression into the generation loop. It takes a base_press, a compression_interval (default 512 steps), a target_size (default 2048 tokens) and a hidden_states_buffer_size (default 256, settable to 0 for presses that do not need buffered hidden states). The semantics differ from prefill compression in a way worth reading twice. DecodingPress does not use a compression ratio; it uses target_size, compresses every compression_interval steps, and derives the ratio automatically so the cache lands at target_size. The README example pairs KnormPress with compression_interval=10 and target_size=512. The stated limitation is blunt: not all existing presses are fully compatible with DecodingPress, because compression during decoding differs fundamentally from compression during prefilling, and only ScorerPress subclasses are supported as base presses. That excludes the wrapper from presses that are not ScorerPress descendants, and it means the prefill and decode paths are not interchangeable.
Getting it installed and the optional extras that matter
The base install is pip install kvpress. For a local checkout the README uses uv: git clone https://github.com/NVIDIA/kvpress.git, cd kvpress, uv sync. Optional dependencies are pulled with uv sync --extra eval --extra flash-attn. That flash-attn extra is not cosmetic. The decoding example builds the pipeline with model_kwargs={"attn_implementation": "flash_attention_2"} and device="cuda:0", and the README also describes the press set as benchmarked using transformers. So the documented path assumes a CUDA device and, for that example, a FlashAttention-2 build. If your environment is CPU-only, or your serving stack uses a different attention implementation, the README does not describe a supported configuration for those presses. The repository also ships a Wikipedia notebook demo and a Colab notebook, plus a Hugging Face Space and leaderboard, which is where the comparative claims live rather than in the README text.
Where kvpress is the wrong tool
Three boundaries are visible in the material. First, scope: the README frames the project around implementing and benchmarking methods, not around serving. There is no mention of a serving integration, a batching story, or continuous batching, so treating it as a production memory layer is an assumption the documentation does not support. Second, the decoding path is labelled experimental and restricted to ScorerPress base presses, so any plan that depends on periodic decode-time compression inherits that restriction. Third, and most important for evaluation, the README gives no per-press quality results, no model coverage table, and no statement about which compression_ratio is safe for which task. The papers are cited, but a paper's evaluation conditions are not your prompt distribution. The honest position is that kvpress gives you the machinery to measure this and does not measure it for you. If you need a guarantee before you ship, this library is the instrument, not the answer.
How it differs from writing your own pruning hook
The obvious alternative is to implement pruning directly against transformers, hooking the attention layers and masking KV pairs yourself. That approach has one advantage: no dependency on kvpress's press interface, and no constraint that your method be expressible as a ScorerPress. The difference in approach is the abstraction. kvpress fixes the contract at BasePress with a compression_ratio attribute, and for the scorer family at ScorerPress with a per-KV-pair score. Anything that fits that shape, including a method you are developing, gets the pipeline, the tokenization handling and the context-only compression behaviour for free. Anything that does not fit, such as a method that restructures the cache rather than scoring and pruning it, will fight the interface. DecodingPress makes this explicit by accepting only ScorerPress subclasses. So the trade is real in both directions, and it is a trade about how your method is shaped, not about which library is faster.
Version cadence, licence and what to check before adopting
The release history shows v0.5.2, v0.5.3 and v0.5.4 between April and July 2026, with the last push to main in September 2026 and no archived flag, so the project is active on a roughly quarterly release rhythm. That cadence implies an upgrade cost: pin a version, read the release notes between it and your target, and assume the experimental DecodingPress surface may change, since it is described as new and experimental in the README itself. The licence is Apache-2.0, which permits commercial use and modification and includes an explicit patent grant; the LICENSE file in the repository is the authoritative text and this is not legal advice. Two things to verify first, both answerable from the repository without running anything: whether your attention implementation appears in the documented examples, and whether the press you intend to use is a ScorerPress if you plan to use DecodingPress.
Editorial conclusion
Adopt kvpress if you are a researcher or inference engineer who wants to compare existing KV cache compression methods, or prototype a new one, without writing the pruning plumbing yourself. Do not adopt it as a drop-in production memory fix: several presses require flash-attention-2 and a CUDA device, decoding compression only accepts ScorerPress subclasses, and the README documents no quality guarantees per model or context length. Before committing, verify that your chosen press runs on your attention backend, measure the compression_ratio your task tolerates, and read the Apache-2.0 patent grant in the LICENSE file.
Community notes