R-KV: decoding-time KV cache compression for reasoning models

The memory wall that reasoning traces create

Long chain-of-thought generation is a decode-time problem, not a prefill problem. A model that emits tens of thousands of thinking tokens keeps every one of those tokens' keys and values resident in the KV cache, and the cache grows linearly with sequence length while the arithmetic per step stays roughly flat. The result is that a reasoning workload becomes memory-bound well before it becomes compute-bound, and the number of concurrent requests a server can hold falls as each request's context stretches out.

R-KV targets that specific asymmetry. The README describes it as discarding repetitive tokens on the fly, with the stated goal of full-accuracy reasoning at a fraction of the memory. The intended user is someone serving R1-style distilled models on math and reasoning benchmarks, or running the same models behind a production-style server, who wants more requests in flight without retraining or changing the model weights. It is not a training technique and it does not touch the model itself.

What the redundancy scoring actually does at decode time

The mechanism is a scoring and eviction loop that runs during decoding rather than after prefill. The HuggingFace entry point exposes the knobs directly: kv_budget, window_size, mix_lambda, divide_method and divide_length. The divide_method value shown in the README example is step_length with divide_length 128, which suggests the sequence is partitioned into fixed-length blocks and scored at that granularity. mix_lambda 0.1 blends two scoring signals, and window_size 8 keeps a recent-token window that is exempt from eviction.

That window is the interesting design choice. Pure redundancy scoring can delete a token that looks repetitive but is still being attended to by the next few steps, so a sliding window of protected recent tokens is the standard guard against that failure. The name of the method, redundancy-aware, implies the score is derived from how similar a token's representation is to others already retained, and the vLLM port notes batched cross-layer redundancy scoring and a cross-rank eviction-score all-reduce for tensor parallelism. That last detail matters: if every tensor-parallel rank scored independently, ranks could evict different tokens and the sharded attention would read inconsistent caches. The all-reduce exists to keep the eviction set identical across ranks, and the release note states that every TP rank evicts the identical set.

The SGLang port adds a fused Triton redundancy kernel and a two-phase compaction step, plus compression-aware admission, which means the scheduler accounts for the compressed footprint when deciding whether to admit a new request. Physical eviction on the FlashInfer decode path is claimed, as opposed to masking, so the freed blocks return to the allocator. In the vLLM port that return path is named FREE_BLOCKS.

Three install paths with different levels of commitment

The lowest-commitment path is the HuggingFace implementation. Install the minimal dependencies, then build the package in editable mode:

pip install -r requirements.txt pip install -e .

The evaluation toolkit is built separately, and the README points at HuggingFace/evaluation/latex2sympy with its own pip install -e . before the toolkit's requirements. Once that is done, scripts/run.sh runs R1-like models on math benchmarks, or you can call the script directly with explicit flags:

python3 ./run_math.py --dataset_path ./data/aime24.jsonl --save_path ./outputs/output.jsonl --model_path deepseek-ai/DeepSeek-R1-Distill-Llama-8B --max_length 32768 --eval_batch_size 1 --method rkv --kv_budget 1024 --window_size 8 --mix_lambda 0.1 --divide_method step_length --divide_length 128 --do_s

Note that the HuggingFace path defaults to FlashAttention 2 through attn_implementation="flash_attention_2" in the model loading call.

The serving paths are heavier. Both vLLM and SGLang use a patch-not-fork layout: a script clones a pinned upstream checkout, drops in an rkv/ directory, and applies a wiring patch. For SGLang that is bash scripts/apply_rkv.sh inside SGLang/, followed by an editable install of sglang-src/python against the SGLang wheel index and then requirements-rkv.txt. For vLLM it is the same pattern against vLLM v0.25.1, followed by pip install -e vllm-src, which the README notes needs CUDA and a GPU because it is a source build.

Configuration in the vLLM port is by environment variable: VLLM_V1_R_KV_BUDGET and VLLM_V1_R_KV_BUFFER. Compression activates only when budget and buffer are both greater than zero, so a zero in either place silently disables the whole thing. The README also states explicitly that you should not pass --enforce-eager, because R-KV auto-selects PIECEWISE cudagraph. The Nano-vLLM reference port is the opposite: it requires enforce_eager=True.

A stale data file that produced wrong scores for weeks

The repository's own news section contains the most useful warning in it. Before commit e9f54c45, the shipped data/gsm8k.jsonl carried a stale generation field from an earlier experiment, and eval_math.py preferred that field over the output produced by the fresh run. The effect was that GSM8K scores froze near 40 percent regardless of method or budget. MATH and AIME24 were unaffected. The maintainers ask anyone who computed GSM8K numbers with the repo before that commit to rerun them, and cite issues #26 and #23.

This is worth stating plainly because it is a data-plumbing bug, not a modelling one, and it is exactly the kind of bug that makes a compression method look broken or look neutral when it is neither. Any accuracy claim you read about R-KV on GSM8K that predates that fix should be treated as unverified. The current state, per the release notes, is that all four serving integrations plus the HuggingFace path are GPU-validated on A100 with R-KV on and off, with artifacts under tests/smoke/ and results/validation-2026-07-02-a100/.

The second limitation is structural. There is no packaging story beyond the patch scripts. If your serving stack is not one of vLLM v0.25.1, SGLang v0.5.14, Nano-vLLM or Mini-SGLang, there is nothing to install. Upstream version drift breaks the patch, and the patch is the integration. A team on a different vLLM release is not a supported user; they are a fork author.

Third, the reported throughput gains are described as memory-bound wins that appear as the KV pool shrinks. The release notes give the range as plus 17 to plus 32 percent tokens per second at gpu_mem 0.40 to 0.25, with lossless accuracy at budget=512 on GSM8K. If your GPU memory fraction is not tight, there is less cache pressure to relieve, and the benefit shrinks with it. Compression on an underutilised card is overhead.

How this differs from attention-sink and heavy-hitter eviction

The obvious comparison is with eviction policies that keep tokens receiving high cumulative attention and drop the rest, the family usually associated with H2O and with attention-sink style retention. The difference is in what counts as disposable. Attention-score methods rank tokens by how much attention they have absorbed, which is a query-dependent signal. R-KV ranks by redundancy, which is a property of the stored representations themselves: if two cached entries carry near-identical information, one of them is not paying for itself no matter how much attention it drew.

That distinction has a practical consequence for reasoning traces. Chain-of-thought output is repetitive by construction. Models restate intermediate results, re-derive the same sub-expression, and loop over similar verification phrasing. Attention-based eviction can protect a token precisely because the model keeps looking at it, even when the content is a duplicate of something already in cache. Redundancy-based eviction removes the duplicate instead. The trade-off is that redundancy scoring needs a similarity computation over the cache, which is why the serving ports bother with fused Triton kernels and batched cross-layer scoring. The scoring is not free, and the ports' optimisation work is largely about making it cheap enough to sit inside the decode loop.

The other axis of comparison is masking versus eviction. A masked cache still occupies memory; it just stops some entries from contributing to attention. R-KV's serving ports perform physical eviction and return blocks to the allocator, which is the only version of this that improves how many requests fit. If you only need shorter effective context, masking is simpler and needs no upstream patch.

Licence status and what maintenance you are signing up for

The repository metadata does not list a licence, and no release was retrieved. That is a real blocker for commercial adoption and it is not something to reason around. Without a licence file, the default position is that no rights are granted, and the code is a patch against vLLM and SGLang, both of which carry their own licences that the patch does not override. Anyone planning to ship this in a product needs to resolve the licence question with the authors before writing code against it. I am not giving legal advice here, only noting that the material does not answer the question.

Maintenance cost is dominated by the pinning. The patch-not-fork layout means your upgrade path for vLLM or SGLang is gated on the R-KV patch being rebased onto the new upstream. The vLLM port tracks v0.25.1 and the SGLang port tracks v0.5.14; there is no statement about how quickly either tracks upstream releases. The open TODO list is a reasonable proxy for where the project is: GPT-OSS and QwQ support, VeRL integration, GPQA and liveCodeBench datasets, and expanded Qwen-3 evaluation coverage are all listed as not done. The items that were completed (benchmark coverage across the four ports, and the performance hardening of the vLLM and SGLang ports) are marked as struck through, so the list is being maintained rather than abandoned. Last push to the default branch is 2026-07-20.

If you adopt the HuggingFace path only, your maintenance surface is much smaller: a Python package, a requirements file, and a data directory. The serving ports are where the ongoing rebase work lives.

Editorial conclusion

Adopt R-KV if you are serving long-chain-of-thought models on H100-class hardware and can live with a pinned upstream checkout plus a wiring patch; the HuggingFace path is the least invasive way to reproduce the paper's math benchmarks. Do not adopt it if you need a drop-in library, a published licence, or support for a serving stack outside the four ports listed. Verify first that your serving version matches the pinned tags (vLLM v0.25.1, SGLang v0.5.14), that budget and buffer are both set above zero, and that your GSM8K data file postdates commit e9f54c45 before you trust any accuracy number.

Official sources

1. Issues
2. Project website
3. README
4. Zefan-Cai/R-KV on GitHub