KVCache-Factory: One Runner, Many KV Cache Compression Methods

The problem: every KV cache paper ships its own harness

Long-context inference work produces a steady stream of cache reduction methods, and each one tends to arrive with its own evaluation script, its own dataset plumbing and its own idea of what a cache budget means. Comparing PyramidKV against SnapKV or H2O then becomes a porting job rather than a measurement job. KVCache-Factory addresses that by collecting the methods under one interface. The README describes it as a unified playground for KV cache compression, retrieval, merging and quantization, and notes the project started from PyramidKV before being renamed on 2024-11-28 to reflect the broader goal. The intended user is someone running LongBench, RULER or needle-in-a-haystack evaluations on Llama or Mistral models and who needs several methods to answer the same question with the same data path. It is not aimed at someone who wants a serving stack; there is no server, no batching scheduler and no HTTP surface in the material.

One runner, a method flag, and a shared budget argument

The architecture is deliberately shallow. A single script, run_longbench.py, takes --method and dispatches to the implementation registered under that name. The supported list is long and heterogeneous: FullKV as a baseline, StreamingLLM and H2O and SnapKV and L2Norm as eviction or retrieval schemes, Quest for query-aware page selection, NACL for encoding-time eviction, Scissorhands for persistence-based eviction, MiniCache for cross-layer compression, PyramidKV and AdaKV and HeadKV for budget allocation, CAM for merging, ThinK for key-channel pruning, HeadInfer for lossless offloading, MInference for sparse prefill, and KIVI, KVQuant and GEAR for quantization. The unification point is the budget flag. --max_capacity_prompts sets a target KV cache budget per layer, and the README states that PyramidKV redistributes the total budget across layers, which is why the same number means different things depending on the method. A second axis is --kv_cache_granularity, either query_head (described as the default legacy layout) or kv_head, the GQA-efficient layout. The README says the kv_head layout is supported for snapkv, pyramidkv, h2o, streamingllm, cam and l2norm, plus adakv and headkv with GPU validation pending, and that --gqa_score_agg controls how per-query-head scores collapse to a KV head via mean, max or sum. That aggregation choice is a real modelling decision, not a plumbing detail, and the default of mean is worth overriding in a sensitivity check.

Getting it running: clone, requirements, PYTHONPATH

Installation is four commands from the README: git clone the repository, cd into it, pip install -r requirements.txt, then export PYTHONPATH="$PWD:${PYTHONPATH}". The pinned dependency list is short: transformers==4.44.2, torch, and flash-attn>=2.4.0.post1. The README states that flash-attn is optional when using --attn_implementation sdpa or eager but required for FlashAttention v2 experiments, and that it should be installed after torch with pip install flash-attn --no-build-isolation. The MInference integration is kept out of the base requirements and installed separately with pip install -r requirements-minference.txt. A LongBench run looks like this: export CUDA_VISIBLE_DEVICES=0, then python3 run_longbench.py with --method pyramidkv, --model_path pointing at a local or Hugging Face Llama-3-8B-Instruct, --max_capacity_prompts 128, --attn_implementation flash_attention_2, --save_dir ./results_long_bench and --use_cache True. The README notes the quickstart budget of 128 while the PyramidKV paper reports results at 128 and 2048, so a single 128 run is a smoke test, not a reproduction. There is also a wrapper, bash scripts/scripts_longBench/eval.sh, whose positional order is CUDA_VISIBLE_DEVICES, method, max_capacity_prompts, attn_implementation, source_path, model_path, merge_method, quant_method, nbits. Nine positional arguments is a fragile interface; the direct python3 call is easier to audit.

Where the abstraction leaks

The method table is wider than the runner support. The README says so directly: some newer methods currently have narrower runner/model coverage, and it advises checking the runner argument choices before launching large jobs. Llama and Mistral attention paths are named as supported for the main compression methods, which leaves the rest of the model space unaddressed. Backend requirements fragment the matrix further. HeadInfer is described as lossless, head-wise CPU offloading with async prefetch that keeps the full cache rather than approximating it, and it requires --attn_implementation flash_attention_2 while ignoring --max_capacity_prompts entirely. ThinK requires eager attention. So a sweep across methods is not a sweep across one configuration; each method constrains the attention backend, and on a GPU without FlashAttention v2 support the README's own guidance is to fall back to sdpa, which may exclude the methods you wanted to compare. The kv_head GQA layout carries its own caveat: adakv and headkv are listed as supported with GPU validation pending, which is an explicit statement that the path has not been verified on hardware. Treat any kv_head result for those two as provisional.

Quantization paths and their extra knobs

Quantization is folded into the same runner rather than treated as a separate tool. --quant_method accepts kivi, kvquant or gear, --nbits sets the bit width, and --quant_backend defaults to hqq. The README documents --quant_residual_length as a full-precision residual cache window that defaults to max_new_tokens, which is a meaningful default: it means the residual window scales with generation length unless you set it. GEAR additionally takes --rank and --outlier_ratio. Advanced layout controls are --q_group_size, --axis_key and --axis_value, with the README noting KIVI defaults to key axis 1 and value axis 0. Those axis defaults are the kind of detail that silently changes results if you port a configuration from another codebase, so record them alongside any number you publish. There is also a --merge flag with pivot or weighted strategies, which is orthogonal to compression and can be combined with it, and the eval.sh wrapper exposes merge_method and quant_method as separate positional slots for exactly that reason.

The comparison you would otherwise build yourself

The obvious alternative is the original repository for whichever method you care about. PyramidKV, SnapKV, H2O and StreamingLLM each have their own codebases, and KVCache-Factory is explicit that it grew out of PyramidKV and folds those baselines in. The difference in approach is ownership of the harness. A single-method repository gives you the authors' exact configuration for their own paper, which is the right thing if you are reproducing one result. KVCache-Factory gives you a shared runner and a shared budget flag, which is the right thing if you are asking whether layer-wise budget allocation beats token eviction at a fixed cache size. The trade is that you inherit a dispatch layer and a flag matrix, and you depend on the maintainers having wired each method correctly into that shared path. The README's own warning about narrower runner coverage is the honest version of that risk. For quantization specifically, KIVI, KVQuant and GEAR have independent upstream implementations, and routing them through --quant_method means the version you get is the one vendored here, not necessarily the latest upstream commit.

Maintenance, licence and what to check before you commit

The repository is MIT licensed, which permits commercial use and modification provided the licence and copyright notice are retained; that is a description of the licence text, not legal advice, and anyone shipping derived code should read the full MIT terms and check the licences of vendored method implementations separately, since those may differ from the top-level licence. On maintenance, the news entries run from 2024-06-10 (FlashAttention v2 and SDPA paths for PyramidKV, SnapKV, H2O and StreamingLLM), through 2024-06-25 (multi-GPU inference for large models including Llama-3-70B-Instruct), to the 2024-11-28 rename. No releases are listed, so expect to track the main branch rather than pin a version, and the dependency pins are narrow: transformers==4.44.2 will conflict with anything newer in an existing environment. Budget for that conflict as the first upgrade cost. The concrete next step is to run one method end to end at two budgets, 128 and 2048, on a single LongBench dataset via --datasets narrativeqa,qasper, confirm the numbers land where you expect, and only then widen the sweep. If that first pair of runs disagrees with the paper you are trying to reproduce, the problem is in the harness or the backend, not in the method.

Editorial conclusion

Adopt KVCache-Factory if you already run LongBench-style evaluations and want to compare budget-allocation methods such as PyramidKV and AdaKV against eviction baselines without rewriting a harness per paper. Do not adopt it as a production inference server; it is an evaluation playground, and HeadInfer plus the quantization paths add their own backend requirements. Before committing, verify three things in your own checkout: that your GPU supports flash-attn>=2.4.0.post1 or that sdpa runs your chosen method, that --method think is launched with --attn_implementation eager as the README requires, and that the runner argument choices actually list your target method, since the README warns some newer methods have narrower runner and model coverage.

Official sources

1. Issues
2. License: MIT
3. README
4. Zefan-Cai/KVCache-Factory on GitHub