TriAttention: trigonometric KV cache compression for long reasoning

The KV cache, not the weights, is what runs out of memory

Long reasoning models generate very long chains of thought, and every token in that chain adds key and value vectors to the attention cache. The result is a serving bottleneck that has little to do with how large the model is. The weights sit in memory once; the cache grows with every token of context and keeps growing.

TriAttention is an attempt to shrink that cache without changing what the model outputs. The README states the headline figures plainly: a 10.7x KV memory reduction with trigonometric frequency domain compression, a 2.5x throughput improvement on AIME25 long reasoning while matching Full Attention accuracy at 40.8 against 40.8, and no accuracy loss. Those are the authors' reported numbers on their chosen benchmark, not independent measurements, and the reproduction guide is linked separately for anyone who wants to rerun the experiments.

The practical framing that makes it interesting is local deployment. The README describes OpenClaw compatible deployment on a 24GB RTX 4090. A compressed cache changes what fits on one consumer card, and that is the scenario the project is aimed at rather than a multi GPU serving cluster.

Pre-RoPE centres decide which keys are worth keeping

The mechanism is described in a short paragraph, and it is worth reading closely because it explains why the method avoids the cost of existing attention based approaches.

Pre-RoPE query and key vectors in long reasoning models concentrate around fixed centres. Those centres determine distance preferences via a trigonometric series. TriAttention scores keys using these centres and norms, instead of requiring representative query selection.

So the compression decision is driven by the model's own internal geometry rather than by a sample of queries that would have to be gathered and scored. The README frames this as the reason accurate compression is possible without the overhead of existing attention based methods.

Two consequences follow from the description. First, the centres are a property of a trained model, which is why calibration exists as a separate step and why the runtime path needs precomputed statistics. Second, since keys are scored by norms and centres, the selection is deterministic per model rather than dependent on the incoming request, which is what makes it workable as a server side cache policy.

Prefix caching has to be turned off for the compressed cache to be correct

This is the sharpest constraint in the README and the one most likely to cause a silent wrong answer rather than a crash. When starting a vLLM server with TriAttention, the README instructs you to pass --enable-prefix-caching false, and explains why: prefix caching is incompatible with KV compression currently, and it must be disabled to avoid incorrect cache hits on compressed entries.

That is a real loss. Prefix caching is how a chat server reuses the KV of a shared system prompt across requests, and turning it off costs throughput on exactly the multi turn, long context workloads that TriAttention is trying to speed up. The mitigation the README offers is a larger KV budget instead, so that history is kept rather than evicted, but kept history is not the same as reused computation.

The second server flag exists for a related reason. --max-num-batched-tokens 1024 limits the prefill chunk size, because large chunks can overshoot the KV budget in a single step before compression has a chance to trigger, leading to an out of memory error. In other words the budget is not a hard ceiling enforced continuously, it is a soft target that a large prefill can blow past in one go.

The KV budget is a runtime knob that chat sessions should raise

Two environment variables control the runtime behaviour, and the README is explicit about both.

The first points at the precomputed frequency statistics and is marked as required.

export TRIATTN_RUNTIME_SPARSE_STATS_PATH=triattention/vllm/stats/qwen3_32b_int4_stats.pt
export TRIATTN_RUNTIME_KV_BUDGET=12000

The second sets the budget, with a documented default of 2048 and a recommendation of 12000 for multi turn chat, since chat sessions accumulate context across many turns and a larger budget keeps more history available and avoids aggressive eviction.

That default of 2048 tokens is small for a long reasoning model, and the reason for the difference is worth noticing. Offline benchmarks replay a fixed prompt and measure throughput. Interactive chat is long running, and the README warns that conversations differ from offline benchmarks because prefill chunks can trigger compression at unexpected points. The same compression policy that is fine for a benchmark run is a poor fit for a conversation that keeps growing.

The example server invocation combines these with the flags above.

vllm serve <model_path> \
    --dtype bfloat16 \
    --max-model-len 32768 \
    --enforce-eager \
    --trust-remote-code \
    --enable-prefix-caching false \
    --max-num-batched-tokens 1024

The vLLM plugin entry point is the whole integration surface

There is no separate fork of vLLM to maintain. The package registers itself through the vllm.general_plugins entry point, which the setup.py declares as a single mapping.

entry_points={
    "vllm.general_plugins": [
        "triattention = triattention.vllm.plugin:register_triattention_backend",
    ],
}

That is a small and clean surface, and it explains why the same code can also reach SGLang, TensorRT-LLM and MLX without the core library changing shape.

Installation is a source checkout, since the package is not presented as a published wheel in the README.

git clone https://github.com/WeianMao/triattention.git
cd triattention
pip install -e .
pip install flash-attn --no-build-isolation  # recommended (takes 105m in DGX Spark / GB10)

The build cost of flash-attn is called out in the README itself, at 105 minutes on DGX Spark with GB10. The package metadata requires Python 3.10 or later and pulls in torch, triton, transformers, datasets, numpy and scipy among the core dependencies, with an optional flash extra for flash-attn and an eval extra for the benchmarking stack.

Calibration statistics are tied to a specific model

The path in the server example names a specific file, qwen3_32b_int4_stats.pt, and the repository ships statistics under triattention/vllm/stats. A calibration guide is documented separately for generating custom Q and K statistics, which is the intended route for a model that has no shipped stats file.

This is a real operational constraint rather than a footnote. The compression decision depends on where a model's pre-RoPE vectors concentrate, so the statistics are a property of the weights. Serve a different model, or a different quantisation of the same model, and the shipped stats no longer describe what the cache contains.

The supported model table lists three entries with their HuggingFace identifiers and a Verified status: Qwen3-8B, DeepSeek-R1-Distill-Llama-8B and DeepSeek-R1-Distill-Qwen-7B. Anything outside that list means generating your own statistics first, or accepting that the compression decisions are being made with numbers that do not match the weights in memory.

The calibration step is exactly where a mistake is costly, because a miscalibrated stats file does not raise an error. It just selects the wrong keys to keep, and the result is a model that is quietly worse while still running.

The same compression idea now runs on four different backends

The project has spread beyond its original vLLM path, and the news entries in the README are a useful map of the ecosystem.

SGLang support was added in April 2026, so TriAttention runs on SGLang as well as vLLM, with a dedicated integration document. A community port for llama.cpp targeting C and ggml with HIP and ROCm brings it to AMD GPUs, and the README reports around 6.8x KV reduction when composed with TurboQuant. Experimental MLX and TurboQuant support targets Apple Silicon machines from M1 to M4.

In August 2026 TriAttention was integrated into NVIDIA TensorRT-LLM as a KV cache compression method, with a guide in that project's own examples directory. That is the strongest signal in the list, because it means the technique now sits inside a mainstream serving stack rather than only in a research repository.

There is also a video generation path. The repository includes a longlive directory and a README for KV cache compression for long video generation with LongLive, and the news log records support for AR video generation in April 2026.

The repository itself is small and readable, with a triattention package directory, scripts, tests, docs, data and longlive at the top level. Only one release tag is listed, v0.2.0 on 2026-04-22, whose title names SGLang, LongLive and multi backend and multi hardware support.

Editorial conclusion

TriAttention suits serving teams running long reasoning models such as Qwen3-8B or the DeepSeek R1 distills on a single 24GB card, and the vLLM plugin entry point means it can be enabled without patching the inference server. It does not suit setups that rely on prefix caching, since the README states the two are incompatible, and it does not suit models without matching calibration statistics, because TRIATTN_RUNTIME_SPARSE_STATS_PATH must point at a stats file produced for your model. Before you enable it in production, check the stats path against your actual checkpoint, keep --enable-prefix-caching false and --max-num-batched-tokens 1024 as the README requires, and confirm the last push to the repository was on 2026-08-04, which is the most recent activity recorded there.

Frequently asked questions

Official sources

1. Issues
2. License: Apache-2.0
3. README
4. Releases
5. WeianMao/triattention on GitHub