# RotorQuant, KV cache quantization that trades a full transform for 2D and 4D rotations

> RotorQuant is a research project on compressing the attention key and value cache of an LLM by rotating cache vectors before scalar quantization, using small independent blocks rather than one large transform. The Python package here is the research implementation, while the production path is a fork of llama.cpp, and the reported comparison against TurboQuant comes from the project's own benchmarks.

**scrya-com/rotorquant** — KV cache compression via block-diagonal rotation. Beats TurboQuant: better PPL (6.91 vs 7.07), 28% faster decode, 5.3x faster prefill, 44x fewer params. Drop-in llama.cpp integration.

- Repository: https://github.com/scrya-com/rotorquant
- Website: https://www.scrya.com/rotorquant
- Stars: 1,057 · Forks: 87
- Language: Python
- License: not declared
- Published: 2026-10-19 · Updated: 2026-10-19 · Language: en
- Canonical page: https://hysenlabs.com/projects/scrya-com-rotorquant

## The idea, stated as a substitution

The project frames its contribution as a replacement, and the quote it opens with says it plainly: replace the d by d random orthogonal matrix with Clifford rotors, exploiting algebraic sparsity. The context is TurboQuant, which the project compares itself against. TurboQuant applies a d by d Walsh-Hadamard Transform, a butterfly network with log2(d) stages across all 128 dimensions, before quantizing the cache. PlanarQuant and IsoQuant instead apply independent 2D and 4D rotations per pair or quartet, which the project puts at O(d) instead of O(d log d), fully parallelizable, with no inter-element dependencies. The payoff is that the rotation step is cheap enough not to matter, and the claim is that the block structure preserves the directional structure of the cache vectors better than a global scrambling does. The reported PPL results are offered as evidence for that, with the simpler rotations scoring better rather than worse.

```
Replace the d×d random orthogonal matrix with Clifford rotors... exploiting algebraic sparsity
```

## The reported comparison, and who measured it

The headline table is for Llama 3.1 8B Instruct with symmetric 3-bit K and V compression on an RTX 5090, with the baseline at 140 decode tokens per second and a perplexity of 6.63 on wiki-2. The iso3 configuration is given at 118 decode and 3,397 prefill tokens per second with a perplexity of 6.91, and planar3 at 119 decode and 3,822 prefill with a perplexity of 7.05. The turbo3 row is 93 decode, 722 prefill and 7.07 perplexity, and the comparison summary states the PPL as 6.91 against 7.07, the decode as 119 against 93, the prefill as 3,822 against 722, and the parameter count as 128 against 16,384. These are the project's own measurements on a single model and a single GPU, and the description of the numbers as beating a competitor on every axis is the project's claim rather than an independently verified result.

```
| f16 / f16 | 140 | 6,156 | 6.63 | baseline | 1x |
| **iso3 / iso3** | **118** | **3,397** | **6.91** | **+4.2%** | **10.3x** |
| turbo3 / turbo3 | 93 | 722 | 7.07 | +6.6% | 10.3x |
```

## Three implementations, trading richness for speed

The project documents how it arrived at the simpler schemes, and the table is the clearest summary of the design space. The original paper proposed Clifford algebra Cl(3,0) rotors, a rotor sandwich product with only 4 non-zero multivector components, and the insight behind it is that you do not need a full-rank d by d transform to decorrelate KV cache vectors, because small orthogonal blocks suffice, since real attention vectors live on low-rank manifolds. From that came PlanarQuant with 2D Givens rotations and IsoQuant with 4D quaternions, developed by another author and built on the block-diagonal rotation idea. The cost column makes the trade explicit: the Cl(3,0) rotor sandwich needs roughly 2,400 floating point multiply-adds at d of 128 with 372 parameters, the 4D quaternion and the 2D Givens each need 512 and 256 with 128 parameters, and TurboQuant needs 16,384 of each. The original rotor implementation is marked research in Triton, while the two simpler ones are marked production in llama.cpp.

```
| **RotorQuant** | Cl(3,0) rotor sandwich | 3 | ~2,400 | 372 | Research (Triton) |
| **[IsoQuant](https://github.com/ParaMind2025/isoquant)** | Quaternion 4D | 4 | 512 | 128 | **Production (llama.cpp)** |
| TurboQuant | WHT butterfly | 128 | 16,384 | 16,384 | Production (llama.cpp) |
```

## The llama.cpp path, which is the one to use

The quick start marks the llama.cpp route as recommended, for being fastest, and it is a fork with a feature branch rather than a pip install. You clone the fork, check out the feature/planarquant-kv-cache branch, and build with CUDA or with Metal for Apple Silicon, using the same cmake invocation you would use for a normal llama.cpp build. The server is then started with a cache type flag, and the documentation gives both the symmetric 3-bit case and the K-only case where the value cache stays in f16. The cache types available are planar3, iso3, planar4 and iso4 from this work, plus turbo3 and turbo4 from the Walsh-Hadamard implementation. The benchmark and perplexity commands are also given, with the latter needing a dataset step that fetches wikitext through the datasets library before running llama-perplexity against it.

```bash
# Symmetric 3-bit (best quality per bit)
./build/bin/llama-server -m model.gguf --jinja -ngl 99 \
    --cache-type-k iso3 --cache-type-v iso3 --host 0.0.0.0 --port 8080

# K-only (zero PPL loss, 5x compression)
./build/bin/llama-server -m model.gguf --jinja -ngl 99 \
    --cache-type-k planar3 --cache-type-v f16 --host 0.0.0.0 --port 8080
```

## The Python side is the research track

The Python package in this repository is the research implementation, and the documentation is explicit that the recommended path is llama.cpp. The Python install is a pip editable install plus Triton, and the API mirrors the research paper rather than the server. The class names say which is which: IsoQuantMSE is the 4-bit configuration and PlanarQuantMSE is the 3-bit one, each constructed with a dimension of 128, a bit count and a device, and each returning a dequantized tensor and a set of indices. The MSE in the name refers to the fact that the rotation is chosen to minimize mean squared error, which is the criterion the paper uses to pick the rotation. The requirements file pins the stack this research path expects, with torch, scipy, transformers, accelerate and bitsandbytes, and the setup script can optionally build CUDA kernels, with a flag to force the PTX path on Blackwell-class architectures.

```python
from turboquant import IsoQuantMSE, PlanarQuantMSE

# IsoQuant: best 4-bit quality (PPL 9.03)
iq = IsoQuantMSE(d=128, bits=4, mode='fast', device='cuda')
x_hat, indices = iq(x)

# PlanarQuant: best 3-bit quality (PPL 10.12)
pq = PlanarQuantMSE(d=128, bits=3, device='cuda')
x_hat, indices = pq(x)
```

## How the pipeline is put together

The mechanism section explains that rotation decorrelates KV cache vectors before scalar quantization, and the first steps are normalization, where the norms are stored separately, and rotation. The rest of that section is truncated in what is visible, but the supporting detail elsewhere fills in the shape of the design. The deferred K-cache is described as f16 during prefill, which eliminates rotation overhead entirely during prompt processing, and the commit history in the llama.cpp fork shows the work: CUDA flash attention dequantize kernels, CUDA set_rows kernels, asymmetric FA kernels, a fix dispatching on static constants and a V=f16 check, a fix adding an inverse rotation to the V dequant path, an F16 plus Q8_0 VEC template for deferred prefill, a fix for the ggml context size for double buffering, and the double-buffered deferred quantization with CUDA conversion kernels. The commits are dated at the end of March and the start of April 2026, which is when the work happened.

```
1. **Normalize** → store norms separately
2. **Rotate**
```

## The repository, and what to be careful about

The repository holds the paper, the Python package in a turboquant/ directory, a set of scripts for validation and benchmarking, and a Claude configuration file. The scripts are worth naming because they indicate the project's own approach to checking its claims: benchmark_vram.py, benchmark_vs_reference.py, validate.py, a test_turboquant.py and a tests/ directory, plus a grok_prompt.md, which is presumably a prompt used to generate or review something with a language model. There is a combined distribution file and a setup.py whose docstring explains the two install modes, a PyTorch-only build and one that also builds the CUDA kernels, with the build_ext invocation for an in-place build. The main branch here is the research repository and the production code lives in the fork. The last push to this repository was on 2026-04-23 and it is not archived, there are no GitHub releases, and the licence is not one that GitHub can classify, so the terms are the thing to check before you vendor any of it.

```
benchmark_vram.py
benchmark_vs_reference.py
validate.py
test_turboquant.py
grok_prompt.md
```

## Conclusion

Adopt RotorQuant if you run a local llama.cpp server and need the key and value cache to fit in less memory without giving up much quality, and skip it if you are not prepared to maintain a fork of llama.cpp, because the production path is a branch of somebody else's server rather than a drop-in package. Verify three things before you rely on the numbers: that the reported perplexity and throughput figures were measured by the project on one GPU and one model, since they are the project's own benchmarks rather than an independent comparison; that the cache type you pick exists in your build, since the supported types are planar3, iso3, planar4 and iso4 alongside turbo3 and turbo4; and that the deferred K-cache behaviour during prefill matches what your serving pattern needs, because the cache is held in f16 while the prompt is processed.

## FAQ

### What is RotorQuant and what does it change in an LLM server?

It is a KV cache compression method that rotates the key and value cache vectors before scalar quantization, using independent 2D Givens or 4D quaternion blocks per pair or quartet instead of one large Walsh-Hadamard transform across all dimensions, so the rotation costs O(d) rather than O(d log d) and has no inter-element dependencies.

### How do I use RotorQuant with llama.cpp?

Clone the provided fork, check out the feature/planarquant-kv-cache branch, build with CUDA or Metal, and pass a cache type flag to llama-server, for example `--cache-type-k iso3 --cache-type-v iso3` for symmetric 3-bit or `--cache-type-k planar3 --cache-type-v f16` for K-only compression.

### What are the differences between RotorQuant, IsoQuant, PlanarQuant and TurboQuant?

RotorQuant uses Cl(3,0) rotor sandwiches with 4 non-zero multivector components, about 2,400 multiply-adds and 372 parameters at d of 128, and is research in Triton. IsoQuant uses 4D quaternions and PlanarQuant 2D Givens, each with 128 parameters, and both are production in llama.cpp. TurboQuant uses a 128-wide Walsh-Hadamard butterfly with 16,384 multiply-adds and 16,384 parameters.

### What does the deferred K-cache do during prefill?

The K cache is kept in f16 while the prompt is being processed, which the project says eliminates the rotation overhead entirely during prompt processing, with a double-buffered deferred quantization using CUDA conversion kernels to move between representations.

## Sources

- [Issues](https://github.com/scrya-com/rotorquant/issues)
- [Project website](https://www.scrya.com/rotorquant)
- [README](https://github.com/scrya-com/rotorquant/blob/main/README.md)
- [scrya-com/rotorquant on GitHub](https://github.com/scrya-com/rotorquant)

---

Hysen Labs editorial analysis, written from the project's own repository and release notes. Cite the canonical page: https://hysenlabs.com/projects/scrya-com-rotorquant