TurboQuant: 3-bit KV Cache Quantization with vLLM Integration

The problem TurboQuant targets: KV pool size, not throughput

The README frames the motivation through capacity rather than speed. On a single RTX 5090 running Qwen3.5-27B-AWQ, the documented baseline is 457,072 max tokens and the TurboQuant configuration is 914,144, a 2.0x increase attributed to 30.0 GB of KV cache freed across four GPUs. On the 8x RTX 3090 MoE setup, baseline capacity is 1,411,680 tokens and with TurboQuant it is 2,043,808, a 1.45x multiplier. The README also notes that the freed VRAM alternatively supports three additional concurrent 131k-context requests.

That is the actual product: more concurrent long-context requests per GPU, not faster tokens. The throughput numbers in the same tables are small and the project says so itself. Prefill on the RTX 5090 goes from 1,804 to 1,907 tok/s and decode from 1.264 to 1.303 tok/s, and the adversarial audit labels the 30k claim as "within noise" with N=1 and total wall time actually slower for TurboQuant. Anyone shopping for a speedup is looking at the wrong column.

The intended user is someone serving a dense transformer on vLLM who is running out of KV cache before running out of compute. The README's own framing of the hybrid case makes the boundary explicit: on the 8x RTX 3090 MoE model, only 10 of 40 layers are full attention, so TurboQuant touches 40% of the KV and the measured savings are 30.9%. On a pure dense transformer the README projects 77%, which it restates as 4.4x compression.

Rotation, Lloyd-Max codebooks, and the QJL residual bit

The pipeline is a five-step chain described in the README. First, a random orthogonal rotation spreads information across dimensions, which is what makes the subsequent scalar quantization well behaved. Second, Lloyd-Max optimal scalar quantization at b-1 bits is applied to the Beta-distributed rotated values. Third, a QJL projection encodes the residual sign, one bit per dimension. Fourth, values are quantized separately with group quantization at 2-bit or 4-bit using per-group scales and zeros. Fifth, the result is bit-packed, four values per byte at 2-bit and two per byte at 4-bit.

The estimator is stated to be unbiased: the expected value of the estimated inner product equals the true inner product. The paper validation table backs this with a relative bias under 0.1% for Theorem 2, and reports MSE distortion within Theorem 1's bounds for unit-norm vectors, with distortion scaling at 0.70x, 0.82x, and 0.97x of the 1/4^b bound for 2, 3, and 4 bits.

The quality split matters more than the compression ratio. Measured at head_dim=256, key compression is reported at cosine similarity 1.000000 for both 3-bit and 4-bit, while value quantization at 2-bit gives 0.940 and at 4-bit gives 0.997. The README states plainly that value quantization dominates the degradation and recommends 4-bit values for quality-sensitive use. If you are choosing a configuration, that recommendation is the one concrete tuning decision the material supports.

Getting it running: vLLM 0.18.0 and the quantization flags

The README does not include an install command block, so the exact pip or source invocation cannot be confirmed from the supplied material. What it does document is the environment the benchmarks ran under: vLLM 0.18.0 on both the RTX 5090 and the 8x RTX 3090 setups, with gpu_memory_utilization=0.90 on the single-GPU dense run and 0.92 on the eight-GPU MoE run. The project describes itself as Triton kernels plus vLLM integration, so the integration path is a vLLM patch rather than a standalone server.

The configuration surface visible in the material is the bit split. The headline configuration is 3-bit keys and 2-bit values, written as "3b key / 2b val" in the benchmark tables, and the quality table also reports 4-bit keys and 4-bit values as alternatives. The README's recommendation for quality-sensitive workloads is 4-bit values. There is no config key list in the supplied material, so I cannot name the parameter that selects the split; that is a gap you will have to close by reading the repository.

One more operational detail worth noting: the 8x RTX 3090 run used an AMD EPYC 7443P with 504GB of RAM and PCIe interconnect with no NVLink, and the README reports no CPU offloading and no KV offloading, with VRAM essentially flat across context lengths. The GPU utilization table shows 0% idle at both 1,000 and 131,000 tokens and 57% peak at 32,000. That pattern suggests the workload is capacity-bound rather than continuously compute-bound at long context.

The adversarial audit is the most valuable file in the repository

Most projects bury caveats. This one ships audit_claims.py and a table that retracts its own headline figures. The "5.1x compression" claim is called misleading because it does not count the Pi and S matrices or the ring buffer, with an honest figure of about 4.6x at 4k tokens and about 5x at 32k and above. The needle-in-haystack result is called true but trivial, on the grounds that a query equal to the key is too easy a test. The recall bar is called low: recall@8 at 3-bit is 0.55 against a 0.40 threshold, but recall@1 at 3-bit is only 38%.

The sharpest entry concerns hybrid decode. The README says the memory saving is real but the compute saving is not, because the implementation dequantizes all history to float32 on every decode step. That is a structural cost, not a tuning issue, and it explains why the decode throughput gains in the benchmark tables are in the low single digits. The audit also flags the 30k speed claim as within noise and the 200k context claim as unverified, since it did not crash but output quality was never checked.

Two entries were corrected in TurboQuant's favor. The distortion scaling claim was initially audited as wrong and then confirmed once vectors were normalized to unit norm. The 2x context claim on a dense model was confirmed at 30 GB freed on Qwen3.5-27B with four RTX 3090s. A project that publishes both directions of correction is easier to trust than one that only publishes wins, though it also means the README's top-line numbers should be read as upper bounds.

Where TurboQuant is the wrong tool: hybrid attention and decode-bound serving

The 30.9% savings figure on the 8x RTX 3090 setup is not a tuning artifact. It is arithmetic. The model has 10 full-attention layers and 30 linear-attention layers, TurboQuant compresses only the full-attention layers, and those are 40% of the KV. The README states that the 30 linear-attention layers, 60% of KV, are not compressible by this method. If your model is a hybrid with a small full-attention fraction, the ceiling on your savings is that fraction, and no bit-width choice changes it.

Decode-bound workloads are the second mismatch. The audit's note that hybrid decode dequantizes all history to float32 each step means the compression buys storage and costs compute. The 8x RTX 3090 table shows decode falling from 133.0 tok/s at 16,000 tokens to 98.3 at 131,000, which the README attributes to KV readback cost in the full-attention layers. If your service is already latency-sensitive at long context, adding a dequantization pass over the history is the wrong direction.

Quality-sensitive tasks are the third. With 3-bit keys and 2-bit values, combined cosine similarity is 0.940, and the README attributes that to value quantization rather than key quantization. Keys at 3-bit measure 1.000000. So the degradation is concentrated in one half of the pair, and the documented remedy is 4-bit values at 0.997. If you cannot afford that bit width, the benchmark evidence for retrieval quality rests on tests the project itself describes as too easy.

The alternative: leave the KV cache in bf16 and buy concurrency elsewhere

The comparison the README sets up is against bf16 KV cache, which is the baseline in every table. The difference in approach is straightforward. bf16 keeps every key and value at full precision, so nothing is dequantized during decode and no layer-type analysis is needed. TurboQuant trades precision for capacity, and the trade is only favorable when the KV pool, not the compute, is what limits your request count.

Concretely, on the 8x RTX 3090 MoE configuration, the baseline KV pool at 131k context is 755.7 MB per GPU and the TurboQuant pool is 521.9 MB per GPU, a saving of 233.8 MB per GPU. The total reserved VRAM is 22,610 MB per GPU against a 24,576 MB card, and the README breaks that into roughly 6,750 MB of weights, 9,035 MB of KV pool, and about 6,825 MB of CUDA overhead and graphs. A 233.8 MB saving against a 9,035 MB pool is a real but modest change in headroom, and it is what produces the 1.45x capacity multiplier.

The honest alternative framing is that bf16 plus a smaller model, or bf16 plus more GPUs, achieves the same capacity without the dequantization cost and without the layer-composition constraint. TurboQuant makes sense when neither of those is available: you have a dense transformer, you have fixed hardware, and you need the extra context or the extra concurrent requests on that exact machine. The README's 30 GB freed figure on the four-GPU dense setup is the case where the argument is strongest.

Licence, maintenance surface, and what to verify before adopting

TurboQuant is GPL-3.0. That is a copyleft licence, and it matters here because the project is described as a vLLM integration, which means the delivered artifact is likely a patch or a modified build rather than a library you import at arm's length. I am not giving legal advice, but if you are linking this into a proprietary serving stack, the licence terms are the first thing to have someone qualified read, not the benchmark tables.

The maintenance surface is larger than a Python package. It includes Triton kernels, vLLM integration pinned to version 0.18.0 in the benchmarks, and a paper implementation tracking ICLR 2026 work. There are no releases retrieved, the last push is 2026-09-03, and the homepage field is empty, so there is no published versioning or upgrade path in the material. Every vLLM upgrade is a potential integration break, and the compute cost of dequantizing history to float32 per decode step is baked into the design rather than deferred to a later optimization.

What to verify first, in order: read audit_claims.py and reproduce the compression ratio on your own model's layer mix, since the 4.4x figure applies to full-attention layers only and your hybrid fraction determines the ceiling. Then check the value bit width against your quality bar, because 2-bit values measure 0.940 cosine similarity and 4-bit values measure 0.997. Then measure decode latency at your target context length rather than trusting the single-digit throughput gains, which the project itself describes as within noise. The README's own audit is a better starting document than its benchmark tables.

Editorial conclusion

Adopt TurboQuant if you run a pure dense transformer on vLLM and your bottleneck is KV pool capacity rather than decode latency. Do not adopt it for hybrid linear-attention models where you expect the 4.4x figure to apply, because the README measures 30.9% savings there, not 77%. Before installing anything, read audit_claims.py and the adversarial audit table in the README, and verify the compression figure against your own layer mix.

Official sources

1. 0xSero/turboquant on GitHub
2. Issues
3. License: GPL-3.0
4. README