buun-llama-cpp keeps the KV cache in FP16 until memory forces it down

A fork whose whole pitch is one number: the bits per value

The README puts the warning first. This is a highly experimental fork of llama.cpp, to be used at your own discretion, and the project description calls it an experimental llama.cpp fork for inference research and development.

The idea is variable bit rate, a KV cache codec system. The question the README asks is why pay 3-bit or 4-bit quality for a context length you only sometimes reach, and the answer is that VBR quantizes the KV cache dynamically as your session grows, giving the highest quality cache possible at any given depth.

The mechanism is to quantize the cache layer by layer using the best KV codecs, from the least sensitive layers to the most sensitive, and only as far down the ladder as the VRAM and context actually require. The cache starts at FP16 and stays there until there is real budget pressure. Then it degrades one layer and side tensor at a time, first to an aggregate 15.75 bits per value, then 15.51, and so on down the ladder, which the README writes as f16, then turbo8, turbo4, turbo3_tcq, turbo2_tcq and turbo1_tcq, following a per model price order measured on KLD panels.

The granularity is the interesting part. For Qwen35 with 16 attention layers there are 160 distinct steps from FP16 down to turbo1_tcq, and for Gemma4-31B with 60 layers there are 600. The README calls that the finest grained quality control available at every moment of a session, and the reason is stated plainly: because VBR always draws from the best codecs on the ladder, you never have to track KV formats again, since new codecs and research roll straight into VBR.

The audience is people doing inference work on a specific GPU with a specific context problem, which is a much narrower group than llama.cpp's users in general.

Specify two of the three things, and let the cache solve the third

The README frames the configuration as a small optimisation problem, and the framing is the most useful thing in the document.

VBR balances three quantities: context length, KV VRAM and minimum quality. The instruction is to specify at most two and let VBR solve the third, and the table gives the combinations. If you know how many tokens you need and how much KV VRAM you can spare, you use the context and VRAM flags and VBR determines the highest quality terminal layer mixture that fits. If you know the token count and the lowest quality you will accept, VBR determines the safe KV VRAM budget available on the machine. If you know the VRAM and the quality floor, VBR determines the largest fillable context.

The note under the table is the warning. Setting all three usually just over constrains the same calculation, and the advice is to pick the two requirements you actually know so that VBR can optimize the remaining dimension instead of receiving three potentially contradictory answers.

That is a well made argument for a feature that otherwise looks like a pile of flags. The three settings that are not fixed are context length, which you set with the context flag, the KV VRAM budget, and the quality floor, and the design says that two of them are enough.

The default behaviour also follows from it. VBR derives a KV VRAM budget from whatever is left after the weights and the compute, advertises the largest context that fits without going below the selected codec's default floor, capped at the model's training length, and degrades tiers on the fly as the context fills. The cache is still FP16 until memory pressure actually requires compression, so a short conversation never pays for a long one.

The flags, and the difference between the default and the opt in

On a dedicated GPU, the README says to just run the server, and VBR is the default cache:

llama-server -m model.gguf

Codec selection is automatic. Models whose complete KV geometry supports Turbo use a turbo4 quality floor, while BailingMoE3 and Ling fall back to the classic ladder. For maximum context you have to opt in explicitly with the VBR context flag, which opens the selected codec's complete ladder unless a floor is also set:

llama-server -m model.gguf -ct vbr

Running with the verbose flag shows the degrade steps as they fire.

The settings table is where the detail is. The VBR context flag, also spelled with the key or value variants, says that VBR is already enabled by default, that explicitly selecting it opens the full ladder when no floor is supplied, that explicitly pinning a side holds it at fixed bits and never degrades it, and that using a concrete type such as f16 opts out of VBR entirely. The context flag caps the context at N tokens, and the README explains the consequence in terms of what you get: VBR then spends your whole VRAM budget running that window at the highest quality it can, rather than advertising the maximum capacity of the floor tier.

The VRAM flag takes an explicit budget with a default of auto, which is whatever is left after weights and compute. The codec flag chooses the representation ladder, with auto preferring Turbo when every KV layer supports the complete ladder. The entry flag sets the tier the session starts at, with the default f16 preserving maximum quality and a lower tier explicitly trading quality for lower KV bandwidth and memory from the first token. And the floor flag sets a literal aggregate bits per value floor, with the implicit default at 4.125 and an explicit opt in at 1.25, and degradation stops at the last step still at or above that floor.

The requirements are strict, and dynamic mode switches features off

The requirements paragraph is short, and two of its sentences will decide whether the fork works on your machine at all.

A CUDA or ROCm backend is required, because turbo typed KV needs the TurboQuant interface, and layers whose KV lands on the CPU fall back to q8_0. Flash attention is required and force enabled. Dynamic mode uses unified KV, which is forced automatically when the parallel slot count is above one.

The third sentence is the cost. Context shift and self extend, and slot and session save and restore, are disabled in dynamic mode, with the reason given: they would snapshot tier typed KV that cannot be restored across a degrade. Tier aware save and restore is planned. Context checkpoints stay enabled on hybrid models, and generation stops cleanly when the context fills.

That is the trade in the whole feature. A cache that changes its own precision as the session grows cannot be snapshotted and restored, because a snapshot taken at one tier cannot be restored into another. If your workflow depends on saving a session and resuming it later, the classic ladder is the one to use, and the README says so in the section on the classic path, where the ladder keeps the model's native KV width and uses ordinary Q8_0 and Q4_0 codecs, without allocating Turbo rotations or interpreting Turbo model price tables.

The classic path also has a documented limitation of its own. Portable projected prompt artifacts are currently Turbo only, so classic mode retains live KV but cold prefills when a live prefix is no longer available, rather than restoring that prefix from the host cache. Models without a baked price order use a generic cross model order.

TCQ constrains the quantisation indices to a trellis

The second research contribution in the fork is TCQ, the trellis coded KV cache, and it is a different idea from VBR.

Standard KV cache quantisation treats each value independently. TCQ constrains the quantisation indices to follow a 512 state trellis, which gives a much larger effective codebook at the same bit rate, with a fast Walsh Hadamard transform rotation and context adaptive norm scaling on top.

The numbers in the README are the argument. At the 3-bit setting the trellis cuts median KL divergence by about 40 percent against scalar quantisation while using slightly fewer bits, 3.25 against 3.50 bits per value, and it lands perplexity on par with an f16 KV cache.

That combination is what makes the idea interesting for inference research. If you can spend 3.25 bits per value and keep the perplexity of a 16-bit cache, then the memory ceiling on long contexts moves, and the code that the whole VBR ladder is built on is TCQ. The turbo3_tcq, turbo2_tcq and turbo1_tcq entries in the ladder are this codec at different bit rates, which is why the lowest rungs of the ladder are named for it.

The README links a paper on the work, and the repository holds a codebooks/ directory, which is where a trellis codebook would live. The rest of the tree is a full llama.cpp checkout with the fork's additions layered on top, including a codebooks directory, a conversion/ directory, a scripts/ directory, a ci/ directory, a pocs/ directory and a skills/ directory, plus the usual CMakeLists.txt, CMakePresets.json, Makefile, CODEOWNERS, CONTRIBUTING.md, SECURITY.md and CLAUDE.md at the root.

Loading safetensors directly, including EXL3 quants

The fork also removes a step that most people hit first: converting a model before they can run it.

It loads supported .safetensors model directories directly, including EXL3 quants, with no GGUF conversion required, and points to the safetensors guide in the docs directory for the supported quantization types and their limitations. Support depends on the model architecture and the backend, which is the honest caveat in a sentence.

For a researcher this is the feature that decides whether the fork is worth trying at all. The conversion scripts are still in the tree, convert_hf_to_gguf.py, convert_hf_to_gguf_update.py, convert_llama_ggml_to_gguf.py and convert_lora_to_gguf.py, and the pyproject.toml exposes them as console entry points such as llama-convert-hf-to-gguf and llama-convert-lora-to-gguf, so the classic path is still fully supported. But if your model is already in safetensors, the fork will read it as it is, and the EXL3 quants are the case where the old path would have needed a conversion step you did not have a tool for.

The Python side of the project is unchanged in shape. The pyproject.toml names the package llama-cpp-scripts, requires Python from 3.10 up to but not including 3.15, and depends on numpy, sentencepiece, transformers pinned at 4.57.6, protobuf, torch and the gguf package from the local gguf-py directory. The requirements.txt is a set of references to per script requirement files, with a comment explaining that it includes all dependencies for all top level Python scripts and that packages should not be added there directly, because versions must stay compatible across all of the scripts.

The build moved to CMake, and the Makefile says so out loud

One small file in the root is worth quoting, because it is an example of a build system telling you the truth instead of failing obscurely. The Makefile is a stub whose only job is to stop you:

$(error Build system changed:$(newline)\
The Makefile build has been replaced by CMake.$(newline)$(newline)\
For build instructions see:$(newline)\
https://github.com/ggml-org/llama.cpp/blob/master/docs/build.md$(newline)${newline})

The file raises an error that says the Makefile build has been replaced by CMake, and points at the upstream build documentation. That is a small courtesy that saves a lot of confusion, since the fork inherits a tree where the old build path once existed.

The rest of the repository is a full llama.cpp checkout with the research additions. There is a CMakeLists.txt and CMakePresets.json at the root, a cmake/ directory, a flake.nix for Nix builds, a .pre-commit-config.yaml, a .clang-format and a .clang-tidy for the C++ style, a mypy.ini and a pyrightconfig.json for the Python, an SD-080-benchmark-notes.txt at the root, and directories for benches/, examples/, tools/, tests/, models/, grammars/ and media/.

The examples directory is the clearest index of what the project supports, with directories for batched inference, embedding, diffusion, eval callbacks, gguf handling, idle mode, lookup decoding, model conversion, parallel decoding, the Swift and SwiftUI and Vim bindings, an Android example, a passkey example and a deprecation warning example.

On the state of the project itself: it is MIT licensed, the repository is not archived, the last push was on 2026-10-16, and there are no GitHub releases. For a fork whose purpose is research, that is consistent with the way such work is published, and it also means there is no versioned artefact to point at, only a branch.

Editorial conclusion

Adopt buun-llama-cpp if you are doing inference research, or if you run long contexts on a single GPU and want the largest context that fits without pinning a fixed low precision cache, and if you can tolerate an experimental fork with no tagged releases. Do not adopt it for production inference on an unmodified llama.cpp schedule, because the README states plainly that this is a highly experimental fork to be used at your own discretion, and the fork disables several llama.cpp features in dynamic mode. Before you build, confirm you have a CUDA or ROCm backend, since turbo typed KV needs the TurboQuant interface, and read the requirements list, because flash attention is required and force enabled, and context shift and slot save and restore are switched off in dynamic mode.

Frequently asked questions

Official sources

1. Issues
2. License: MIT
3. README
4. spiritbuun/buun-llama-cpp on GitHub