RedKnot: Head-Aware KV Reuse and SegPagedAttention for Long-Context Serving
Efficient Long-Context LLM Serving with Head-Aware KV Reuse and SegPagedAttention
At a glance
- What is it?
- RedKnot is a Python framework built on SGLang that splits attention heads into reusable and online groups, adds token-selective FFN and MoE execution, and reorganizes KV pages per head and segment. It ships one reproducible DeepSeek-V4-Flash path on TP8 and lists the rest as experimental.
- Who is it for?
- Adopt RedKnot if you serve a long-context model on the DeepSeek-V4-Flash TP8 profile and you are willing to run its Recomputed reference on the same checkpoint and input IDs to see whether the head policy holds for your traffic. Do not adopt it if you need a stable release, a non-NVIDIA or Ascend path today, or a guarantee about end-to-end throughput, since the project's own compute ledger excludes memory traffic, kernel-launch cost and TP communication.
- Can I use it commercially?
- Yes. Apache-2.0 is a permissive licence: you can use, modify and sell software built on it, as long as you keep its copyright and licence notices.
- Is it still maintained?
- Yes. The repository last received commits 2 days ago.
- What is it written in?
- Mainly Python, according to GitHub's language statistics.
Answers come from the project's GitHub data, last synced on September 15, 2026, and from our analysis. They are not legal advice.
DEEP OPEN-SOURCE ANALYSIS
The problem RedKnot targets: long-context prefill where most heads do not need the whole context
Long-context serving costs are dominated by work that repeats on every request. A standard online prefill processes the full prompt through every attention head and every FFN row, even when a large share of heads behave locally and only a minority actually retrieve across the full window. Prefix caching attacks a different problem: it only helps when two requests share an identical prefix. RedKnot's README frames its gains as measured against a full online Recomputed path on the same checkpoint and input IDs, explicitly not against a prefix-cache hit. That distinction matters, because it means the framework is targeting the case where cache reuse is unavailable and the prefill itself must get cheaper.
The intended user is an inference engineer running a long-context model on multi-GPU hardware who can afford to freeze a per-model policy offline. The README describes offline preparation of reusable local heads and an online path for global, retrieval and recovery heads, with the projected contributions merged back so the model's external interface does not change. That last constraint is the design's main selling point: no client-side changes, no new tokenizer, no altered output contract.
Head decomposition, sparse FFN and SegPagedAttention: how the three mechanisms fit together
The README describes three composable mechanisms rather than a single cache trick. The first is head decomposition and aggregation. Attention heads are classified by long-context behavior. Reusable local heads are prepared offline. Global, retrieval or recovery heads stay online. Their projected contributions are merged back into the model. The README states this abstraction maps to MLA, MHA, GQA and native sliding-window attention, with model-specific projection and RoPE handling. The DeepSeek-V4-Flash configuration gives a concrete instance: layers 0 to 2 and 40 to 42 are fully online, while layers 3 to 39 use 8 online global heads and 56 reusable local heads, with online RoPE relocation and projection merge.
The second mechanism is sparse FFN and MoE execution. Token-level importance decides which rows enter expensive FFN work, and adaptive expert Top-K assigns more experts only when the router distribution requires it. Dense boundary layers and protected query rows are kept on the critical path. The shipped configuration expresses this as checkpoint-island row selection plus plan-scoped adaptive expert Top-K, with cumulative router mass 0.50 and physical Top-K buckets of 3, 4, 5 and 6.
The third is SegPagedAttention, which organizes KV pages and visibility per head and per segment. Global, local and retrieval heads can then consume different context scopes without forcing one uniform cache layout. The three mechanisms operate at different granularities (head, token, expert) and the README presents them as independent, which is the reason a model-specific cache shortcut would not generalize the same way.
Getting the DeepSeek-V4-Flash path running: what the repository actually ships
The primary reproducible path is the DeepSeek-V4-Flash release, which the README says runs on a TP8 server and ships frozen inputs, head policy, sparse-MoE policy and execution manifests. The frozen release setting names the model as `deepseek-ai/DeepSeek-V4-Flash-0731`, the published run hardware as 8x NVIDIA H200 with 143,771 MiB per GPU, TP8, driver 570.148.08, and the runtime as CPython 3.11.13, PyTorch 2.9.1 with CUDA 12.8, and Triton 3.5.1. The kernel stack is listed as FlashMLA `1.0.0+9241ae3`, SGL Kernel 0.3.20 and FlashInfer 0.5.3. The README describes this as a one-command reproduction over frozen 64K, 128K, 256K and 440K LongBench-derived RAG suites, though the truncated README text does not show the command itself, so the exact entrypoint has to be read from the repository rather than from this excerpt.
The benchmark geometry is specified per suite. 64K targets 65,536 prompt tokens over 4 documents of 16,384 tokens, with a runtime static-memory fraction of 0.45. 128K targets 131,072 tokens over 4 documents of 32,768, fraction 0.40. 256K targets 262,144 tokens over 8 documents of 32,768, fraction 0.45. 440K targets 450,560 tokens over 8 documents of 56,320, fraction 0.29. Each suite holds 15 cases: 10 short-answer and 5 supplemental 30-token long-output cases. The TTFT protocol is a hot state with 3 unmeasured paired warmups followed by 10 measured Recomputed/RedKnot pairs per case, reporting p50 and p95 on the streaming first output token. If you cannot reproduce that protocol, your numbers are not comparable to the published ones.
The compute-ledger caveat is the most important line in the README
The performance section states targets of quality regression within 1 percentage point, a 2 to 5x hot-state TTFT speedup, and 70 to 90 percent arithmetic compute-ledger saving on qualified long-context profiles. It then says the achieved point depends on the model, context length, GPU topology and frozen policy, and that per-suite result JSON is the source of truth. That is an unusually direct admission that the headline range is a range, not a measured constant.
More consequential is the next paragraph. The compute ledger, by the project's own description, excludes memory traffic, kernel-launch cost, TP communication and all uncredited runtime components, and is therefore not a claim about total system energy or universal end-to-end throughput. Anyone reading the 70 to 90 percent figure as an end-to-end serving improvement is misreading it. On a TP8 deployment, tensor-parallel communication is a real cost that the ledger does not count, and kernel-launch overhead grows as execution becomes more selective and more fragmented. A framework that reduces arithmetic while adding per-head and per-segment dispatch has a plausible path to negative net gains at short context lengths, where the fixed overheads dominate. The README does not publish a crossover point, and the frozen suites start at 64K.
Support breadth versus reproducibility: one frozen path, several experimental ones
The repository's maturity is uneven in a way the README is fairly clear about. DeepSeek-V4-Flash is the packaged, frozen, one-command path. The July 2026 entry describes experimental adapters and RAG benchmarks for Mistral, Qwen3, Qwen3.5 MoE and Llama 3.3, covering native SWA, GQA/MHA head policies and sparse-FFN execution. The word experimental is doing real work there: there is no equivalent frozen release table for those models in the material provided. If your production model is Qwen3 or Llama 3.3, you are on the adapter path, not the qualified path.
The Ascend port has the same shape. The September 2026 note says the upstream SGLang Ascend baseline has landed with Atlas 800I A2 and A3 containers, `docker/npu.Dockerfile` and `docs/platforms/ascend/`, and that RedKnot's port status, known gaps and bring-up workflow are documented in `docs/ASCEND.md`. The same note says the port is work in progress and that Ascend numbers are preliminary until qualification profiles are co-published. The Future Work section adds that short-term the target is functional parity with the upstream SGLang NPU baseline in RedKnot's Recomputed reference path, meaning the RedKnot-specific mechanisms are not yet the thing being qualified there. Read `docs/ASCEND.md` before planning any NPU deployment, and treat the absence of a qualification profile as the answer to whether the gains apply.
Where RedKnot is the wrong tool, and what to use instead
RedKnot is the wrong tool when your workload is short-context, when your requests share long identical prefixes, or when you cannot freeze a per-model policy. In the first case the selective execution overhead has nothing to amortize against. In the second, a prefix cache such as the one SGLang already provides addresses the redundancy directly and without a per-model head policy, because the repeated work is skipped entirely rather than reorganized. In the third, the framework's core premise fails: head classification and sparse-MoE policy are prepared offline, so a model revision that changes attention head behavior or router distributions invalidates the frozen artifacts.
The closest comparison in the material is SGLang itself, since RedKnot is built on it and the badge says so. The difference in approach is that SGLang's RadixAttention-style prefix reuse operates on token sequences and reuses KV across requests that share a prefix, while RedKnot operates inside a single request's prefill, splitting heads into reusable and online groups and letting SegPagedAttention give each group a different context scope. One is cross-request reuse; the other is intra-request work reduction. They are complementary in principle, and the README notes RedKnot's gains are measured against a full online Recomputed path rather than a prefix-cache hit, which suggests the two are not being conflated. If your traffic has heavy prefix sharing, measure the prefix-cache baseline first, because RedKnot's published numbers deliberately exclude it.
Maintenance posture, licensing and what to verify before adopting
The licence is Apache-2.0, stated in the README badge and the LICENSE file. That is a permissive licence with an explicit patent grant and no copyleft obligation on your own code, but it says nothing about the licences of the model checkpoints you point it at, and DeepSeek-V4-Flash has its own terms that this repository does not restate. The kernel dependencies (FlashMLA, SGL Kernel, FlashInfer, Triton) carry their own licences and their own version constraints, and the README pins exact versions for the published run. Treat those pins as the supported combination rather than as examples.
On maintenance, the repository is not archived, the default branch is `main`, and the last push is dated 2026-09-03. There are no retrieved releases, so there is no tagged version to pin against and no changelog to read between commits. The README's Future Work section commits to DeepSeek V4 Pro adaptation, Qwen3.5-to-Qwen4 and GLM-5.3 adaptation results in September to October 2026, and continued Ascend work driven by Huawei Cloud. Those are stated intentions, not shipped code. The practical cost of adopting RedKnot today is that you are tracking `main` on a research repository with one frozen configuration, and any upgrade means re-validating the head policy, the sparse-MoE policy and the execution manifests against your checkpoint. Budget for that re-validation as a recurring cost, not a one-time setup step.
Before you commit, verify three things against the repository itself. First, the per-suite result JSON for the context length closest to your production traffic, since the README names it as the source of truth and the summary chart is not. Second, that the frozen head policy in the execution manifest matches the exact model revision you intend to serve, because the layer ranges and head counts in the DeepSeek-V4-Flash table are specific to that checkpoint. Third, whether the Recomputed reference path runs cleanly on your hardware before you enable any of the RedKnot mechanisms, since that reference is the baseline every published comparison depends on.
Editorial conclusion
Adopt RedKnot if you serve a long-context model on the DeepSeek-V4-Flash TP8 profile and you are willing to run its Recomputed reference on the same checkpoint and input IDs to see whether the head policy holds for your traffic. Do not adopt it if you need a stable release, a non-NVIDIA or Ascend path today, or a guarantee about end-to-end throughput, since the project's own compute ledger excludes memory traffic, kernel-launch cost and TP communication. Before committing, verify the per-suite result JSON for your context length, confirm the frozen head policy in the manifest matches your model revision, and check docs/ASCEND.md if your target hardware is Huawei.
Community notes