MInference: Sparse Attention for Long-Context LLM Pre-filling
[NeurIPS'24 Spotlight, ICLR'25, ICML'25] To speed up Long-context LLMs' inference, approximate and dynamic sparse calculate the attention, which reduces inference latency by up to 10x for pre-filling on an A100 while maintaining accuracy.
At a glance
- What is it?
- MInference, from Microsoft, targets the pre-fill stage of long-context LLM inference with per-head sparse attention patterns and custom kernels. The documentation claims up to 10x pre-fill speedup on an A100, but the real decision is whether your deployment can accept an offline pattern-profiling step and a Triton dependency.
- Who is it for?
- Adopt MInference if your workload is pre-fill dominated, your prompts run into the hundreds of thousands of tokens, and you can afford an offline head-pattern profiling pass plus a Triton dependency in your serving stack. Avoid it if your traffic is decode-heavy with short prompts, if you cannot run the profiling step, or if you need a drop-in library with no kernel-level integration.
- Can I use it commercially?
- Yes. MIT is a permissive licence: you can use, modify and sell software built on it, as long as you keep its copyright and licence notices.
- Is it still maintained?
- Yes. The repository last received commits 5 days ago.
- What is it written in?
- Mainly Python, according to GitHub's language statistics.
Answers come from the project's GitHub data, last synced on September 15, 2026, and from our analysis. They are not legal advice.
DEEP OPEN-SOURCE ANALYSIS
The pre-fill bottleneck MInference was built for
Standard attention scales quadratically with sequence length, so the pre-fill phase, where the model processes the entire prompt before generating the first token, becomes the dominant cost once contexts reach tens or hundreds of thousands of tokens. Decode is cheap by comparison because it processes one token at a time. MInference targets exactly this asymmetry. The README frames the goal as processing 1M context 10x faster in a single A100 using long-context models such as LLaMA-3-8B-1M and GLM-4-1M. The project is aimed at engineers serving long-context models who have measured their latency and found pre-fill, not generation, at the top of the profile. If your prompts are short or your workload is decode-bound, the mechanism has nothing to act on.
Per-head sparse patterns, profiled offline and approximated online
The TL;DR in the README describes a two-stage design. First, the system determines offline which sparse pattern each attention head belongs to. Second, it approximates the sparse index online and computes attention with custom kernels selected for that pattern. The premise is that attention in long-context LLMs is dynamically sparse but exhibits some static structure per head, so the pattern can be classified ahead of time and the runtime only needs to approximate the index rather than search for it. That two-stage split is the core architectural decision: the offline step moves expensive analysis out of the request path, and the online step keeps only what depends on the actual input. The README does not spell out the profiling cost or how many heads must be classified, so treat that step as a fixed setup expense whose duration you should measure yourself.
Getting it running: install, config objects, and backend choice
Installation is a single pip command: pip install minference. The README lists Torch, FlashAttention-2 (optional), Triton, and Transformers >= 4.46.0 as requirements. The version floor on Transformers is not incidental; it reflects the API surface the integration relies on, so an older pinned version in your environment will block you. Supported attention and KV cache types are discoverable at runtime rather than only from docs: from minference import MInferenceConfig, then MInferenceConfig.get_available_attn_types() and MInferenceConfig.get_available_kv_types(). That is a useful design choice, because the supported list grows between releases and hardcoding it in your own code would drift. The README groups the supported methods under KV Cache Generation, including MInference itself, xAttention, FlexPrefill, A-shape, Tri-shape, MInference w/ static, Dilated, and Strided. For serving, the project notes that SGLang and vLLM have merged the MInference sparse attention kernel, and that pip install sglang gives access to the optimized kernels, with SGLang additionally adapting it for FlashAttention-3. If you already run one of those servers, that path avoids writing your own kernel integration.
The speedup numbers and what they actually measure
The README states up to 10x speedup for pre-filling on an A100 while maintaining accuracy, and the news section lists a finer-grained set of figures through the SGLang integration: up to 1.64x at 64K, 2.4x at 96K, 2.9x at 128K, 5.2x at 256K, 8x at 512K, and 15x at 1M. The pattern is the point. Gains grow with context length because the quadratic term being avoided grows with it, so a 64K prompt sees a modest improvement while a 1M prompt sees an order of magnitude. Anyone reading the 10x headline as a general-purpose multiplier will be disappointed at shorter contexts. The README also claims even better accuracy in its promotional line, but it does not present the per-task accuracy tables inline, so the accuracy claim needs to be checked against the linked paper and, better, against your own evaluation set before you turn the method on in production.
Where MInference is the wrong tool
The mechanism is scoped to pre-filling. The README repeats this in the TL;DR and in the MMInference description, which explicitly says prefilling-stage. Decode latency is not the target, so a deployment dominated by token generation will see little benefit regardless of context length. The offline pattern determination is a second constraint: it assumes a stable model whose head patterns can be classified once, which fits a fixed checkpoint and fits poorly when you swap models frequently or fine-tune per tenant. The kernel dependency is a third. Triton is listed as a requirement, and the optimized kernels live in SGLang and vLLM rather than in the base package, so teams on a different serving stack either port the integration or accept the standalone path. Finally, the README does not document accuracy degradation boundaries per task type, so for workloads where a single missed retrieval in a long context is unacceptable, the sparse approximation is a risk you have to quantify rather than assume away.
Alternatives and how the approach differs
The README's own supported-methods list contains the most direct comparisons. MInference w/ static uses a fixed sparse pattern instead of the dynamic per-input approximation, which removes the online approximation step at the cost of adapting less well to varying inputs. A-shape and Tri-shape are fixed structural patterns rather than per-head classified ones, so they need no offline profiling pass but cannot match a pattern to the head that needs it. Dilated and Strided are positional sparsity schemes, again input-independent. xAttention and FlexPrefill are listed alongside MInference as separate methods in the same KV Cache Generation category, which tells you the project treats them as peers rather than replacements. Outside this list, the README points to RetrievalAttention, described as KV cache offloading that accelerates long-context inference via vector retrieval. That is a different axis entirely: it moves KV cache rather than sparsifying the attention computation, so it addresses memory pressure where MInference addresses compute. The choice between them depends on whether your bottleneck is FLOPs during pre-fill or KV cache capacity.
Maintenance, releases, and licence terms
The release history shows an uneven cadence. v0.1.5 landed in July 2024, v0.1.5.post1 in August 2024 with support for LLaMA-3-70B and multi-GPU plus kernel and sqrt(dk) fixes, and v0.1.6 arrived in June 2025 adding SCBench. That is roughly a ten-month gap between the last two releases, and the version number has stayed in the 0.1.x range, which is a signal about API stability expectations rather than a verdict on the code. The repository is not archived and the last push is recent, so the project is active. The licence is MIT, which is permissive and places few obligations on how you redistribute or modify the code, but this is a factual note about the licence identifier and not legal advice; if you are embedding the kernels in a commercial serving product, have your own counsel review the dependency chain, including Triton and any FlashAttention components you pull in. Upgrade cost is dominated by the Transformers version floor and by kernel changes in SGLang or vLLM, not by the minference package itself.
What to verify before you switch it on
Start by calling MInferenceConfig.get_available_attn_types() in your target environment to confirm the attention type you intend to use is present in the installed version, since the supported list is release-dependent. Confirm Transformers >= 4.46.0 and Triton resolve cleanly against your existing pins, because a conflict there will surface at import time rather than at request time. If you serve through SGLang or vLLM, verify the merged kernel is in the version you run rather than assuming the merge is available in your build. Then run your own accuracy evaluation at the context lengths you actually serve, because the published gains scale with length and a 64K workload will not look like the 1M headline. The project is a research artifact with a serving integration, not a drop-in switch, and the profiling step plus kernel dependency are the two costs that decide whether it fits.
Editorial conclusion
Adopt MInference if your workload is pre-fill dominated, your prompts run into the hundreds of thousands of tokens, and you can afford an offline head-pattern profiling pass plus a Triton dependency in your serving stack. Avoid it if your traffic is decode-heavy with short prompts, if you cannot run the profiling step, or if you need a drop-in library with no kernel-level integration. Before committing, verify which of the supported attention types matches your model family, confirm that Transformers >= 4.46.0 and Triton are pinned in your environment, and reproduce the accuracy comparison on your own task rather than trusting the headline speedup figure.
Community notes