1Cat-vLLM: A Volta/SM70 Engineering Fork of vLLM

The problem is software neglect, not silicon age

The README states the premise directly: Tesla V100 shipped in 2017, its Tensor Cores did not become useless, and the software stack stopped being optimized seriously for SM70. That is the gap this fork exists to fill. Upstream vLLM supports Volta in the sense that a model can load and produce tokens, but the kernel selection, attention paths and quantization support are tuned for newer architectures. 1Cat-vLLM is an engineering fork that treats SM70 as the optimization target rather than a compatibility checkbox. The audience is therefore narrow and identifiable: operators with V100 16GB cards already in racks, who want to serve current-generation models such as Qwen3.8-27B-NVFP4 or DeepSeek-V4-Flash without replacing hardware. The README frames the goal as making modern models run fast on V100, not merely start on V100, and every benchmark in the document is expressed as a V100 result.

Attention kernels and the precision contract behind the numbers

The most concrete engineering claim is in the long-context attention path. The README reports useful causal attention compute moving from 17.92 TFLOP/s on the v1.2.2-era baseline, to 46.63-47.1 TFLOP/s with a D256 Split-D / N32 layout in v1.3.0, to roughly 60.8 TFLOP/s with GQA-packed wide QK/PV in PR #286, where six GQA heads are packed into wider Tensor-Core GEMMs. A research result in PR #315 reaches about 79 TFLOP/s, but the README explicitly labels it not a release or default quality claim. That distinction matters: the project separates what ships from what has been demonstrated in a branch. Equally important is the precision contract. SM70 Flash-V100 now resolves `--kv-cache-dtype fp8` to E4M3, DFlash2 E4M3 verification uses repaired FP32 attention state, and the Qwen3.8 DFlash2 configuration enables FP32 logits by default. The README instructs users to rebuild Flash-V100 for precision revision 4 and points at `docs/design/sm70_dflash2_fp32_defaults.md`. Historical E5M2 and FP16-partial results are kept in their original configuration and are explicitly not speed claims for the new defaults. Anyone reading the headline throughput figures without reading that paragraph will misattribute them.

Speculative decoding is where the throughput comes from

The benchmark tables show the largest gains arriving from speculative decoding rather than from a single kernel. Qwen3.6-27B-AWQ with MTP4 reaches 100.564 tok/s at 64K decode and 85.258 tok/s at 128K, which the README describes as +87.64% versus no-MTP. Qwen3.6-35B-A3B NVFP4 goes from 116.99 tok/s without MTP to 174.76 tok/s with it, a 1.49x ratio. Qwen3.8 Flash-Next-NVFP4 moves from 80.732 tok/s pure decode to 138.26 tok/s with MTP4. The DFlash2 path is reported at 206.06 tok/s streaming decode on a historical web prompt, with 17.463 ms per round and 3.599 emitted tokens per round, and 251.60 tok/s on an MBPP item with an acceptance length of 4.686 and an EvalPlus result of 1/1. An opt-in lookup augmentation with q16 is reported at 316.27 tok/s, but the README labels it a special opt-in contract, so it is not the default configuration. The pattern is consistent: the fork's throughput story depends on draft models, acceptance length and CUDA Graph capture working together, and the README publishes acceptance length alongside the rate rather than the rate alone.

Quality gates sit next to the throughput table

The README makes an argument worth taking seriously: raw tok/s alone can turn optimization into a benchmark game, so the project records model throughput, dataset score, natural-stop health, output validity and speculative acceptance together. The practical 16K coding gate for Qwen3.8-27B-NVFP4 with DFlash2 spells out the contract in detail: 4x V100, TP4, NVFP4 target, official BF16 DFlash2 drafter, FP8 E5M2 target KV, FlashAttention-V100, full CUDA Graph, prefix cache, Mamba align, `temperature=1.0`, `top_p=0.95`, `top_k=20`, xhigh reasoning, a 16K natural-EOS output cap, and three predeclared sampling seeds. That level of specification is unusual in inference project documentation and it is the strongest signal in the repository that the maintainers expect their numbers to be reproduced rather than quoted. The DeepSeek-V4-Flash rows follow the same pattern, pairing 73.539 tok/s on a PP2xTP4 strict control with GSM8K 64/64 and HumanEval 29/32. The cost of this rigor is that the tables are dense and configuration-specific, and the README says so: attention TFLOP/s, prefill tok/s, target-only decode tok/s and speculative decode tok/s are not interchangeable.

Getting it running, and what the README does not tell you

The material supplies fewer installation specifics than the performance sections. What it does give are the configuration surfaces that matter. The KV cache dtype is set with `--kv-cache-dtype fp8`, which on SM70 Flash-V100 resolves to E4M3. Flash-V100 must be rebuilt for precision revision 4, and the precision contract and validation live in `docs/design/sm70_dflash2_fp32_defaults.md`. The benchmark contracts name the operational flags in prose: TP4 or TP8 or PP2xTP4 depending on card count, full CUDA Graph, prefix cache, Mamba align, and sampling parameters such as `temperature=1.0`, `top_p=0.95` and `top_k=20`. What the README does not provide in the supplied text is a pip install line, a Docker image name, a CUDA toolkit version requirement, or a build command for Flash-V100. The homepage field is empty and the releases are tagged rather than accompanied by installation notes in this material. Anyone adopting the fork should expect to read the repository's build files and the precision document directly, and should not assume a wheel exists that matches their environment.

Where a V100-specific fork is the wrong tool

The clearest limitation is structural: this fork optimizes for one architecture. If your fleet is Ampere, Hopper or newer, the kernel work described here does not apply to you, and you would be taking on a fork's maintenance surface for no benefit. The second limitation is precision. The README's own framing shows that the reported numbers depend on a specific KV dtype resolution (E4M3), on FP32 attention state repair for DFlash2 E4M3 verification, and on FP32 logits being enabled by default for the Qwen3.8 DFlash2 configuration. Changing any of those defaults invalidates the corresponding performance figures. The third is model coverage. The benchmark tables cover a specific set of models (Qwen3.6, Qwen3.8, DeepSeek-V4-Flash, GLM-5.3-Flash-NVFP4) and quantization formats (AWQ, NVFP4, FP8, MXFP4 experts). A model outside that set has no published evidence behind it here, and the README's own warning that its numbers retain their individual hardware, model, context length, batch size, KV dtype, sampling policy and speculative-decoding contract means you cannot extrapolate a rate from one row to a different workload.

Upstream vLLM versus a Volta-specific fork

The obvious alternative is upstream vLLM itself. The difference in approach is not a feature checklist, it is where the optimization effort goes. Upstream vLLM is a general-purpose serving engine that supports many architectures and quantization schemes, and its default kernel paths target the hardware most of its users run. 1Cat-vLLM takes the same codebase and a much narrower target, spending its effort on SM70-specific attention layouts such as D256 Split-D / N32 and GQA packing, on Flash-V100 precision handling, and on speculative decoding configurations tuned for V100 memory budgets. The practical consequence is that upstream vLLM will likely be easier to install, better documented for general cases, and more predictable across hardware, while 1Cat-vLLM publishes V100 decode rates that upstream does not attempt to match. Choosing between them is a question of what hardware you already own. If the answer is V100, the fork's benchmark tables are the only evidence in this material that current models run at the rates shown. If the answer is anything else, upstream is the default and this fork adds risk without a corresponding gain.

Maintenance cost, licence and upgrade path

1Cat-vLLM is licensed Apache-2.0, the same licence family as upstream vLLM, which means the fork can be used, modified and redistributed under those terms. That is a statement about the licence text, not legal advice, and teams with compliance requirements should read the LICENSE file and any bundled third-party notices themselves. The maintenance picture visible in the material is active: three releases in roughly two months (v1.2.2 in July 2026, v1.3.0 in August, v1.5.0 in September) and a last push date of 2026-09-10. That cadence cuts both ways. It means fixes arrive quickly, and it means the precision contract can move under you: the README already instructs users to rebuild Flash-V100 for precision revision 4, and it explicitly quarantines older E5M2 and FP16-partial results as not applicable to the current defaults. Upgrading therefore is not a version bump. It is a rebuild plus a re-validation of your own acceptance lengths and dataset scores against the new precision revision, because the fork's own history shows that a precision change can invalidate previously published throughput numbers.

Editorial conclusion

Adopt 1Cat-vLLM if your inference fleet is Tesla V100 and you are prepared to rebuild Flash-V100 for the current precision revision and to track the project's own acceptance-rate and dataset gates. Do not adopt it if you run Ampere, Hopper or newer parts, or if you need a drop-in upstream vLLM wheel. Verify first that your model and quantization combination appears in the benchmark tables, that `--kv-cache-dtype fp8` resolving to E4M3 matches your accuracy budget, and that you can reproduce the documented precision contract on your own hardware before committing.

Official sources

1. 1CatAI/1Cat-vLLM on GitHub
2. Issues
3. License: Apache-2.0
4. README
5. Releases