Speculators: training speculative decoding draft models for vLLM

The problem Speculators solves is the gap between a draft model and a serving engine

Speculative decoding is lossless in the sense the README states: every accepted token is guaranteed to match what the main model would have generated on its own. A smaller draft model proposes several tokens, and the base model verifies them in a single forward pass. The latency win comes from replacing several sequential decode steps with one verification step, and the acceptance rate decides how much of that win you actually keep. The hard part has never been the idea. It has been the plumbing: a draft model trained in a research repository rarely loads into a production inference server without a conversion script, a custom attention implementation, or a patch to the engine. Speculators targets that seam. It provides an end-to-end framework to train draft models with reusable formats, and the trained artifacts are meant to run in vLLM directly. The intended user is an engineer who already serves a model on vLLM and wants lower decode latency without changing output quality. It is not aimed at people choosing a base model, and it is not a general inference optimizer.

Training data comes from vLLM itself, which is the architectural decision that matters most

The README describes offline training data generation using vLLM: hidden states are produced by the engine, saved to disk, and reused for draft model training. That choice explains most of the library's shape. Because the verifier's internal states are the training signal, the trainer does not need to hold the full base model in memory for every algorithm. The MTP finetuning path makes this explicit, noting that the native Multi-Token Prediction head is small (roughly 100M to 400M parameters) and can therefore be trained on pre-extracted hidden states without loading the full verifier. For multi-node setups the library adds a separate plugin package, hs_connectors, with pluggable backends for moving hidden states between vLLM and the trainer. One backend uses a shared filesystem; the Mooncake backend uses a distributed store for environments without shared storage. That is an online training path, and it is the part of the design with the most operational surface: you are now running an inference engine and a trainer in the same loop, and the connector between them becomes a dependency you have to keep healthy.

Four training algorithms with different drafting strategies

The release notes describe several algorithms rather than one. DFlash uses anchored-block drafting and auxiliary hidden states from multiple verifier layers, with CLI options for block size and max anchors. DSpark extends DFlash's anchored-block drafting with a Markov head that conditions each draft position on the previous token within the block, plus a confidence head that predicts per-position acceptance probability; DSpark checkpoints can warm-start from existing DFlash checkpoints. P-EAGLE extends the EAGLE-3 architecture with parallel multi-token prediction via Conditional-On-Distribution sampling, so multiple tokens are predicted in a single forward pass instead of sequentially, which the notes frame as a reduction in drafting latency. MTP finetuning adapts the native Multi-Token Prediction heads of models such as Qwen3-Next on domain-specific data, following the FastMTP approach. DFlash and DSpark speculators use sliding window attention on all draft layers by default, with --sliding-window setting the window size and --full-attention-indices opting specific layers into full attention. The stated motivation is reduced KV cache allocation for long-context sequences, and the notes claim it can improve per-position acceptance rates compared to full attention. That last claim is one I would treat as a hypothesis to test on your own data rather than a settled property, because acceptance rate depends on the draft and verifier pair.

Getting a run started: the commands and flags the documentation exposes

The README does not print a full quickstart, so the concrete surface visible in the material is the CLI flag set rather than a copy-pasteable command sequence. What is confirmed: training runs expose --sliding-window for the window size and --full-attention-indices to select layers for full attention; DFlash exposes options for block size and max anchors; the hs_connectors plugin selects a backend, with a file-based backend over a shared filesystem and a Mooncake backend over a distributed store. Data generation is a separate offline step that uses vLLM to emit hidden states to disk. Python support is 3.10 through 3.13, and the package is on PyPI under the name speculators. Deployment is the part with the least friction: the README states that DFlash models trained through Speculators can run in vLLM as of vLLM PR #38300, and that trained models run in vLLM without a described conversion step. If you are evaluating this, the first thing to read is the documentation site rather than the README, because the flag semantics for block size, anchors, and sliding windows are what determine whether a run is even valid for your verifier.

Where it breaks down: hidden states, architecture coverage, and the cost of a training loop

The dependency on hidden states is the main limitation. If you cannot run vLLM against your target model to produce those states, offline training is not available to you, and the online path pulls in hs_connectors plus either shared storage or a distributed store. Neither is free. The second constraint is architecture coverage. The material names MoE, non-MoE, and Vision Language models as supported for draft model training, and names Qwen3-Next as an MTP finetuning target, but it does not enumerate which verifier architectures each algorithm accepts. Warm-starting is described only between DSpark and DFlash, which suggests checkpoint compatibility is algorithm-specific rather than universal. Third, this is a training project, so adopting it means owning a training pipeline: data generation, checkpoint storage, and a re-run whenever the base model changes. If your base model is swapped frequently, the draft model is stale each time. Finally, speculative decoding helps decode-bound workloads. If your traffic is prefill-heavy or your batch sizes are already large enough that the GPU is compute-saturated, a draft model adds work without removing a bottleneck.

The alternative is a general-purpose training stack, and the difference is the output format

The obvious alternative is to train a draft model in a general framework such as Hugging Face Transformers or a research repository and then port it. That path gives you more freedom in architecture and loss design. The difference in approach is not the training loop, it is what comes out the other end. A model trained in a general stack produces weights and a config that you then have to make loadable by vLLM, which in practice means writing the conversion and matching the engine's expectations for the draft model's attention and layer structure. Speculators inverts this: the output is a Hugging Face-compatible format for defining speculative models, and the README also mentions tools to convert from external research repositories into the standard Speculators format. So the library is not only a trainer, it is a format definition with converters on both sides. The trade-off is that you accept the algorithms and formats the project supports. If your research idea falls outside DFlash, DSpark, P-EAGLE, or MTP finetuning, the standard format is a constraint rather than a convenience.

Maintenance, releases, and licence

The project is active: v0.8.0 shipped on 2026-09-03, following v0.7.0.1 and v0.6.0.1 in the preceding month, and the last push to main is dated 2026-09-10. That release cadence is fast enough that pinning a version is sensible for a training pipeline you intend to reproduce. The licence is Apache-2.0, which permits commercial use and modification and includes a patent grant, but the repository's NOTICE and any bundled third-party components are worth reading before you redistribute checkpoints, and I am not in a position to give legal advice on how the licence interacts with model weights you publish. One maintenance cost deserves naming: the vLLM integration is version-coupled. DFlash support in vLLM arrived in a specific pull request, so a Speculators release and a vLLM release have to be chosen together. Upgrading one without the other is the most likely way to break a working deployment. The README points to a vLLM Community Slack with #speculators and #feat-spec-decode channels, which is where version-compatibility questions are most likely to be answered by people running the same combination.

Editorial conclusion

Adopt Speculators if you already serve a fixed base model on vLLM, can generate hidden states offline or stream them online, and want draft checkpoints in a format vLLM loads without conversion glue. Do not adopt it if you cannot produce hidden states for your target model, if you are unwilling to run a trainer alongside the inference server, or if you need a speculative method the library does not implement. Before committing, verify three things in your own environment: that your vLLM build contains the DFlash support added in vLLM PR #38300, that your chosen algorithm (DFlash, DSpark, P-EAGLE, or MTP finetuning) accepts your verifier architecture, and that the acceptance-rate metrics printed by the training run are measured on your traffic rather than a generic corpus.

Official sources

1. License: Apache-2.0
2. Project website
3. README
4. Releases
5. vllm-project/speculators on GitHub