What is Speculative decoding?
Speculative decoding makes autoregressive generation faster by having a small draft model propose several tokens at once and letting the large model verify them in a single batched pass, keeping only the tokens it would have produced itself. The open-source work ranges from training frameworks for draft models to Apple Silicon ports, and the gain depends almost entirely on how often the drafts turn out to be right.
How it works: draft, verify, repeat
Autoregressive decoding is serial by construction. To produce the tenth token a model has already produced the first nine, and each step is a separate pass over the weights. That makes generation memory-bandwidth bound: the arithmetic units sit idle while the weights stream in.
Speculative decoding attacks the serialism rather than the arithmetic. A small draft model, often several times smaller and therefore several times faster, proposes the next k tokens cheaply. Those k tokens are then handed to the large target model all at once, as a prefix to score, which is a single forward pass where decoding would have taken k of them. Because the target sees a real prefix rather than a sampled continuation, it can check all k positions at once.
The accept rule is what makes the technique interesting. Walk the drafted tokens and the target's own predictions in order: where the drafted token is the one the target would have chosen, keep it and continue. At the first disagreement, keep the target's token instead and throw away the rest of the draft. Whatever the outcome, the tokens you keep are exactly the ones plain decoding would have produced, because the target's distribution, not the draft's, decided every position.
The draft does not have to be a separate small model. Some variants reuse multi-token prediction heads the target model already has, some train a drafter to match the target's internal representations rather than its output, and some use a diffusion process to propose a block of tokens at once. The acceptance test is the same in each case.
Why identical output is the selling point
Most inference optimisations trade accuracy for speed. Quantisation rounds weights to fewer bits, and sparse attention drops context, and both change what the model computes. Speculative decoding does not, and that is the reason it is easy to adopt: our analysis of ARahim3/mlx-dspark describes it as lossless because the target model verifies every draft token, so outputs are bit-for-bit identical to plain decoding.
The catch is that identical output is not the same as a speedup. The arithmetic is a rough sum. Proposing k tokens costs k draft steps, verifying them costs one target pass, and if the drafts are wrong you have paid both and thrown the rest away. So the speedup is a function of three things: how cheap the draft model is relative to the target, how many tokens you speculate at once, and the acceptance rate, which is the probability that the drafted token matches what the target would have said.
That makes acceptance rate the number to measure, and it is not a property of the target alone. It depends on the pairing, on the temperature you sample at, and on the workload. Greedy decoding and low temperatures accept more drafts; high temperatures accept fewer, because the distribution has more mass outside the target's top choice. A draft model trained for a different domain also accepts less, no matter how good it is in general. When people report large gains, they are usually reporting a specific pairing, a specific temperature and a specific workload, which is why those numbers travel badly.
When it helps, and when it does not
The gain lives in one place: single-stream decoding where the accelerator is waiting on memory. That describes interactive chat, agent loops, and code completion, where one request is generating tokens one at a time and the batch is nearly empty.
It is worth setting up when the target model is large relative to the hardware, when the output is long enough that the fixed cost of one extra pass is amortised, and when a draft model exists that is trained for the same target. Drafts of a few tokens at a time are the common setting, since acceptance falls as you speculate further ahead.
It is not worth setting up when the workload is prefill-heavy, since prefill is compute bound and there is no serial chain to shorten. It is not worth it when you are already running large batches, because then the memory system is being used efficiently and the serial bottleneck you are removing was never the constraint. It is rarely worth it for a small target model, where one pass is cheap enough that the draft and verify overhead dominates. And it is not worth it if you cannot measure acceptance rate per workload, because without that number you are guessing whether the extra machinery earned its place.
The limits that decide it on your machine
Five practical limits decide whether a speculative setup is worth running.
Memory. The draft model has to be resident at the same time as the target, and the verification pass needs room for k positions of activations. Our analysis of youssofal/MTPLX describes a setup that runs Qwen 3.8 Flash Next and Qwen 3.8 27B on a Mac using the model's own multi-token prediction heads with an OpenAI and Anthropic compatible server on port 8000, and our analysis of MTPLX notes that the memory floor is real and the project is Mac only. ARahim3/mlx-dspark is likewise Apple Silicon, via MLX.
Checkpoint availability. A draft is only useful paired with the model it was trained for. Our analysis of z-lab/dflash says its usefulness depends entirely on which checkpoint you pair it with, and deepseek-ai/DeepSpec ships pre-released Hugging Face checkpoints alongside its training and evaluation code, which removes that problem for its own algorithms and only for those.
Framework support. Our analysis of vllm-project/speculators says the training pipeline is well defined but the deployment path still depends on vLLM support for each algorithm, and our analysis of sgl-project/SpecForge says it is opinionated about serving and thin on installation detail. Our analysis of NVIDIA/Model-Optimizer adds that the useful path runs through NVIDIA's stack, with documentation that is uneven outside the LLM examples.
Sampling temperature. Acceptance is a comparison against the target's most likely token, so it drops as the distribution flattens. Specifying a fixed acceptance behaviour is the setting where the method behaves most predictably.
And the fallback. Any implementation has to define what happens when every draft is rejected, because that case must still produce a correct token. A design that assumes drafts are usually right is a design that will eventually emit something wrong.
Where the open-source work sits
The projects divide into training, serving and hardware ports.
Training draft models. vllm-project/speculators is a Red Hat library that trains draft models for speculative decoding and packages them in a Hugging Face-compatible format for vLLM. sgl-project/SpecForge is the SGLang team's framework for the same job, with typed configs for online and offline topologies. deepseek-ai/DeepSpec is a full-stack MIT-licensed Python repository covering data preparation, draft model training and evaluation, with three algorithms implemented, DSpark, DFlash and Eagle3, plus checkpoints and an evaluation suite across nine benchmark datasets.
Draft models themselves. z-lab/dflash is a block diffusion draft model shipped as a Python CLI with Transformers, MLX and OpenAI-compatible server backends, described as a beta package. youssofal/MTPLX is the unusual one of the group, since it uses the target model's own multi-token prediction heads rather than a separate drafter.
Compression libraries that include it. NVIDIA/Model-Optimizer lists speculative decoding among quantization, distillation, pruning and neural architecture search, and our analysis of it says it turns a Hugging Face, PyTorch or ONNX checkpoint into a compressed one that TensorRT-LLM, TensorRT, vLLM or SGLang can serve.
Hardware ports. ARahim3/mlx-dspark is a native MLX port of DeepSeek's DSpark and z-lab's DFlash for Apple Silicon, and our analysis records up to 4.3x faster token generation with the lossless guarantee described above.
Teaching and reference material. zengxiao-he/tessera is the most complete example: a Python project that builds a stack from scratch, distilling a forty-million-parameter teacher into a six-million-parameter student with custom Triton and CUDA kernels, paged KV cache, continuous batching, speculative decoding and a Rust HTTP gateway, running on CPU and Apple MPS without a GPU, with every component unit-tested against reference implementations. patchy631/time-to-first-token is the opposite end, a fifty-session, twenty-five-hour plan for learning inference serving by building one OpenAI-compatible service on rented GPUs.
In practice
Speculative decoding is a lossless speedup whose value is set by three numbers you can measure: the acceptance rate, the ratio between draft and target cost, and how many tokens you speculate at once. If you have a large target model and a low-batch workload, read deepseek-ai/DeepSpec for the algorithms, vllm-project/speculators or SpecForge for training a drafter, and zengxiao-he/tessera for the whole stack built from scratch. Before wiring it up, check that a draft checkpoint exists for your exact target, count the memory for both models at once, and instrument acceptance per workload so you can tell whether the extra pass is paying for itself.