Speculative decoding is the technique where a small model proposes tokens and a large model verifies them in one pass. It is on every inference server's feature list and almost every article about it is written for people with a research cluster. Here is the version for a person with one GPU.
The idea in one paragraph
The large model is slow because it runs one token at a time, and each step is memory-bound: it has to read the weights from the card. A small model, which fits in the same memory and is fast, proposes the next few tokens. The large model then checks all of them in a single forward pass, which costs barely more than checking one, because the cost is reading the weights, and the weights are read either way. If the proposals are right, you get several tokens for the price of one. If they are wrong, the wrong token is discarded and you are back to one token per step, minus a little overhead.
The gain is therefore entirely about how often the small model agrees with the large one. That rate is called the acceptance rate, and it is high for predictable text — code, structured output, boilerplate — and low for anything requiring reasoning.
When it is worth trying
The output is predictable. Code completion, JSON in a schema, templated text, translating between similar languages. If the model is writing something a smaller model would also write, the acceptance rate is high and the speed-up is real.
The draft model is small relative to the target. A 1B draft for a 32B target is a good ratio. A 7B draft for a 32B target is a bad one: you are paying for two large forward passes.
The target model is the bottleneck. If your request spends its time waiting on the network, or the GPU is already saturated by other traffic, speculative decoding buys you nothing and adds a model to load.
When it is not
If your target model fits comfortably and you are one user, the arithmetic is unfavourable: you add a second model to load, to keep in memory, and to serve, and the win only materialises on long generations with predictable text. On a single GPU where memory is the constraint, loading a draft model may be the thing that pushes the real model off the card.
Both vLLM and SGLang expose this as a flag. /compare/sglang-vs-vllm has the pair, and /learn/speculative-decoding has the background. The honest summary: it is a throughput feature for a server with spare memory, not a latency feature for a laptop.