SSD: drafting and verification on separate GPUs, in parallel

What changes in SSD

The README states the claim flatly, that SSD is a new LLM inference algorithm, that it is exact, and that it is extremely fast.

The mechanism is best understood by contrasting it with ordinary speculative decoding. In normal speculative decoding, a small and fast model guesses the next few tokens that a larger slower model may generate, and the large model then verifies them in one forward pass. The important detail is that drafting and verification happen one after the other on the same hardware, so the GPU runs the drafter and the target model in turn, and the drafter's work is time the target model spends waiting.

In SSD, they happen in parallel, on distinct hardware. The small model anticipates likely verification outcomes in advance and speculates for all of them at once. And if it guessed correctly, the speculation can be returned immediately, so the drafting overhead is eliminated entirely.

The word exact matters too, and it is the property that separates this from the usual trade in speculative decoding. A speculative method that changes the output distribution is a different model, and any measurement of it is measuring a different thing. The README leads with exactness for that reason, and the Borges epigraph about choosing all the alternatives at once is a fair description of the mechanism.

The paper is on arXiv, the citation is to ICLR 2026, and the authors are Tanishq Kumar, Tri Dao and Avner May. The project is MIT licensed and Python.

The hardware consequence, which is the whole design

It is easy to read SSD as a software trick, and it is not. The parallelism requirement changes what the machine has to look like, and that is what the README's benchmark commands are really about.

In the standard speculative decoding setup, the drafter and the target model share the same devices, so the drafter's forward passes are serialised with the target's. In SSD the drafter runs on its own devices while the target model verifies on others, which is why the benchmark commands for the asynchronous variant allocate a fifth GPU where the synchronous one uses four.

The three benchmark lines in the README show the whole matrix. The autoregressive baseline is the 70B model on four GPUs. The synchronous speculative decode is the same 70B target with a 1B draft on four GPUs. And the asynchronous speculative decode, which is SSD, is the 70B target on four GPUs plus the 1B draft on one more, with a different set of parameters for the number of speculated tokens and the number of alternatives.

So the honest description of the result is that SSD trades hardware for latency, and the claim that drafting overhead is eliminated is a claim about a configuration where the drafter has a device to itself. On a single device the method would be no better than the synchronous one, and the README does not pretend otherwise.

The roadmap confirms the shape of the problem. A planned item is draft data parallelism, to increase the speculation cache size, on up to four devices, and the stated reason is to avoid getting compute bound. In other words, the authors are still working on the point where the drafter becomes the bottleneck.

What the engine actually contains

The engine supports a defined list of things, and the list is a reasonable summary of the scope of a research artefact.

There is a reference implementation of the SSD algorithm, alongside optimised speculative decoding and autoregressive baselines, so all three approaches are measured in the same harness with the same model and the same data. The model families are Qwen3 and Llama3, and the concurrency work includes Tensor Parallelism.

On the performance side, the list includes PagedAttention, CUDAgraphs, torch compilation and prefix caching. Those are the four usual levers in a modern inference engine, and the fact that they are present matters for a benchmark, because a speculative method compared against a baseline that lacks them would be measuring the baseline's implementation quality rather than the method.

The repository layout is small and says what is where. There is a package directory for the engine itself, a bench directory for the benchmark and chat harnesses, a scripts directory for the model and dataset downloaders, the assets directory for the figure in the readme, and the project metadata with a lock file.

The project metadata describes the package as an async tree-based speculative decoding research engine, at version 0.2.0, and the version number says the same thing as the missing GitHub releases, which is that this is code that is used and changed rather than packaged and consumed.

Setup, and the three environment variables that matter

The requirements are narrow and stated up front. Python 3.11 or later, CUDA 12.8 or later, and the code was written and tested on H100s.

That last sentence is the one to read twice, because it is the difference between a project you can try and a project you need the right hardware for. The architecture is configurable through an environment variable, and the README gives the mapping, with 9.0 for H100, 8.0 for A100 and 8.9 for the L40 and the 4090.

The install path uses uv, and the project assumes it is present, with a one line installer and a note about adding the local bin directory to the path if the shell cannot find it. Then the clone and the sync.

git clone https://github.com/tanishqkumar/ssd && cd ssd
uv sync                    # core SSD deps
source .venv/bin/activate
python -c "from ssd import LLM; print('ok')"

The one line python check at the end is the smoke test, and it is worth doing before downloading tens of gigabytes of weights.

Three environment variables configure the paths. SSD_HF_CACHE has to point at the HuggingFace hub directory, the one that contains the model directories with the double dash in the name, and the README is emphatic that it is not the parent of it. SSD_DATASET_DIR points at the directory holding the dataset subdirectories. And SSD_CUDA_ARCH takes the architecture value.

export SSD_HF_CACHE=/path/to/huggingface/hub
export SSD_DATASET_DIR=/path/to/processed_datasets
export SSD_CUDA_ARCH=9.0   # 9.0=H100, 8.0=A100, 8.9=L40/4090

The model and dataset downloaders live behind an optional extra, because they pull in the datasets and hub libraries that the engine itself does not need.

The benchmark harness and how to read its numbers

All the commands run from inside the bench directory, and the README gives two pieces of advice that decide whether your numbers mean anything.

The first is that large models take a few minutes to load, warm up and compile before generation starts, so a run that appears to hang at the start is normal. The second is to always use the -O flag on the python invocation, because the debug overhead otherwise distorts the timings of a method whose entire claim is about latency.

The harness itself has a structure worth understanding. The all flag runs the full evaluation across four datasets, and the number of sequences is per dataset, so a run with 128 sequences and the all flag is 512 prompts in total. The README explains why the aggregate matters, since different data distributions are predictable to varying degrees, the speed of speculative decoding and SSD depends a lot on the dataset, and averaging over many prompts from many types of datasets gives an overall picture.

That is an honest piece of benchmarking advice, and it is a caveat you should apply to any result you read elsewhere too. A method that is fast on prompts the drafter predicts well and slow on the others will produce a flattering number on one dataset and a damning one on another.

The three reference commands are the autoregressive baseline, the synchronous speculative decode and the asynchronous one, all at temperature zero with a fixed output length and the same number of sequences, and all with the output and trajectory saved for inspection.

Chat mode, and comparing against other engines

The chat harness is the interactive path, and its limits are stated plainly. It is interactive streaming chat with Llama-3.1 70B only, and it supports the autoregressive mode, the synchronous speculative decoding and the asynchronous one, with a metrics flag that prints the token count, the speed and the time to first token after each response.

The time to first token is the number that matters for a chat interface, and printing it per response is the right choice for a method whose claim is about latency.

cd bench

# Async spec decode (SSD) — 5 GPUs, k=7, f=3
python -O chat.py --ssd --spec --async --k 7 --f 3 --gpus 5 --metrics

The other half of the comparison work is the external baselines. The SGLang and vLLM chat backends are supported, and the launchers start their servers automatically so the comparison is a one line change to the command.

python -O chat.py --sglang        # spec decode
python -O chat.py --sglang --ar   # autoregressive

That matters for how you read the results. If the only baseline is the author's own implementation, then a difference could be the implementation rather than the method, and the fact that two other engines can be launched with the same interface removes that objection. The full argument list is in the bench readme, and the project also points at it for the SGLang and vLLM baselines.

The pinned dependencies, the roadmap and what is not here

The dependency list explains what kind of project this is, because the versions are exact.

The core is torch 2.8.0, triton 3.4.0 and transformers 4.57.1, with numpy, safetensors, a tokeniser library, a progress bar and a hash library. Around that sit the specialised inference packages, with flashinfer-python, sgl-kernel and the CUDA kernel DSL from NVIDIA, plus a weight and bias client and a transfer helper for model downloads.

Exact pins on torch, triton and transformers are normal in a research repository, and they are a warning to anyone planning to build on it. The engine is not a library with a stable interface, it is a codebase pinned to a working combination, and the optional scripts extra with the datasets and hub libraries is separate for the same reason, so the core install stays small.

The roadmap lists three things: draft data parallelism on up to four devices to increase the speculation cache size and avoid becoming compute bound, OpenAI compatible inference over HTTP, and new model families with mixture of experts support, naming GPT-OSS and Kimi-K2.5.

The OpenAI compatible endpoint is the one to watch, because it is the difference between a research harness and something you could put in front of a client library. It is on the list rather than in the code.

There is also a hard Python boundary in the metadata, a range of 3.11 up to but not including 3.13, and the package is not published with releases of its own. So the project is a repository to clone and read, not a dependency to add, and the README's closing line says that contributions are welcome.

Editorial conclusion

Adopt this engine if you are reproducing the speculative decoding paper or comparing decoding strategies under identical conditions, because the repository ships the optimised speculative and autoregressive baselines next to the new method and the same benchmark harness drives all of them. Do not adopt it as a production inference stack, because the pinned CUDA toolchain, the H100 assumption and the absence of releases make it a research artefact. Before you run anything, set the three environment variables the code reads, and check the CUDA architecture value against your own GPU, since the default is written for the H100.

Frequently asked questions

Official sources

1. Issues
2. License: MIT
3. README
4. tanishqkumar/ssd on GitHub