DeepSpec: a training and evaluation stack for speculative decoding draft models

The problem DeepSpec solves: training the draft model, not serving it

Speculative decoding needs two models: a large target model and a small draft model that proposes tokens the target then verifies. Most open source effort goes into the serving side, where the draft model is a fixed artifact you download. DeepSpec takes the other half of the problem. It is a training and evaluation codebase for draft models, aimed at researchers and engineers who want to produce a draft model for a specific target rather than accept whatever draft ships with an inference engine.

The README is explicit about scope. DeepSpec contains data preparation utilities, draft model implementations, training code and evaluation scripts. There is no server, no inference engine and no runtime integration in the repository layout, which lists config/, deepspec/, eval.py, eval_datasets/, scripts/, train.py and requirements.txt at the top level. If your goal is to make an existing deployment faster today, this is not the tool. If your goal is to understand or improve how a draft model is trained against a target model, it is.

The intended audience is narrow in a useful way. The supported target families are Qwen3 and Gemma, and the released checkpoints cover Qwen/Qwen3-4B, Qwen/Qwen3-8B, Qwen/Qwen3-14B and google/gemma-4-12B-it. A team running a different model family has to check whether a matching config exists under config/ before anything else in this repository is worth their time.

Three algorithms, one training framework: DSpark, DFlash and Eagle3

DeepSpec ships three draft model designs. DSpark is the project's own method, described in the cited paper as confidence-scheduled speculative decoding with semi-autoregressive generation. DFlash and Eagle3 come from other projects: the README credits the DFlash repository for the draft-model design and training recipe, and SpecForge for the overall training framework and the Eagle3 implementation, with portions of Eagle3 modeling, loss, optimizer, attention and evaluation code adapted from it.

That provenance matters when you read the code. Adapted files carry an in-file attribution comment, and the full notice is recorded in NOTICE. The practical consequence is that DeepSpec is partly a wrapper around an existing training framework plus two new or ported draft architectures, not a from-scratch reimplementation. If you have already worked with SpecForge, the training loop and config conventions will look familiar.

The algorithm choice is a config choice. Each algorithm has its own directory under config/, for example config/dspark/dspark_qwen3_4b.py, and training selects both the algorithm and the target model by pointing config_path at one of those files. The released checkpoints follow the same naming: eagle3_qwen3_4b_ttt7, dflash_qwen3_4b_block7 and dspark_qwen3_4b_block7 are the three variants for the same 4B target, and each is described as the direct output of the corresponding training configuration.

Installing DeepSpec and running the three-stage workflow

Installation is a single pip command against the pinned requirements file. The README notes that data preparation additionally requires an inference engine to serve the target model when regenerating answers, and points at scripts/data/README.md for that detail. The dependency list pins torch, transformers, numpy, PyYAML, tqdm, tensorboard, matplotlib, triton, typing_extensions, sentencepiece, safetensors and prettytable, plus datasets and openai for data preparation. The file's own comment says to install the CUDA build of torch that matches your machine if the default wheel is not appropriate.

python -m pip install -r requirements.txt

After installation the workflow has three ordered stages, and the README states that each stage's output feeds the next: data preparation, then training, then evaluation. Training is launched by a shell script that runs train.py, which spawns one worker per visible GPU.

bash scripts/train/train.sh

The script header documents how to override config_path and target_cache_dir and how to use --opts to override individual config fields. Checkpoints land in ~/checkpoints/<project_name>/<exp_name>/step_*. The default configs and scripts assume a single node with 8 GPUs; for fewer GPUs the README says to reduce CUDA_VISIBLE_DEVICES.

bash scripts/eval/eval.sh

Evaluation runs eval.py against a trained draft checkpoint over the benchmark set in eval_datasets/, which the README lists as gsm8k, math500, aime25, humaneval, mbpp, livecodebench, mt-bench, alpaca and arena-hard-v2. Two variables must be set: target_name_or_path, the target the draft was trained against, and draft_name_or_path, either a local checkpoint path such as ~/checkpoints/deepspec/dspark_block7_qwen3_4b/step_latest or one of the Hugging Face repo IDs in the released checkpoint table. The metric is speculative-decoding acceptance.

The 38 TB target cache is the real cost of admission

The data pipeline downloads and splits training data, regenerates answers with the target model, then builds a target cache. The README carries a storage warning for that third step: the cache can be very large, roughly 38 TB for the default Qwen/Qwen3-4B setting. That number is not a typo and it is not an upper bound for an exotic configuration. It is the default.

This is the constraint that decides most adoption questions. Regenerating answers requires a served target model, which means an inference engine outside this repository, and the regenerated outputs are then cached so training does not repeatedly call the target. The trade is deliberate: you pay once in storage and generation time to avoid paying on every training step. For a 4B target the price is tens of terabytes.

There are two honest ways around it. Train on a subset of the data, which changes the result and makes the released checkpoints no longer a fair comparison. Or use the released checkpoints directly and skip training entirely, if your target model is one of the four in the table and you do not need domain adaptation. The README's own warning about citation points the same direction: results are only comparable if your setup matches the training settings in the repository, and for domain-specific use it recommends fine-tuning the draft model again, especially if the target model is expected to run in thinking mode. The released checkpoints were trained on open-perfectblend data generated by their target model in non-thinking mode.

Where DeepSpec is the wrong tool

DeepSpec does not serve inference. Nothing in the repository layout is a server, and the README never describes a deployment path. If you want speculative decoding in production, you need an inference engine that supports the draft model format you trained, and that integration is outside this project. The related search phrase about vLLM reflects a reasonable question, but the README does not document a vLLM integration, so treat that as unverified.

It is also the wrong tool for a single-GPU workstation. The default configs and scripts assume one node with 8 GPUs. The README's guidance for fewer GPUs is to reduce CUDA_VISIBLE_DEVICES, which reduces parallelism rather than removing the memory requirement. Nothing in the repository gives a minimum memory figure for training, and the README is silent on whether a reduced-GPU run reproduces the released results.

The third boundary is model family. Only Qwen3 and Gemma targets appear in the supported list and the checkpoint table. If your target is something else, you are looking at porting work, not configuration. There are also no releases retrieved for this repository, so there is no versioned artifact to pin beyond the current main branch and the pinned requirements.txt.

How DeepSpec differs from SpecForge

SpecForge, from the sgl-project, is the closest comparison and also the most instructive one, because DeepSpec is partly built on it. The README states that SpecForge provides the overall training framework and the Eagle3 implementation, and that portions of Eagle3 modeling, loss, optimizer, attention and evaluation code in DeepSpec are adapted from it, under Apache-2.0 with attribution in NOTICE.

The difference in approach is what each project puts at the center. SpecForge is a general training framework for speculative decoding draft models. DeepSpec keeps that framework and adds its own algorithm, DSpark, alongside a ported DFlash design, then packages all three behind a single staged workflow with a shared data pipeline and a shared evaluation harness over nine benchmarks. The released checkpoint table is the concrete artifact of that packaging: three algorithms times four targets, each checkpoint tied to a config under config/ and to the same training data.

So the choice is not framework versus framework. If you want the broadest set of draft architectures and the upstream framework, go to SpecForge. If you specifically want DSpark, or you want the DFlash and Eagle3 variants trained under one consistent recipe so they can be compared against each other, DeepSpec is the narrower and more directly useful option. The MIT licence on DeepSpec is also more permissive than SpecForge's Apache-2.0, though the adapted files retain their own attribution requirements.

Licence, attribution and upgrade cost

DeepSpec is released under the MIT License, and the repository carries a NOTICE file for third-party attribution. That combination is worth reading before you redistribute anything. MIT covers the project's own code, but the README states plainly that it includes code adapted from third-party projects under their own licenses, with the full attribution in NOTICE. The Eagle3 portions come from SpecForge under Apache-2.0 and the DFlash design and recipe come from the DFlash repository under MIT. If you ship a modified copy, the NOTICE file is the document that tells you what else applies. This is a description of what the repository says, not legal advice.

Upgrade cost is dominated by pinned dependencies rather than by the project's own API. requirements.txt pins exact versions of torch, transformers, numpy and triton, among others, and the file itself warns that the default torch wheel may not match your machine. Moving to a newer transformers or torch means re-validating the training and evaluation scripts, and there is no release history retrieved to tell you how often that has been needed. The last push to the repository was on 2026-07-09, but the absence of releases means there is no changelog to read between commits.

On the checkpoint side, the upgrade path is unusually clean: draft_name_or_path accepts either a local checkpoint directory or a Hugging Face repo ID, so swapping between the released Eagle3, DFlash and DSpark checkpoints for the same target is a one-line change in the evaluation script.

Editorial conclusion

Adopt DeepSpec if you already serve a Qwen3 or Gemma target model and want to train a draft model for it, and you can afford the default 38 TB target cache or regenerate answers at a smaller scale. Do not adopt it as a drop-in inference accelerator: it has no serving path and no released runnable demo. Before committing, verify that your target model appears in the config/ directory, that your GPU count matches the single-node 8-GPU assumption, and that the target cache you plan to build fits your storage.

Frequently asked questions

Official sources

1. deepseek-ai/DeepSpec on GitHub
2. Issues
3. License: MIT
4. README