Qwen3.8-27B MTP: the flag, the cache flags, and seven rules the community mapped

The MTP head was already inside the GGUF you downloaded

Qwen trained multi-token-prediction layers, called nextn, into Qwen3.8 itself. The quantizers kept them: unsloth's GGUF builds carry `blk.*.nextn.*` tensors, so llama.cpp loads those tensors at startup and, with no flag, discards them. Support for exactly this layout arrived in llama.cpp PR #22673 in July 2026, where the server drafts candidate tokens with the head built into the weights and then verifies each draft against the main model. A draft that survives verification costs a fraction of a full forward pass, and a draft that fails costs the verification. Nothing needs converting or rebuilding, and the repository ships no weights of its own. The last commit on master is dated 2026-09-26 and the project has published no GitHub releases, so there is no tagged version to pin.

One flag, and the two cache flags it travels with

The whole change is three arguments:

--spec-type draft-mtp --spec-draft-n-max 2 --parallel 1

Dropped into an ordinary serve it reads like this:

llama-server -m Qwen3.8-27B-Q4_K_M.gguf \
  -c 131072 -ngl 999 -fa 1 \
  --cache-type-k q4_0 --cache-type-v q4_0 \
  --spec-type draft-mtp --spec-draft-n-max 2 --parallel 1

The two cache-type arguments are not part of the speed trick, and they are not optional either. Without them, context creation fails past roughly 90K tokens when the weights sit beside 17GB of VRAM. With them, the full 262K window fits on a 24GB card at 22.2GB, which is the pairing that makes a 27B model with a 262K context reachable on consumer silicon at all. If you want the widest window rather than the fastest decode, set `-c 262144` and drop the spec arguments, trading decode throughput for room. The quantized weights come from unsloth/Qwen3.8-27B-GGUF, and the unquantized model is Qwen/Qwen3.8-27B. No conversion step is involved and no second model file has to be fetched: the spec arguments read tensors that were already sitting in the file you loaded.

n-max 2 is a starting point, not a result

`--spec-draft-n-max` sets how many tokens the draft head proposes per step. The launch command uses 2, and most of the 53 recorded configurations also sit at 2, which makes 2 look like a law of physics. The sweeps say otherwise. The sweet spot moves with card, topology and workload, and one paired RTX 3090 run over JSON and tool-call prompts was still gaining at n-max 4. The procedure that falls out of the data is to sweep n-max from 2 through 4 on your own card, and to sweep representative prompts rather than a single canned one, because the mechanism behind the variation has not been pinned down. Generation length matters just as much: where per-token overhead dominates, the gain scales with how much you generate, and short answers can cost more than they return. On rigs whose baseline is already bandwidth-bound, the whole gain shows by 400 tokens and extra length adds nothing.

The p-min gate helps starved cards and hurts fast ones

`--spec-draft-p-min` puts a confidence floor on accepting a drafted token. The band contributors converged on is 0.60 to 0.75, and what it does depends entirely on which side of the memory bus you sit on. On bandwidth-poor rigs a gate near 0.60 makes deep drafting nearly free, because weak drafts get discarded before they cost a forward pass. On desktop Blackwell the same gate inverted the result: three separate RTX 5090s ran fastest with no gate at all. The reason this is easy to get backwards is that acceptance rate moves the wrong way. Gating raised acceptance and lowered throughput at the same time, which makes the acceptance column in the results table look like progress. Treat it as a vanity metric, sweep the gate on your own card, and do not adopt the community setting because it paid off on a different machine. Every contributor's full sweep is kept under `sweeps/` rather than summarized to one number.

Tensor split has to be fixed before any spec flag

llama.cpp defaults to `--split-mode layer`, which serializes decode across the cards in a box. On a pair of 5060 Ti cards, moving to `--split-mode tensor` was worth +68% on its own, before a single speculative decoding argument was added, and the two levers together reached 3.14x. The multi-GPU rows use tensor parallel as well: a 3090 paired with a 3090 Ti went from 49.1 to 81.1, and a two-card RX 9070 Vulkan run on 16GB parts went from 22.1 to 41.6. Acceptance on the 3090 pair ranged from 0.52 to 0.96 across the sweep, another reason to read throughput and ignore the acceptance column. Because the split-mode change is that large, tuning spec flags first on a multi-GPU box produces numbers that cannot be compared to anything. Reorder the work: set the split mode, re-measure the baseline, then sweep the spec arguments on top of it.

A parallel 2 baseline reads 20% low

Speculative decoding is a single-stream optimization, and the measurement protocol has to match that. The advantage is gone by `--parallel 4`. For anyone reproducing a table row there is a sharper trap: a baseline served at `--parallel 2` reads about 20% low, so a flagged run measured against a parallel-2 baseline inflates the gain by construction, and a launch line copied straight out of the launch command with `--parallel 1` is the fix rather than the problem. Both arms of the comparison have to be served at `--parallel 1`, one without the spec arguments and one with them, on the same binary and the same prompts. The complication is that a production host may genuinely want more than one stream, and the project is explicit that some hosts do serve with higher parallel. That is a serving decision, not a measurement decision. Pick the concurrency your users need and the concurrency your measurement needs separately, then do not quote a decode rate from one while implying it describes the other.

Rebuild llama.cpp, then bench headless

Two failure modes produce wrong numbers without producing any error. The first is a stale binary. Upstream is optimizing this architecture weekly: a current build came in 10 to 15% faster on every quant before a single flag was set, and a fresh RTX 3090 baseline now equals the day-one with-flag figure. A configuration that paid off on a bandwidth-starved APU inverted below baseline on a 960 GB/s card. Rebuild first, then re-derive the entire flag stack for your card class. The second failure is a shared desktop. With a live compositor and a browser open, 3.5GB of weights spilled to host RAM over PCIe while the health endpoint stayed green and decode halved with nothing in the log. Bench headless, or confirm the weights are actually resident before believing a number: `mem_info_gtt_used` on AMD, or the equivalent your vendor exposes. The failure mode here is a silent halving, not a crash.

probe.py, serve_mtp.sh, and what lives in sweeps/

The repository is small: a README, a license, a contribution guide, and three working pieces. `probe.py` is the streaming client that produced every published number. It clocks each generated token, reasoning and content deltas both, against a live server, and prints per-prompt medians rather than one average:

python3 probe.py                 # defaults to http://127.0.0.1:8080
python3 probe.py http://127.0.0.1:8090

`serve_mtp.sh` is the serve recipe as a script, and `sweeps/` holds each contributor's full parameter sweep rather than a summary row. The method is deliberately plain: run the client against a baseline serve, run it again with the flag, change nothing else. Two caveats travel with the numbers. Prompt processing takes a small hit from device-to-host embedding transfers. And the recorded figures are day-one speeds through the young qwen3_5 code path, whose hybrid attention kernels are still early, so the floor should rise as upstream work lands. Read the deltas as the durable part, and your own absolute tokens per second as the part that moves with hardware, drivers and thermals.

Editorial conclusion

qwen38-mtp earns its place if you serve Qwen3.8-27B locally on hardware you control and can rebuild llama.cpp before you start measuring. Leave it alone if you depend on concurrent streams, if you are locked to a stock binary, or if you want a single number that transfers to your card. Before trusting any row of the table, including your own, confirm the weights stayed resident, that both arms ran at --parallel 1, and that the binary is newer than the baseline you are comparing against.

Frequently asked questions

Official sources

1. Issues
2. License: Apache-2.0
3. README
4. sudoingX/qwen38-mtp on GitHub