MTPLX: native MTP speculative decoding for Qwen 3.8 on Apple Silicon

What MTPLX is for, and who it is not for

MTPLX is a local inference engine for Apple Silicon that runs Qwen 3.8 Flash Next, the 125B mixture of experts, and Qwen 3.8 27B, plus Qwen 3.6, Qwen 3.5 and Gemma 4. It ships as a native Mac app and as a command line. The pitch is narrow and specific: use the model's own multi-token prediction (MTP) heads as a speculative decoder, so you get faster decoding without swapping in a second draft model or changing the output distribution.

That framing tells you who it is for. If you are on an Apple Silicon Mac, you want a local model behind an OpenAI or Anthropic compatible API, and you care about decode throughput on long coding sessions, this is aimed at you. The README recommends Qwen 3.8 27B Optimized Speed for coding and names Qwen 3.8 Flash Next Optimized Speed as the pick on a Mac with 96 GB or more.

It is not a general-purpose serving stack. There is no CUDA path, no Linux container, no Windows build in the documentation. The requirements are Apple Silicon (M1 or newer) and macOS 14+. If your fleet is x86 or your deployment target is a Linux box, this project is the wrong tool regardless of how the numbers look. The memory floor also rules people out: 16 GB runs the 4B and 9B models comfortably, but the models the README leads with are much larger.

How the MTP speculative decoder actually works

The mechanism, as the README describes it: the model drafts several tokens ahead of itself, one batched forward pass verifies the draft, and tokens are committed through exact rejection sampling with residual correction. The acceptance rule is the Leviathan and Chen rejection sampling theorem with residual correction. That is the part worth reading twice. Speculative decoding normally risks changing the sampling distribution, and the README is explicit that this implementation does not: `temperature=0.6, top_p=0.95` should behave like normal decoding, just faster.

The verification claim is stated concretely. On every release the project draws a thousand four-token samples from the fast path and a thousand from the plain path at temperature 1, top-p 0.95, top-k 20, and compares the two distributions by token id. The README says that on 2.11.3 they match within the plain path's own noise on both Flash Next and the 27B Quality pack. That is a self-reported test, not an independent audit, and the README does not describe the statistical threshold used for "within noise". Treat it as a design goal the project tests against, not a third-party guarantee.

The speed claim is similarly scoped. The README reports around twice the speed of plain decoding, measured 1.6x on a 16 GB M4 Mac mini and 2.24x on an M5 Max. The headline table is measured on a MacBook Pro M5 Max with 128 GB, and reports 125.8 tok/s for Qwen 3.8 Flash Next on one OpenCode request with MTP depth 3. Every row carries a version and a date, which is more than most projects of this kind publish. None of those numbers transfer automatically to your machine; the app measures your hardware to pick a decoding depth, which is an admission that the right depth is machine-specific.

Installing MTPLX and pointing a client at port 8000

The README gives two install paths. The Mac app is described as the easiest: download the DMG, drag it to Applications, and the app checks your hardware, recommends a model that fits your memory, downloads it, sets up its own Python engine, installs fan control, puts `mtplx` on your PATH, and measures your machine to pick a decoding depth. If you want the CLI on its own, the README shows Homebrew first:

brew install youssofal/mtplx/mtplx
mtplx start

Alternatively, `python3 -m pip install mtplx` if you prefer pip. The package requires Python 3.11 or newer according to pyproject.toml, and the mlx dependency is pinned to `mlx>=0.32.2,<0.33` for darwin on arm64.

To serve a specific model from the terminal, the README gives this for the Flash Next pack:

mtplx serve --model Youssofal/Qwen3.8-Flash-Next-MTPLX-Optimized-Speed

After that, point any client that speaks the OpenAI or Anthropic API at `http://127.0.0.1:8000`. The README names OpenCode, Pi, Hermes, Claude Code, Cline and Cursor. The repository ships worked examples for both API shapes, including `examples/curl-chat-completions.sh`, `examples/curl-messages.sh`, `examples/openai-python-client.py` and `examples/anthropic-python-client.py`, so you can confirm the endpoint answers before wiring up an editor. What you should see is a local server on port 8000 answering chat completions and messages requests; the README does not document a health check endpoint, so the curl examples are the practical way to verify it is up.

The exactness claim is the interesting part, and the weakest part

Most speculative decoding writeups sell speed and quietly note a quality caveat. MTPLX inverts that: the README argues the output is unchanged, and the speed is the bonus. The argument rests on rejection sampling with residual correction, which is a sound theoretical basis for exactness when implemented correctly. The gap is between theory and the evidence offered. The evidence is an internal distribution comparison run on every release, plus a note that the 27B Optimized Speed pack agrees with the bf16 model on 96.0 percent of top-1 tokens on a mixed corpus of code, prose and JSON. Ninety-six percent top-1 agreement is not the same claim as exactness, and the README presents both. A reader who needs bit-exact reproducibility should not read the 96.0 percent figure as a guarantee.

There is a second thing worth flagging. The README states there is no second draft model eating your RAM, and no greedy shortcut that quietly changes what the model would have said. That is a real architectural advantage over draft-model speculative decoding, where you pay memory for a separate small model. Here the draft capability comes from heads already in the weights. It also means the speedup depends on how well those heads predict for your particular prompt, which is why the reported figures vary so much by task: 125.8 tok/s on one OpenCode request, 61.8 tok/s on a 109k-token turn, 50.3 tok/s on a 200k-token turn. Long context is where the advantage narrows.

Memory, context and the Flash Next n-gram table

Qwen 3.8 Flash Next is a 125B-A6B preview of the Qwen4 architecture, described as a hybrid GatedDeltaNet mixture of experts with Qwen Sparse Attention and a 51B-parameter n-gram table. Two packs exist, both for Macs with 96 GB of unified memory or more. Optimized Speed is dynamic 4-bit with sparse-attention projections at 8-bit: 115.1 GB download including the 32 GB n-gram table, about 83 GB resident. Bare Speed is flat 4-bit: 106.3 GB download, about 74 GB resident.

The n-gram table streams from SSD by default, so the weights stay resident and the table does not have to. That is a sensible design decision for a machine where unified memory is the binding constraint, but it makes SSD speed part of your inference path. The README does not state what happens when the SSD cannot keep up, and it does not document a way to force the table fully resident. Context window is 262,144 tokens, and the README says 261,120-token prompts decode on 2.11.3, which is a near-full-window claim rather than a theoretical maximum.

For Qwen 3.8 27B the numbers are friendlier. Optimized Speed is 4-bit dynamic, 20.4 GB, 23.6 GB peak, and is the default for coding. There are three packs, each with an FP16 sibling for M1 and M2 that the app and CLI pick automatically. That automatic FP16 selection on older chips is a detail worth knowing: on an M1 or M2 you are not running the same build as the benchmarks, even though the README calls the weights the same.

MTPLX compared with mlx-lm and Ollama

The README links a comparison page covering mlx-serve, oMLX, LM Studio, Ollama, llama.cpp and mlx-lm, with a version, a machine and a date on every number. The relevant difference is architectural. mlx-lm is Apple's general MLX language model library: it loads and runs a wide range of architectures and does not, on its own, implement MTP speculative decoding for these Qwen models. MTPLX is narrower and deeper, built around the MTP heads of a specific model family.

Ollama takes the opposite approach: a model registry, a simple pull-and-run workflow, and broad hardware support including non-Apple platforms. If you need one command to run a quantized model on a Linux server, Ollama is the answer and MTPLX is not in the conversation. If you need the fastest decode for Qwen 3.8 on an M-series Mac specifically, the trade is different: MTPLX accepts a narrower model list and a Mac-only constraint in exchange for a decoding path the README claims is roughly twice as fast as plain decoding.

llama.cpp is the portability play, running across CPU, CUDA, Metal and more. MTPLX's bet is that a Metal-native implementation with model-specific MTP support beats a portable one on Apple Silicon for these particular models. That bet is plausible and the README supports it with dated measurements, but it is a bet you can only evaluate on your own hardware.

Maintenance, dependencies and licence

The repository is not archived, and the last push was on 2026-09-17. Release v2.11.3 is dated 2026-09-17, v2.11.2 is dated 2026-09-06, and v2.11.1 is dated 2026-09-04. That is a fast release cadence, and the CHANGELOG.md and HISTORY.md files at the repository root suggest the project keeps a written record of it.

The upgrade cost is not zero, and pyproject.toml documents why. The mlx floor was raised to 0.32.2 in 2.9.3 after a clean shipped-wheel A/B on an M5 Max measured 0.32.0 to 0.32.2 at +29% decode and +41% prefill at 88.4k tokens, with the same wheel and the same flags and only mlx differing. The comment notes that a 0.32.0 virtualenv silently forfeits that gain. The dependency range is `mlx>=0.32.2,<0.33` and `mlx-lm>=0.31,<0.32`, both gated to darwin on arm64. The upper bounds mean an mlx 0.33 release will require an MTPLX update before you can move, which is a deliberate stability choice with a maintenance cost attached.

Licence is Apache-2.0, and pyproject.toml declares `license-files = ["LICENSE", "NOTICE"]`, so a NOTICE file ships with the distribution. Apache-2.0 is permissive and includes a patent grant, but it also carries attribution and notice-retention obligations, and the model weights MTPLX downloads are separate artifacts under their own terms. Check the licence attached to the Hugging Face packs you pull; the repository licence does not automatically cover them. This is not legal advice.

Editorial conclusion

Adopt MTPLX if you have an Apple Silicon Mac with 32 GB or more and you want Qwen 3.8 served locally behind an OpenAI or Anthropic compatible endpoint, because that is the exact configuration the README documents and benchmarks. Do not adopt it if you are on Intel, Linux, Windows, or a Mac below the memory floor for the pack you want: the 27B Optimized Speed pack is 20.4 GB on disk with 23.6 GB peak, and Flash Next needs 96 GB or more. Before committing, verify the pinned mlx range in pyproject.toml resolves cleanly on your machine, check TROUBLESHOOTING.md for your failure mode, and confirm the license file and NOTICE shipped with the wheel.

Frequently asked questions

Official sources

1. License: Apache-2.0
2. Project website
3. README
4. Releases
5. youssofal/MTPLX on GitHub