Tutel: a C MoE kernel library that also ships as a DeepSeek and Kimi serving container
Tutel MoE: Optimized Mixture-of-Experts Library, Support GptOss/DeepSeek/Kimi-K2/Qwen3 using FP8/NVFP4/MXFP4
At a glance
- What is it?
- Microsoft's Tutel is an MIT-licensed mixture-of-experts library written primarily in C, and its README now doubles as an operations manual for running NVFP4 and MXFP4 MoE checkpoints on MI300X, A100 and H100 hosts. The interesting question is which of those two things you are actually adopting.
- Who is it for?
- Adopt Tutel if you are serving quantized MoE checkpoints (NVFP4, MXFP4, BlockwiseFP8) on MI300X, MI325X, A100 or H100 hosts and you want the serving path the README documents, with the --try_path and --max_seq_len flags driving model selection and context length. Do not adopt it as a general PyTorch training drop-in if your experts are not MoE, and do not read the GPU table as a promise for hardware it marks TBD.
- Can I use it commercially?
- Yes. MIT is a permissive licence: you can use, modify and sell software built on it, as long as you keep its copyright and licence notices.
- Is it still maintained?
- Yes. The repository last received commits 9 days ago.
- What is it written in?
- Mainly C, according to GitHub's language statistics.
Answers come from the project's GitHub data, last synced on November 15, 2026, and from our analysis. They are not legal advice.
Star history
- microsoft/Tutel1,028 ★ · +10
| Repository | 2026-09-10 | 2026-11-15 |
|---|---|---|
| microsoft/Tutel | 1k | 1k |
10 Stars in the last 66 days, measured from the daily GitHub snapshot. Compare with other projects
Editorial analysis
The problem Tutel addresses is MoE switching cost, not raw matmul throughput
Dense transformer parallelism is a solved shape problem. Mixture-of-experts is not, because the number of active experts per token changes at runtime, and so does the amount of work each rank receives. Tutel's README describes its core claim as the first parallel solution proposing "No-penalty Parallism/Sparsity/Capacity/.. Switching" for training and inference with dynamic behaviour, and links to an MLSys 2023 slide deck as the reference. That is a specific and testable claim about the cost of changing parallelism degree, sparsity level or expert capacity mid-run. The audience is narrow: people who already run MoE models and have hit the point where re-sharding or re-tuning capacity means a restart. If your model is dense, the switching machinery has nothing to switch.
What the C code is doing and where the Python layer sits
The repository lists C as its primary language, with PyTorch among its topics. That split matches how MoE libraries usually decompose: dispatch, permutation, all-to-all and the expert compute kernels live in C or CUDA, while the module that a training script imports is a thin Python wrapper around them. The README does not spell out the module names or the C entry points, so the exact boundary between the two is something you would confirm by reading the source tree rather than from the documentation. What the README does make explicit is the quantization surface on the inference side: NVFP4, MXFP4 and BlockwiseFP8 are named as directly supported formats for MoE inference. Those are not interchangeable. NVFP4 is a block-scaled 4-bit format with a specific hardware story, MXFP4 is the OCP microscaling variant, and BlockwiseFP8 keeps 8-bit weights with block scaling. A checkpoint quantized in one of them will not silently load as another, which is why the documented commands pass explicit paths rather than a model name and a flag.
Getting it running means Docker, a Hugging Face download, and a list of --try_path entries
The README does not present a pip install as the primary path. It presents containers. Model weights come down first via huggingface_hub, for example hf download --local-dir moonshotai/Kimi-K3 moonshotai/Kimi-K3, or hf download --local-dir nvidia/DeepSeek-V3.2-NVFP4 nvidia/DeepSeek-V3.2-NVFP4. Serving is then a docker run against an image tag that encodes both the accelerator and a build date: tutelgroup/deepseek-671b:mi300x8-chat-20260902 for AMD MI300X PCIe hosts, tutelgroup/deepseek-671b:a100x8-chat-20260707 for A100, H100 and B200 SXM hosts. The flags that matter are --serve, which selects core or webui; --try_path, repeated once per candidate checkpoint so the server picks among local directories; --listen_port; --max_seq_len, shown at 200000 for the GLM-5.3-Flash example and 16384 for the long-context DeepSeek V3.2 example; and --thinking_effort high on one MI300X command. Environment variables LOCAL_SIZE and WORKER are set in every example, with LOCAL_SIZE=8 matching an eight-GPU node. The MI300X commands add --device=/dev/kfd, --device=/dev/dri and --group-add=video; the NVIDIA commands instead bind libcuda.so.1 from the host into the container and run --privileged. That asymmetry is the clearest signal that these images are built per-vendor rather than as one portable artifact.
The README's GPU table is the most useful and the least complete part of the documentation
The table compares vLLM/SGL against Tutel on named Azure GPU types. The vLLM/SGL column shows 0 t/s entries marked OoM on the MI300X rows, and 3.1 t/s on the MI325X row at MTP=0. The Tutel column shows 761 t/s for GLM-5.3 at GPU=8, BSZ=32, 108 t/s for GLM-5.3-Flash on a single GPU, and 73.4 t/s for Kimi K3 at GPU=8, with higher rows marked "xTAR" at MTP=4 and MTP=8. Two rows are explicitly left blank: AMD MI355X and NVIDIA B200, both labelled "TBD, no environment available". Treat that as the honest boundary of the claim. The numbers come from the project's own README on its own Azure hardware, with no stated batch size for several rows, no stated prompt or output length, and no methodology section. They are evidence that the authors ran something, not a benchmark you can plan capacity against. The blank rows are more informative than the filled ones: the maintainers are telling you which accelerators they have not validated.
The failure mode is that the serving path and the library path are documented to different depths
Everything concrete in the README concerns deployment: container tags, --try_path lists, --max_seq_len values, agent setup for Claude Code on Linux, WSL and Windows. The library side, which is what the repository name and the MLSys reference point at, gets one paragraph and a slide link. If you arrived wanting to integrate MoE dispatch into your own training loop, the README gives you a claim about no-penalty switching and not much else: no import examples, no configuration keys for capacity or sparsity, no statement of which PyTorch versions the C extension builds against. That is a real gap, not a stylistic one. It means the cost of evaluating Tutel as a library is reading the source, and the cost of evaluating it as a serving stack is pulling an image and running one of the documented commands. Those are very different amounts of work, and the README only makes the second one cheap. There is also a hardware constraint worth naming: the documented MI300X commands assume PCIe MI300X/MI325X parts with 192GB, and the A100 path assumes 80G SXM. A 40GB A100 or a consumer card is outside everything the README describes.
The alternative is a general inference server, and the difference is where the MoE logic lives
The README's own comparison is vLLM/SGL, and the contrast it draws is not subtle: on the MI300X rows the general servers are recorded at 0 t/s with OoM while Tutel is recorded as serving. Read that carefully rather than as a verdict. A general server such as vLLM or SGLang carries a broad model zoo, a stable OpenAI-compatible HTTP surface, continuous batching, and a plugin system for quantization. Tutel's documented surface is narrower: specific model families (GLM-5.x, DeepSeek-3.x and V3.2, Kimi-2.x and K3, Qwen3, Gpt-OSS), specific quantized checkpoints, specific accelerator images, and a --serve flag with core and webui as the options. The architectural difference is that Tutel puts the MoE dispatch and quantization kernels at the centre and treats serving as the delivery vehicle, while a general server treats MoE as one model architecture among many. If your checkpoint is not in Tutel's list, or you need the long tail of sampling parameters and API compatibility that a general server has spent years accumulating, the general server is the right call even if it needs more GPUs or a smaller context to fit. If you have an NVFP4 Kimi or DeepSeek checkpoint and eight MI300X or H100 SXM cards, the README is arguing that the general server will not start at all.
Licence, maintenance and what a version bump actually costs you
Tutel is MIT-licensed, which is permissive and places few obligations on how you redistribute or modify it. The practical licence question is not Tutel's own terms but the terms attached to the model weights you point --try_path at: the README's examples download from moonshotai, zai-org, Inferact, nvidia and modal-labs namespaces, and each of those carries its own licence that MIT does not override. On maintenance, the release history shows v0.3.2 in May 2024, then v0.4.0 and v0.4.1 in February and March 2025, with the last push to the default branch in September 2026. That pattern suggests bursts of work rather than a steady cadence, and the container tags reinforce it: the images are dated (20260707, 20260902) rather than versioned against the releases. Upgrading is therefore not a matter of moving from v0.4.1 to a later tag. It means changing the image tag, which changes the bundled kernels, the supported quantization formats and the model list together. Pin the tag you validated and treat a tag change as a re-validation, not a patch.
Editorial conclusion
Adopt Tutel if you are serving quantized MoE checkpoints (NVFP4, MXFP4, BlockwiseFP8) on MI300X, MI325X, A100 or H100 hosts and you want the serving path the README documents, with the --try_path and --max_seq_len flags driving model selection and context length. Do not adopt it as a general PyTorch training drop-in if your experts are not MoE, and do not read the GPU table as a promise for hardware it marks TBD. Before committing, verify one thing on your own hardware: that the container tag matching your accelerator (mi300x8-chat-20260902 or a100x8-chat-20260707) starts and answers a request at the --max_seq_len you need, because the README's own table leaves B200 and MI355X rows empty.
Frequently asked questions
What is the newest released version of Tutel?
v0.4.1, published on 2025-03-20, with v0.4.0 from February 2025 and v0.3.2 from May 2024 before it. Changes since then are listed as dated container image tags, the newest visible being 20260902, and the last push to the default branch was 2026-09-15.
Which precisions does Tutel support for inference?
Direct NVFP4, MXFP4 and blockwise FP8 inference for mixture-of-experts models including GLM-5.x, DeepSeek-3.x, Kimi-2.x and 3.x, Qwen3 and gpt-oss, on A100, A800, H100 and MI300 class hardware.
What does the Tutel benchmark table report against vLLM and SGL?
Most rows list vLLM and SGL at 0 tokens per second with out of memory, against Tutel figures such as 761 tokens per second for GLM-5.3 on eight MI300X. On MI355X and the NVIDIA B200 the comparison column has numbers and the Tutel column is marked as having no environment available.
What does the documented Claude Code setup do to the client?
It points Claude Code at a plaintext endpoint on port 8000 with a placeholder API key, and disables the updater, sets the API timeout to the maximum value, and turns off the stream watchdog, the byte watchdog, its Bedrock variant and the async agent stall timeout.
How is the Tutel extension built?
setup.py imports torch and the torch cpp_extension build helpers at build time, reads the HIP extension flag out of torch inside a bare except clause, and branches per platform: Linux gets a linker flag, Darwin a minimum macOS version of 10.13. With CUDA off it forces NCCL off and builds a plain C++ extension.
Official sources
Add this badge to your README
If you maintain this project, the badge below links readers to this analysis and shows its maintenance status from the daily GitHub snapshot. Paste the markdown into your README; add ?metric=license or ?metric=stars to the image URL for a different field.
[](https://hysenlabs.com/projects/microsoft-tutel)Star history chart for your README
[](https://hysenlabs.com/projects/microsoft-tutel)Was this useful?