Tutel: a C MoE kernel library that also ships as a DeepSeek and Kimi serving container

The problem Tutel addresses is MoE switching cost, not raw matmul throughput

Dense transformer parallelism is a solved shape problem. Mixture-of-experts is not, because the number of active experts per token changes at runtime, and so does the amount of work each rank receives. Tutel's README describes its core claim as the first parallel solution proposing "No-penalty Parallism/Sparsity/Capacity/.. Switching" for training and inference with dynamic behaviour, and links to an MLSys 2023 slide deck as the reference. That is a specific and testable claim about the cost of changing parallelism degree, sparsity level or expert capacity mid-run. The audience is narrow: people who already run MoE models and have hit the point where re-sharding or re-tuning capacity means a restart. If your model is dense, the switching machinery has nothing to switch.

What the C code is doing and where the Python layer sits

The repository lists C as its primary language, with PyTorch among its topics. That split matches how MoE libraries usually decompose: dispatch, permutation, all-to-all and the expert compute kernels live in C or CUDA, while the module that a training script imports is a thin Python wrapper around them. The README does not spell out the module names or the C entry points, so the exact boundary between the two is something you would confirm by reading the source tree rather than from the documentation. What the README does make explicit is the quantization surface on the inference side: NVFP4, MXFP4 and BlockwiseFP8 are named as directly supported formats for MoE inference. Those are not interchangeable. NVFP4 is a block-scaled 4-bit format with a specific hardware story, MXFP4 is the OCP microscaling variant, and BlockwiseFP8 keeps 8-bit weights with block scaling. A checkpoint quantized in one of them will not silently load as another, which is why the documented commands pass explicit paths rather than a model name and a flag.

Getting it running means Docker, a Hugging Face download, and a list of --try_path entries

The README does not present a pip install as the primary path. It presents containers. Model weights come down first via huggingface_hub, for example hf download --local-dir moonshotai/Kimi-K3 moonshotai/Kimi-K3, or hf download --local-dir nvidia/DeepSeek-V3.2-NVFP4 nvidia/DeepSeek-V3.2-NVFP4. Serving is then a docker run against an image tag that encodes both the accelerator and a build date: tutelgroup/deepseek-671b:mi300x8-chat-20260902 for AMD MI300X PCIe hosts, tutelgroup/deepseek-671b:a100x8-chat-20260707 for A100, H100 and B200 SXM hosts. The flags that matter are --serve, which selects core or webui; --try_path, repeated once per candidate checkpoint so the server picks among local directories; --listen_port; --max_seq_len, shown at 200000 for the GLM-5.3-Flash example and 16384 for the long-context DeepSeek V3.2 example; and --thinking_effort high on one MI300X command. Environment variables LOCAL_SIZE and WORKER are set in every example, with LOCAL_SIZE=8 matching an eight-GPU node. The MI300X commands add --device=/dev/kfd, --device=/dev/dri and --group-add=video; the NVIDIA commands instead bind libcuda.so.1 from the host into the container and run --privileged. That asymmetry is the clearest signal that these images are built per-vendor rather than as one portable artifact.

The README's GPU table is the most useful and the least complete part of the documentation

The table compares vLLM/SGL against Tutel on named Azure GPU types. The vLLM/SGL column shows 0 t/s entries marked OoM on the MI300X rows, and 3.1 t/s on the MI325X row at MTP=0. The Tutel column shows 761 t/s for GLM-5.3 at GPU=8, BSZ=32, 108 t/s for GLM-5.3-Flash on a single GPU, and 73.4 t/s for Kimi K3 at GPU=8, with higher rows marked "xTAR" at MTP=4 and MTP=8. Two rows are explicitly left blank: AMD MI355X and NVIDIA B200, both labelled "TBD, no environment available". Treat that as the honest boundary of the claim. The numbers come from the project's own README on its own Azure hardware, with no stated batch size for several rows, no stated prompt or output length, and no methodology section. They are evidence that the authors ran something, not a benchmark you can plan capacity against. The blank rows are more informative than the filled ones: the maintainers are telling you which accelerators they have not validated.

The failure mode is that the serving path and the library path are documented to different depths

Everything concrete in the README concerns deployment: container tags, --try_path lists, --max_seq_len values, agent setup for Claude Code on Linux, WSL and Windows. The library side, which is what the repository name and the MLSys reference point at, gets one paragraph and a slide link. If you arrived wanting to integrate MoE dispatch into your own training loop, the README gives you a claim about no-penalty switching and not much else: no import examples, no configuration keys for capacity or sparsity, no statement of which PyTorch versions the C extension builds against. That is a real gap, not a stylistic one. It means the cost of evaluating Tutel as a library is reading the source, and the cost of evaluating it as a serving stack is pulling an image and running one of the documented commands. Those are very different amounts of work, and the README only makes the second one cheap. There is also a hardware constraint worth naming: the documented MI300X commands assume PCIe MI300X/MI325X parts with 192GB, and the A100 path assumes 80G SXM. A 40GB A100 or a consumer card is outside everything the README describes.

The alternative is a general inference server, and the difference is where the MoE logic lives

The README's own comparison is vLLM/SGL, and the contrast it draws is not subtle: on the MI300X rows the general servers are recorded at 0 t/s with OoM while Tutel is recorded as serving. Read that carefully rather than as a verdict. A general server such as vLLM or SGLang carries a broad model zoo, a stable OpenAI-compatible HTTP surface, continuous batching, and a plugin system for quantization. Tutel's documented surface is narrower: specific model families (GLM-5.x, DeepSeek-3.x and V3.2, Kimi-2.x and K3, Qwen3, Gpt-OSS), specific quantized checkpoints, specific accelerator images, and a --serve flag with core and webui as the options. The architectural difference is that Tutel puts the MoE dispatch and quantization kernels at the centre and treats serving as the delivery vehicle, while a general server treats MoE as one model architecture among many. If your checkpoint is not in Tutel's list, or you need the long tail of sampling parameters and API compatibility that a general server has spent years accumulating, the general server is the right call even if it needs more GPUs or a smaller context to fit. If you have an NVFP4 Kimi or DeepSeek checkpoint and eight MI300X or H100 SXM cards, the README is arguing that the general server will not start at all.

Licence, maintenance and what a version bump actually costs you

Tutel is MIT-licensed, which is permissive and places few obligations on how you redistribute or modify it. The practical licence question is not Tutel's own terms but the terms attached to the model weights you point --try_path at: the README's examples download from moonshotai, zai-org, Inferact, nvidia and modal-labs namespaces, and each of those carries its own licence that MIT does not override. On maintenance, the release history shows v0.3.2 in May 2024, then v0.4.0 and v0.4.1 in February and March 2025, with the last push to the default branch in September 2026. That pattern suggests bursts of work rather than a steady cadence, and the container tags reinforce it: the images are dated (20260707, 20260902) rather than versioned against the releases. Upgrading is therefore not a matter of moving from v0.4.1 to a later tag. It means changing the image tag, which changes the bundled kernels, the supported quantization formats and the model list together. Pin the tag you validated and treat a tag change as a re-validation, not a patch.

Editorial conclusion

Adopt Tutel if you are serving quantized MoE checkpoints (NVFP4, MXFP4, BlockwiseFP8) on MI300X, MI325X, A100 or H100 hosts and you want the serving path the README documents, with the --try_path and --max_seq_len flags driving model selection and context length. Do not adopt it as a general PyTorch training drop-in if your experts are not MoE, and do not read the GPU table as a promise for hardware it marks TBD. Before committing, verify one thing on your own hardware: that the container tag matching your accelerator (mi300x8-chat-20260902 or a100x8-chat-20260707) starts and answers a request at the --max_seq_len you need, because the README's own table leaves B200 and MI355X rows empty.

Official sources

1. Issues
2. License: MIT
3. microsoft/Tutel on GitHub
4. README
5. Releases