What is Mixture of experts?
A mixture of experts is a neural network that keeps many parallel sub-networks, called experts, and activates only a few of them for each input token, so the model holds far more parameters than it computes with. In open source the pattern shows up as training engines, serving kernels and research inference engines that stream experts from disk.
How the router picks an expert
Mixture of experts is a conditional computation pattern. A dense layer applies the same matrix to every token, so every parameter in the network does work on every input. A mixture of experts replaces some of those layers with a bank of parallel sub-networks and puts a small gating network in front of them. The gate takes the hidden state of the current token, produces a score for each expert, and passes the token to the few experts with the highest scores, usually a small top-k selection. Their outputs are combined, most often weighted by the same gate scores. Only the selected experts run for that token, which is why descriptions of an MoE model talk about total parameters and active parameters as two different numbers.
Two properties follow. Capacity is decoupled from compute: you can raise the number of parameters the model holds without raising the arithmetic spent on each token by the same factor. And routing is data dependent, so two similar tokens can activate different experts. That is the point of the design, and it is also where most of the trouble comes from.
The pattern is not new, and it is not only for text. A learned gate in front of a bank of specialists is a general shape, and the projects below apply it to text generation, to serving kernels, to time series forecasting and to training engines.
What sparse activation does not save
The saving is arithmetic, not memory. Every expert still exists somewhere, and a decoding step has to have the weights it needs within reach. That is why memory is the hard part of running an MoE model, and why the most interesting engineering in this list is about where the bytes come from.
JustVugg/colibri makes the hierarchy explicit. Its description is running GLM-5.2, a 744B MoE model, on a consumer machine with 25GB of RAM, in pure C with no dependencies, with experts streamed from disk. Our analysis of colibri says it treats VRAM, RAM and NVMe as one hierarchy, and that it is a research platform first and a product second, which the README states.
danveloper/flash-moe is the same idea on different hardware: a C, Objective-C and Metal inference engine that streams a 397B parameter Qwen3.5 MoE model from SSD on a 48GB MacBook Pro. Our analysis calls it a research artifact with a narrow hardware target rather than a general serving stack, and the Metal dependency is the clearest sign of that.
The planning consequence is blunt. Total parameters decide how much storage and how much interconnect you need. Active parameters decide what one token costs to compute. A model can be cheap per token and still be awkward to host.
When MoE is the right shape, and when it is not
MoE earns its complexity in a narrow set of situations, and it is worth being blunt about the rest.
Where it fits. Training or serving a model at a scale where per-token arithmetic is the binding constraint and memory is available to spend. That is the case the pattern was built for: keep raising what a model knows without paying for all of it on every token.
Where it does not. Small models, where the gate's own computation and the all-to-all traffic a distributed layout needs are not paid back. Latency-sensitive single-user work, where pulling in several experts per token adds a hop a dense model does not have. Teams without the hardware or the patience to tune routing, because the pattern trades a modelling problem for a systems problem.
There is also a case for reading before building. datawhalechina/diy-llm is a VitePress course that covers pre-training data, tokenizers, Transformer, MoE, distributed training, scaling laws, inference and alignment across sixteen chapters, with six progressive coding assignments running from a hand-written tokenizer to GRPO alignment. Our analysis of diy-llm describes it as a curriculum rather than something you download and run, which is the right relationship to MoE if the term is new to you.
Limits that show up in practice
The recurring problems are documented, and they are why MoE training code is more than a layer definition.
Load balance. A gate that sends most tokens to a few experts leaves the rest untrained and wastes the parameter budget. Training engines therefore carry balancing losses and capacity controls, and the term dropless means the router may send more tokens to an expert than a fixed capacity would allow without dropping any. Our analysis of InternLM/xtuner describes XTuner V1 as targeting dropless MoE training at 200B to 1T parameters, with a documented focus on Ascend NPU optimization.
Parallelism against sparsity. Our analysis of microsoft/Tutel describes it as an optimized Mixture-of-Experts implementation for PyTorch built around the claim of no-penalty switching between parallelism, sparsity and capacity. The trade-off is real: those three goals pull in different directions, and the switching machinery is the price of moving between them.
Hardware coupling. Tutel ships as a C and CUDA extension with a container path for serving very large MoE checkpoints in FP8 and FP4 formats. XTuner's documented focus is Ascend NPUs. Our analysis of NVIDIA-NeMo/RL says it is documented around NVIDIA hardware and is not a drop-in replacement for a single-GPU fine-tuning script, so a post-training library with recipes for GRPO, DPO, PPO, SFT and on-policy distillation still arrives with a hardware assumption attached. Its release notes record added support for Qwen3.5 dense and MoE models for GRPO training.
How the term appears in open-source projects
A short map of the categories, because the same acronym covers four different kinds of work.
Inference engines that stream experts: JustVugg/colibri in pure C for a 744B MoE on a machine with 25GB of RAM, and danveloper/flash-moe in C, Objective-C and Metal for a 397B Qwen3.5 MoE on a 48GB MacBook Pro.
Serving kernels: flashinfer-ai/flashinfer provides unified APIs for attention, GEMM and MoE operations with multiple backend implementations including FlashAttention-2/3, cuDNN, CUTLASS and TensorRT-LLM. Our analysis says the pip install is one line and the architecture coverage table is where the caveats start, which is the right order in which to read it: the interface is easy, choosing between backends is the work.
MoE libraries: microsoft/Tutel for PyTorch, and InternLM/xtuner for training at 200B to 1T parameters.
Post-training: NVIDIA-NeMo/RL, which runs from one GPU to thousands with recipes for GRPO, DPO, PPO, SFT and on-policy distillation.
Other architectures and teaching material: Time-MoE/Time-MoE, a family of decoder-only time series foundation models with a Mixture of Experts architecture, published at ICLR 2025 as a Spotlight paper, scaling from 50M to 2.4B parameters, supporting zero-shot and fine-tuned forecasting of arbitrary univariate sequences and loading through the HuggingFace transformers API. And datawhalechina/diy-llm, which treats MoE as one unit among the layers of a full-stack LLM curriculum.
In practice
Mixture of experts is a way to hold more parameters than you compute with, and the whole difficulty sits in the memory system rather than the model definition. Read Tutel and XTuner for the training-side mechanics, FlashInfer for how the pattern reaches a serving stack, and colibri or flash-moe if you want to see what the memory problem looks like at 744B and 397B on ordinary hardware. Before committing to a design, decide where expert weights will live and how bytes get to the GPU, because that choice constrains the model you can host more than the routing algorithm does.