Uni-MoE: A Mixture of Experts Family of Omnimodal Models From the Lychee Series

A Family of Models Rather Than a Single Checkpoint

Uni-MoE is described as a MoE-based omnimodal large model that can understand and generate omnimodalities, and it is part of the Lychee model family. The repository is a home for several distinct models rather than one codebase, and the directory listing shows the branches of the family side by side: Uni-MoE-2/, Uni-MoE-2d5/, Uni-MoE-TTS/, UniMoE-Audio/, Uni_MoE/, VideoVista/ and DCMoE/.

The three published generations are the ones the README links first, and each has its own weights, project page and paper:

Uni-MoE 2.0
Uni-MoE-Audio
Uni-MoE

The weight locations differ too. The 2.0 collection is on Hugging Face as a collection under the HIT-TMG organisation, the audio preview model sits under a different account, foggyforest, and the original Uni-MoE is at the plain Uni-MoE repository name. A ModelScope entry for Uni-MoE v1.5 also exists, under a contributor account.

The README describes the project as a fully open-source omnimodal large model, and the openness claim is specific: not only the weights and the inference code, but the training scripts, the distributed MoE modules and an evaluation benchmark are published, which is the part that makes it useful to other groups.

The Mixture of Experts Design and What It Means in Practice

The architecture is a mixture of experts, and the 2.0 generation describes three core contributions: a dynamic-capacity Mixture-of-Experts design, a progressive training strategy enhanced with an iterative reinforcement strategy.

The dynamic capacity part is the interesting engineering claim. A conventional sparse MoE routes each token to a fixed number of experts, and the router decides which. A dynamic capacity design lets the number of activated experts vary per token, which means the compute spent on a token depends on how hard that token is. For a model that has to handle ten-plus modalities in one architecture, that is the difference between spending model capacity uniformly and spending it where the input demands.

The 2.0 Omni model is built on the Qwen2.5-7B dense architecture, trained from scratch, so the point of comparison is a dense 7B model rather than a larger sparse one.

The cost side is the usual one for sparse models. The weights are large, the routing has to be efficient to be worth anything, and the training stack in this repository, which we will come to, assumes a cluster rather than a workstation. The README reports a significant leap in language-centric multimodal understanding, reasoning and generation capabilities, and says the model efficiently supports cross-modal interactions across ten-plus modalities such as images, text and speech.

Uni-MoE-Audio: One Model for Speech and Music

The audio branch is a separate line of work rather than a modality bolted onto the language model. The README describes the Uni-MoE-Audio release as the first audio generation model with a unified speech and music generation.

Unified is the operative word, since a model that does both usually has a compromise, and the claim here is that a single model handles speech and music in the same framework. The audio model has its own project page, its own paper on arXiv at 2510.13344, and its weights published separately as a preview model.

There is also a Uni-MoE-TTS/ directory in the repository, which suggests the text to speech side is handled as its own component, and the v1.5 release from August 2025 is described as having a unified speech encoding approach, published on ModelScope. So the family has iterated on the speech side separately from the main omni model, and the direction of travel has been toward unification within the audio domain rather than bolting audio onto the general model.

For a reader, the practical point is that the audio work is separable. You can take the audio model and the audio paper without touching the language-centric omni model, and the repository layout supports that by keeping them in different directories.

The Training Stack Assumes Multi-Node Clusters

The April 2024 update is the one that tells you what the training code is for. The codebase was upgraded to facilitate training across multiple nodes and GPUs, with a revamped fine-tuning script named finetune_speech_dp.sh, and a version that integrates distributed MoE modules was introduced.

The stated benefit of that distributed version is parallel processing at both the expert and modality levels, for efficiency and scalability. Expert level parallelism is the standard way to train a sparse model, spreading experts across devices, and modality level parallelism is the unusual half, since it lets a model that handles ten-plus modalities process different modalities on different devices at the same time. That is the natural answer to the shape of this problem, where a single forward pass may carry text, image and speech tokens at once.

The 8-expert checkpoint became available for download and inference in May 2024, with details in the Uni_MoE_8e documentation, and the documentation for the 1.0 model with 8 experts covers the distributed variant. The DCMoE/ directory sits in the repository root alongside the model directories.

The practical implication is straightforward. This is a codebase for people with a cluster. A single GPU will not train these models, and fine-tuning the full omni model is not a weekend project.

Evaluation Code and a Published Video Benchmark

Two pieces of the release are about measurement rather than capability.

The first is the integration with the Lmms-eval framework, announced on 2025/11/24, which makes the Uni-MoE-2.0-Omni model available for evaluation inside an existing benchmark harness rather than through a bespoke script. For a research group, that is the difference between a comparable number and a number you have to explain.

The second is VideoVista, released on 2024/8/28, which is both a video evaluation benchmark and a source of automatically generated video instruction tuning data in the VideoVista_Train dataset on Hugging Face. The dual purpose is the interesting part: a benchmark that is also a training data generator, so the same pipeline can produce the evaluation set and the fine-tuning corpus.

The VideoVista/ directory in the repository holds that work, and the benchmark has its own site at videovista.github.io.

The project's own publication record adds one more data point. The paper was accepted by IEEE Transactions on Pattern Analysis and Machine Intelligence in 2025, which is a peer-reviewed venue and a stronger signal than a preprint alone.

What the Repository Actually Holds

The repository is a collection of model directories and assets, and the top level gives a good idea of the scope: DCMoE/, Uni-MoE-2/, Uni-MoE-2d5/, Uni-MoE-TTS/, UniMoE-Audio/, Uni_MoE/, VideoVista/ and assets/.

The naming inconsistency is worth noticing, because it reflects how the work accumulated. There is Uni-MoE-2/ and Uni-MoE-2d5/, with the latter presumably the 2.5 variant, alongside the older Uni_MoE/ with an underscore from the earlier repository layout. The Audio and TTS directories are named without the hyphen. So there are three or four code generations in one tree, and the README's links are the reliable way to find the right one for a given model.

The two linked repositories are worth noting as well. The news items point at UMOE-Scaling-Unified-Multimodal-LLMs for the 8-expert weights documentation and the fine-tuning script, and the Uni-MoE-2 directory holds the current generation, so some of the training material lives in a sibling repository rather than in this one.

The project is not archived, the last push was on 2026-08-06, and there are no GitHub releases, so the versioning of this work happens through the model weights on Hugging Face and the papers, not through tagged source releases.

Editorial conclusion

Uni-MoE fits a research group that wants to study or fine-tune a large mixture of experts model that handles text, image, speech and video in one architecture, and to compare against a published evaluation setup rather than assembling their own. It is a poor fit for a production deployment, since the checkpoints are research artefacts released for study and the training stack assumes multi-node GPU clusters. Before starting, check the hardware you have against what the 8-expert checkpoint and the multi-node fine-tuning path need, and decide which generation you want, because the 2.0 Omni model, the 2.5 variant, the audio model, the TTS directory and the original code all live side by side in the repository. The project is not archived, the last push was on 2026-08-06, and no GitHub releases are published, so the model weights are tracked on Hugging Face rather than through versioned releases.

Frequently asked questions

Official sources

1. HITsz-TMG/Uni-MoE on GitHub
2. Issues
3. Project website
4. README