# Motus: one world model that switches between video generation and action prediction

> Motus is an Apache-2.0 research codebase from thu-ml that fuses a video generation model, a vision-language model and a learned action expert into a single Mixture-of-Transformers, with a UniDiffuser-style scheduler letting the same weights act as a world model, a vision-language-action model, an inverse dynamics model or a video generator. It suits a robotics research lab with the hardware to run an eight billion parameter model, and it is out of reach for anything without a datacenter GPU.

**thu-ml/Motus** — Official code of Motus: A Unified Latent Action World Model

- Repository: https://github.com/thu-ml/Motus
- Website: https://motus-robotics.github.io/motus
- Stars: 1,316 · Forks: 84
- Language: Python
- License: Apache-2.0
- Published: 2026-10-19 · Updated: 2026-10-19 · Language: en
- Canonical page: https://hysenlabs.com/projects/thu-ml-motus

## Three experts in one transformer, and a scheduler that picks the mode

Motus is described as a unified latent action world model that leverages existing pretrained models and rich, sharable motion information. The architecture is a Mixture-of-Transformers integrating three experts, one for understanding, one for action and one for video generation, and the switching mechanism is a UniDiffuser-style scheduler that allows flexible switching between different modelling modes. The README lists those modes as World Models, Vision-Language-Action Models, Inverse Dynamics Models, Video Generation Models and Video-Action Joint Prediction Models. That list is the point of the project. A single set of weights that can be asked to imagine a future frame, to say what action would produce an observed transition, or to do both, removes a family of separate models that would otherwise each need their own training run and their own deployment.

## The parameter budget is built on two public base models

The component table shows where the eight billion parameters come from. The video generation model is Wan2.2-TI2V-5B at roughly 5.00 billion parameters, the vision-language model is Qwen3-VL-2B-Instruct at roughly 2.13 billion, and the two trained components are much smaller, an action expert at about 641.5 million and an understanding expert at about 253.5 million, for a total of about 8 billion. The important consequence is that the two large blocks are pretrained checkpoints you download rather than parameters you train, and the README ships exactly those downloads. The two smaller experts are the parts that Motus actually learns, and the three-phase training pipeline is what turns the video generation checkpoint into something that also predicts actions.

## Optical flow is how the latent action is learned

Motus leverages optical flow to learn latent actions, and adopts a three-phase training pipeline and a six-layer data pyramid, which the README says extracts pixel-level delta action and enables large-scale action pretraining. The framing matters for how the model is used. Rather than predicting joint torques or end-effector poses directly from language, the intermediate representation is a delta action read off pixel motion between frames, and the flow computation is what supplies that signal during pretraining. The three-phase pipeline is visible in the checkpoint names, since stage 1 is the video generation pretrained checkpoint, stage 2 the latent action pretrained checkpoint and stage 3 the RoboTwin2 fine-tuned checkpoint.

## The reported result is a simulation number, on RoboTwin 2.0

The key result is stated for the RoboTwin 2.0 simulation, with 50 clean and 500 randomized data entries per task, and the data from all 50 tasks merged for multi-task training. The average success rate is given as 87.02 percent, described as 15 percent over X-VLA and 45 percent over a baseline written as pi-zero-point-five. The caveats are in the same line rather than elsewhere, since this is a simulation benchmark with a stated data budget, not a real robot result, and the comparison is against named systems on that benchmark. The real-world data support in the repository is separate and smaller, with AC-One and Aloha-Agilex-2 robot data listed among the supported dataset types.

## The hardware requirement is the first filter

The requirements table is short and blunt. Inference with pre-encoded T5 embeddings wants more than 24 GB of VRAM, with an RTX 5090 recommended. Inference without pre-encoded T5 wants about 41 GB, with an A100 at 40 or 80 GB, an H100 or a B200 listed as suitable. Training wants more than 80 GB, again on an A100 at 80 GB, an H100 or a B200. The gap between the 24 GB and 41 GB figures is the cost of not precomputing the text embeddings, which is a reasonable trade if you are iterating and a poor one in a batch job, and it is the kind of detail that decides whether a given team can run the thing at all.

## Installation is a conda environment and pinned wheels

The install path is a clone, an environment and a set of pins.

```bash
git clone https://github.com/thu-ml/Motus.git
cd Motus

conda create -n motus python=3.10 -y
conda activate motus

pip install torch==2.7.1 torchvision==0.22.1 --index-url https://download.pytorch.org/whl/cu128

pip install flash-attn --no-build-isolation

pip install -r requirements.txt
```

There is an optional extra for the LeRobot ecosystem, installing lerobot 0.3.2 with --no-deps and then the requirements/lerobot.txt file, which is the path for teams already working in that format. The requirements file itself is a long pinned list, with transformers at 5.0.0rc0, a release candidate, alongside diffusers, accelerate, opencv, decord for video decoding, deepspeed, wandb and tensorboard, plus the usual scientific stack. The presence of a release candidate in the pinned dependency list is worth knowing before you plan around this environment.

## Checkpoints, data formats and the docs that matter

Three checkpoints are published on Hugging Face under the motus-robotics organisation, and they map onto the training stages. Motus_Wan2_2_5B_pretrain is the stage 1 video generation checkpoint, Motus is the stage 2 latent action pretrained checkpoint, and Motus_robotwin2 is the stage 3 RoboTwin2 fine-tuned checkpoint. The foundation models themselves are downloaded too, Qwen3-VL-2B-Instruct and Wan-AI/Wan2.2-TI2V-5B, and the config paths in an embodiment config file such as configs/robotwin.yaml or configs/ac_one.yaml have to point at wherever you put them. The data side supports three dataset types, RoboTwin 2.0 with clean and randomized splits, real-world data, and the LeRobot and MultiLeRobotDataset formats, with a detailed guide in DATA_FORMAT.md, plus a converter in data/robotwin2/robotwin_data_convert and a multi-camera concatenation script. The updates list in the README is a good index of what changed in the first weeks, including LeRobotDataset support, RoboTwin raw dataset conversion and three-view image concatenation.

## Conclusion

Motus fits a robotics research group that already has multiple A100 or H100 class GPUs and wants a single set of weights that can be prompted as a world model, an action model or a video generator, because the UniDiffuser scheduler is what makes the mode switching possible rather than three separate models. It is a poor fit for a single-GPU team, since even inference with pre-encoded text embeddings wants more than 24 GB of VRAM and training wants more than 80 GB, and it is a poor fit if you need a permissive licence for a product, since Apache-2.0 covers the code but the underlying Wan and Qwen checkpoints carry their own terms. Verify first which of the three checkpoints you actually need, because the stage 1, stage 2 and stage 3 checkpoints are not interchangeable, and confirm the base model licences before you plan a release around them.

## FAQ

### What is Motus?

A unified latent action world model, an Apache-2.0 codebase from thu-ml that introduces a Mixture-of-Transformers architecture integrating understanding, action and video generation experts, with a UniDiffuser-style scheduler that lets the same weights switch between world model, vision-language-action, inverse dynamics, video generation and joint prediction modes.

### Which base models does Motus build on?

A Wan2.2-TI2V-5B video generation model at about 5.00 billion parameters and a Qwen3-VL-2B-Instruct vision-language model at about 2.13 billion, plus a trained action expert of about 641.5 million and an understanding expert of about 253.5 million, for roughly 8 billion in total.

### What hardware does Motus need?

Inference with pre-encoded T5 embeddings wants more than 24 GB of VRAM with an RTX 5090 recommended, inference without them wants about 41 GB on an A100, H100 or B200, and training wants more than 80 GB on an A100 at 80 GB, an H100 or a B200.

### Which Motus checkpoints should I download?

It depends on the stage you need. Motus_Wan2_2_5B_pretrain is the stage 1 video generation checkpoint, Motus is the stage 2 latent action pretrained checkpoint, and Motus_robotwin2 is the stage 3 RoboTwin2 fine-tuned checkpoint, all under the motus-robotics organisation on Hugging Face.

## Sources

- [Issues](https://github.com/thu-ml/Motus/issues)
- [License: Apache-2.0](https://github.com/thu-ml/Motus/blob/main/LICENSE)
- [Project website](https://motus-robotics.github.io/motus)
- [README](https://github.com/thu-ml/Motus/blob/main/README.md)
- [thu-ml/Motus on GitHub](https://github.com/thu-ml/Motus)

---

Hysen Labs editorial analysis, written from the project's own repository and release notes. Cite the canonical page: https://hysenlabs.com/projects/thu-ml-motus