# MIRA: a 5B-parameter Rocket League world model you can play inside

> MIRA generates a 2v2 Rocket League match frame by frame from four players' keyboard actions, and the repository ships the codec, the world model, the training scripts and a 4-player dataset loader. Here is how the pieces fit, what the install actually requires, and where it stops.

**mira-wm/mira** — Code for MIRA: Multiplayer Interactive World Models with Representation Autoencoders

- Repository: https://github.com/mira-wm/mira
- Website: https://mira-wm.com/
- Stars: 574 · Forks: 34
- Language: Python
- License: Apache-2.0
- Published: 2026-09-17 · Updated: 2026-09-17 · Language: en
- Canonical page: https://hysenlabs.com/projects/mira-wm-mira

## What MIRA is for, and who it is actually aimed at

MIRA is a world model of one specific game. The README describes it as a real-time world model of Rocket League in which a 5B-parameter latent diffusion model generates the video frame by frame from all four players' actions, so a full 2v2 match can be played inside the model at 20 FPS on a single GPU. The published artifact is the official code release for a technical report, a collaboration between General Intuition, Kyutai and Epic Games, and it is Apache-2.0 licensed.

The audience is narrow and worth stating plainly. This is not a library you import to add video prediction to an application. It is a research release: a codec, a world model, a training pipeline, an offline evaluation script and a dataset loader, arranged so that someone with a GPU and a reason to reproduce or extend the work can do so. The repository layout confirms that shape. There is a configs directory, a scripts directory, a src directory and a tests directory, plus pixi.toml and pixi.lock for environment management. If you are looking for a hosted playground, the README points at mira-wm.com for live play and treats the repository as the code behind it.

The dataset is a second product of the release. Each sample is a four-second window of a 2v2 match captured from all four players' synchronised views, and for every frame of every view it provides the video frame, the player's keyboard action and the game state covering ball, cars and score. That combination, four time-aligned viewpoints plus per-frame input plus physics, is rarer than the model itself and is the part most likely to outlive this particular checkpoint.

## How the codec, the world model and the 4-player wrapper fit together

The architecture is staged, and the training scripts make the staging explicit. There are three entry points under scripts: train_codec.py, train_world_model.py and eval_world_model_offline.py. The codec is trained first. The single-player world model is trained second and takes a codec_checkpoint. The 4-player world model is trained third, warm-started from a single-player checkpoint through run.finetune_from and pointed at the same codec checkpoint through model.architecture.config.wm_config.codec_checkpoint. The repository name says representation autoencoders, and the codec is where that representation is learned.

The codec stage depends on an external encoder. Codec training uses a frozen DINOv3-L/16 encoder whose weights are gated by Meta, and the README instructs you to download dinov3_vitl16_pretrain_lvd1689m-8aa4cbdd.pth and set RS_DINO_WEIGHTS_DIR to its directory. World-model training and inference do not need it. That asymmetry is the single most useful thing to know before you start: you can load, run and evaluate a world model without ever touching the gated weights, and only pay that cost if you intend to train a codec yourself.

The data flow at inference is short. Four players' keyboard actions go in, the world model produces latent frames, and the codec decodes them to video, one frame at a time, at 20 FPS. The dataset mirrors that contract in reverse: clip.frames is a (P, T, C, H, W) uint8 tensor for P=4 players and T=16 frames, clip.actions is a (P, T, 9) int32 multi-hot keyboard state aligned to each frame, clip.events lists game events overlapping the window, and clip.physics holds per-player per-frame game state. The 9-wide action vector is a keyboard encoding, not a continuous control, which is a real design commitment: it ties the model to the input device the data was captured with.

## Installing MIRA with pixi and loading a first clip

The README gives exactly two commands for setup, both through pixi, and states the hardware requirement in a comment. The environment creation is one-time and installs torch for you.

```bash
pixi run setup   # one-time: creates the environment and installs MIRA (requires an NVIDIA GPU)
pixi run test    # run the test suite
```

Run setup first, then test. The README notes that pixi is required and that torch >= 2.8 is installed for you by that step. If you do not have an NVIDIA GPU, the README's own comment tells you the setup path is not aimed at your machine.

Once the environment exists, the fastest real use is the dataset loader, which does not need a checkpoint. This snippet downloads one shard of the test split from the HuggingFace Hub and pulls the first clip of the first match.

```python
from mira.data import RocketScienceDataset

ds = RocketScienceDataset.from_hub("kyutai/rocket-science", split="test", shards=1)
clips = ds.load_match(ds.match_ids()[0], clip_len=16, target_fps=20)
clip = clips[0]
```

What you should see is a clip holding four synchronised player views over sixteen frames: clip.frames as a (P, T, C, H, W) uint8 tensor, clip.actions as (P, T, 9) int32, plus clip.events and clip.physics. The shards=1 argument is the quick look; omit it for the full split, which the README warns is the larger download. There is also an interactive path if you would rather look at the data than parse tensors.

```bash
pixi run explore
```

The README describes that as a 4-player grid with a keyboard overlay and a top-down overview of the game using the game state. For training, the scripts are Hydra apps, so any config key can be overridden as key=value, and multi-GPU runs go through torchrun.

## The dependency edges that decide whether your install works

pyproject.toml splits the dependencies into extras, and the split tells you what each stage costs. The base install is three packages: pydantic>=2.11, numpy>=1.24 and einops>=0.8, with a comment noting that the codec and world-model layers use einops and that it is needed by every install that loads a model. Everything else is optional.

The decode extra is a single package, torchcodec, and it is the sharp edge. The comment in pyproject.toml states that torchcodec needs a runtime FFmpeg whose libraries match its build, and names a known-good combination: torchcodec 0.7.0 with torch 2.8.0 and FFmpeg 7. torch itself is deliberately not pinned, because consumers provide their own, and the train and inference code targets torch>=2.8. That is a reasonable choice for a research release and an annoying one for anyone who expects a lockfile to settle the question. If video decoding fails, the version triangle is the first thing to check, not the last.

The extras also encode a deliberate exclusion. The offline world-model metrics need Frechet DINO and Inception distances, DINO loads through torch.hub, and the file comments call this heavy and eval-only and say it is kept out of the train extra on purpose. So a plain mira[train] install will not give you the evaluation metrics. The train extra itself pulls hydra-core and omegaconf, matplotlib and plotly and pillow, tqdm, wandb, lpips, rich, torchmetrics and termcolor. wandb being in that list means the default training path expects a logging account unless you override it through the Hydra config, which the README's key=value convention allows but does not spell out.

## Where MIRA is the wrong tool

Three boundaries are visible from the repository. The first is hardware. The README's setup comment requires an NVIDIA GPU, and the 20 FPS figure is stated for a single GPU without naming which one. If you have no NVIDIA GPU, or you need to run on CPU or on a laptop, this release does not offer you a path, and the environment step will not get you there.

The second is scope. MIRA models Rocket League, and the dataset is Rocket League, with a 9-wide multi-hot keyboard action vector and game state covering ball, cars and score. Nothing in the README suggests the pipeline generalises to another game or to real-world video. The architecture may be interesting to read, but adopting the code to predict frames of something other than this game means replacing the codec's training data, the action encoding and the physics fields, which is most of the work.

The third is the gap between a world model and an environment. The README documents offline evaluation through eval_world_model_offline.py against a checkpoint, and it documents live play at mira-wm.com. What it does not document is a reinforcement-learning interface, a reward function, or a rollout API. If your goal is to train agents against a learned simulator, you would be building that layer yourself on top of inference, and the README is silent on what that costs. It is also silent on rollback, checkpoint compatibility across versions, and what happens to a run when the model drifts over long rollouts. Those are the questions a research release usually leaves open, and this one does.

## How MIRA differs from a single-player world model

The obvious comparison is a single-view, single-player world model of the kind that has become common: one camera, one action stream, one predicted view. MIRA's own repository contains that baseline, because the training sequence goes through a single-player world model before the 4-player one, and the 4-player run is warm-started from it via run.finetune_from with model=multi_wrapper_world_model and dataset.n_players=4. That is the cleanest statement of the difference in approach: the multiplayer model is not a separate architecture trained from scratch, it is a wrapper that takes a single-player model and extends it to four synchronised views.

The consequence shows up in the data contract. A single-view model needs one frame stream and one action stream. MIRA needs four views that are time-aligned to the same frames, four action streams, and shared game state so the views stay consistent with each other. The README's description of the dataset, all four players' synchronised views with per-frame actions and game state, is the requirement that makes the problem harder and the release more useful. Multiplayer consistency is the thing a single-player model cannot express, because it has no representation of the other players at all.

## Maintenance, licence and what an upgrade costs

The repository is not archived, and the last push was on 2026-09-16. There is one release, v1.0, dated 2026-07-05. The project's own version field in pyproject.toml still reads 0.0.1, so the release tag and the package version do not agree, which is worth knowing if you plan to pin by version number rather than by commit. The presence of both pixi.lock and uv.lock suggests two supported environment paths, though the README only documents the pixi one.

Upgrade cost is dominated by the unpinned torch. Because torch is not pinned in pyproject.toml and consumers bring their own, a torch upgrade can move you outside the known-good torchcodec 0.7.0, torch 2.8.0 and FFmpeg 7 combination that the file names. The base dependency set is small enough that the Python side is unlikely to break; the media stack is where upgrades bite. The codec path adds a second external dependency, the gated DINOv3-L/16 weights, and Meta controls access to those, so a codec training rerun depends on a download you may not be able to repeat on demand. World-model training and inference avoid that dependency entirely, which is the cheaper path to keep working.

The licence is Apache-2.0, declared in pyproject.toml with license-files pointing at LICENSE, and the README repeats it. Apache-2.0 is permissive and includes a patent grant, which matters for a model release more than for a utility library. It does not resolve the DINOv3 weights, which are gated separately by Meta and carry their own terms, nor the dataset on the HuggingFace Hub, which has its own card. If you plan to ship anything derived from codec training, check those two upstream licences rather than assuming the repository's Apache-2.0 covers the whole pipeline. That is a factual boundary, not legal advice.

## Conclusion

Adopt MIRA if you have an NVIDIA GPU, a reason to work with multi-view synchronised gameplay data, and the patience for a Hydra-configured training stack. Do not adopt it if you want a general-purpose world model, a CPU-only demo, or a drop-in environment for reinforcement learning: the README documents offline evaluation of a checkpoint, not an RL loop. Before committing, verify three things in order: that pixi run setup completes on your GPU host, that the DINOv3-L/16 checkpoint downloads and RS_DINO_WEIGHTS_DIR points at it if you intend to train the codec, and that the torch, torchcodec and FFmpeg combination you install matches the known-good triple of torchcodec 0.7.0, torch 2.8.0 and FFmpeg 7 named in pyproject.toml.

## FAQ

### Where can I find MIRA?

The code is at github.com/mira-wm/mira, under Apache-2.0, and the README points to mira-wm.com for live play. The technical report is on arXiv, and the Rocket Science dataset is on the HuggingFace Hub at kyutai/rocket-science.

### What hardware does MIRA need to run?

The README's setup command carries the comment that it requires an NVIDIA GPU, and the 20 FPS figure for a full 2v2 match is stated for a single GPU. The README does not name a specific GPU model.

### Does MIRA training require the gated DINOv3 weights?

Only codec training does. The README states that codec training uses a frozen DINOv3-L/16 encoder whose weights are gated by Meta, and that world-model training and inference do not need it.

### How do I load the Rocket Science dataset without training a model?

Import RocketScienceDataset from mira.data and call from_hub with the kyutai/rocket-science repository, a split, and optionally shards=1 for a single shard. Then load_match on a match id returns clips with frames, actions, events and physics.

### Which torch and FFmpeg versions work with MIRA's video decoding?

pyproject.toml names a known-good combination of torchcodec 0.7.0, torch 2.8.0 and FFmpeg 7, and notes that torchcodec needs a runtime FFmpeg whose libraries match its build. torch itself is not pinned, so consumers provide their own.

## Sources

- [License: Apache-2.0](https://github.com/mira-wm/mira/blob/main/LICENSE)
- [mira-wm/mira on GitHub](https://github.com/mira-wm/mira)
- [Project website](https://mira-wm.com/)
- [README](https://github.com/mira-wm/mira/blob/main/README.md)
- [Releases](https://github.com/mira-wm/mira/releases)

---

Hysen Labs editorial analysis, written from the project's own repository and release notes. Cite the canonical page: https://hysenlabs.com/projects/mira-wm-mira