Olympus (yuanze-lin): a router that turns one prompt into images, videos and 3D models

What Olympus routes, and who the router is actually for

The problem Olympus addresses is dispatch, not perception. A user writes one instruction in natural language, and the system decides which of 20 vision tasks that instruction implies, sends it to a specialist model for that task, and chains the specialists' outputs together. The README describes the result as one prompt in and finished .png, .mp4 and .glb files out, which is a claim about orchestration rather than about any single model's accuracy.

The intended audience is narrow. This is the official code release for a CVPR 2025 Highlight paper, with training code, an evaluation directory, a fine-tuning dataset and a released checkpoint. Someone evaluating Olympus as a library to drop into a production service will find a research repository: the README's own framing is that the project is helpful for research, and it asks readers to cite the paper. If you are studying how a language model can be taught to emit routing tokens and chain tool calls, the repository is directly relevant. If you want a stable API with semantic versioning, it is not that.

Routing tokens, a fine-tuned Mipha-3B, and a chain of specialist calls

The architecture visible in the repository has three layers. The first is the router model: Olympus is built on Mipha-3B, and the repository includes download_mipha_3b.py to fetch that base model alongside download_olympus.py for the fine-tuned Olympus checkpoint. The second is the routing vocabulary. The README points to routing token definitions in the paper, and the dataset ships 20 JSON files under a folder of 20 individual tasks, one per task, plus coa.json for chain-of-action data. That structure tells you the router is trained to emit tokens that name tasks, and coa.json supplies the sequences in which tasks follow one another. The third layer is execution: run_tools.py takes the routed tokens and calls the corresponding specialist models.

The training data is assembled rather than curated from scratch. Olympus.json, the final fine-tuning instruction set, is described as a combination of llava_v1_5_mix665k.json and OlympusInstruct, the project's own collected instruction data covering the 20 tasks. scripts/merge_data.py performs that merge if you download the LLaVA file yourself, and scripts/merge_tasks.py lets you choose which tasks participate. That is a useful detail for anyone reproducing the setup: the router's general instruction-following behaviour is inherited from LLaVA data, and the routing behaviour is layered on top with OlympusInstruct.

Installing Olympus and running the router for the first time

The README gives a single environment for everything, router and specialists alike, with no per-tool environment. The base install creates a conda environment named olympus on Python 3.10 and installs the router requirements.

git clone https://github.com/yuanze-lin/Olympus.git
cd Olympus
conda create -n olympus python==3.10 -y
conda activate olympus
pip install -r requirements.txt

The README states this is all you need to run the router on its own. Note that requirements.txt pins torch 2.2.0+cu118, while the specialist stack below pins a different PyTorch build. If you intend to execute tasks rather than only route them, install the specialist stack into the same environment, as the README instructs:

pip install torch==2.6.0 torchvision==0.21.0 --index-url https://download.pytorch.org/whl/cu124
pip install -r requirements_tools.txt
bash scripts/install_specialists.sh

Before running anything, fetch the checkpoint and the data. The scripts save into ckpts and train_data respectively.

python download_olympus.py
python download_olympus_dataset.py

The README's example invocation for executing a routed task, with the environment variables that redirect the gated conditioning models to local folders, is:

DINO_MODEL_PATH=/path/to/dinov3-vitl16-pretrain-lvd1689m \
SEG_MODEL_PATH=/path/to/RMBG-2.0 \
  python run_tools.py --prompt "..." --input-image assets/room.jpg

The README notes those variables apply only if the Hugging Face Hub call fails; the Hub is tried first. predict.py and predict.sh are the other top-level entry points listed in the repository.

The 3D backends are the part that will break your install

Olympus splits 3D generation across two backends. The token <3D_gen_image> uses microsoft/TRELLIS.2-4B and <3D_gen_text> uses TRELLIS-text-base, both producing textured meshes with PBR materials. They compile several CUDA extensions and therefore install separately through scripts/install_3d.sh, which also builds Hunyuan3D-2 as an ungated fallback for <3D_gen_image>. The README states each step is independent so one failure does not block the others, and that a 3D token whose backend is missing falls back automatically rather than erroring.

The gating is the sharp edge. TRELLIS.2-4B needs two gated Hugging Face repositories at runtime: facebook/dinov3-vitl16-pretrain-lvd1689m for image conditioning and briaai/RMBG-2.0 for background removal. Both licences must be accepted on the Hub while logged in via huggingface-cli login. If your request is rejected, the README points to microsoft/TRELLIS.2 issue 38 for obtaining the weights locally, and then you supply DINO_MODEL_PATH and SEG_MODEL_PATH. The alternative is to skip TRELLIS.2 entirely and let <3D_gen_image> fall back to Hunyuan3D-2, which is ungated. That fallback is a real design decision with a real cost: you trade a gated, higher-specification backend for an accessible one, and the README does not compare their output quality. Anyone whose work depends on <3D_gen_image> fidelity should treat that comparison as unverified.

Licence, maintenance and the cost of upgrading

The repository does not declare a licence in the README or the file listing, and there are no releases. The README asks readers to cite the paper if they find the project useful, which is a norm of academic code, not a licence grant. Treat the terms as unresolved until you check the repository yourself, and note that the two gated Hugging Face models carry their own licences that you accept separately, as do the Olympus weights and dataset hosted on the Hub. None of that is legal advice; it is a description of what is and is not stated.

The last push to the default branch was on 2026-08-05, which is recent enough that the code is not stale, but there is no release history and no versioning to plan an upgrade around. Practically, upgrading means pulling main and re-resolving two requirements files that pin conflicting PyTorch builds: torch 2.2.0+cu118 in requirements.txt and torch 2.6.0 with cu124 in the specialist install. Because the router and every specialist share one environment, a change to either file can affect the other stack. Budget for a rebuild rather than an incremental update, and expect the CUDA extension compilation in scripts/install_3d.sh to dominate the time.

How Olympus differs from wiring tools together with an agent framework

The obvious alternative is a general agent framework that lets a language model call functions you register, with the routing decision made at inference time by prompting. Olympus takes the opposite approach: the routing behaviour is trained into the model. OlympusInstruct supplies supervision for the 20 tasks, coa.json supplies chain-of-action sequences, and the model learns the routing token vocabulary from the paper. The trade-off is real. A prompted agent can add a new tool by writing a description and a wrapper; Olympus needs new instruction data and a fine-tuning run through the training code in the repository. In exchange, routing is a learned behaviour rather than a prompt-engineering artefact, which is the hypothesis the paper is testing.

A second difference is that Olympus assumes a fixed, curated task set with known specialists, including the 3D backends with their gated weights. A general framework would treat those as just another tool. If your task list changes weekly, the training-based router is the wrong shape for you. If your task list is stable and you want the dispatch decision to be part of the model rather than the prompt, the design is coherent.

Editorial conclusion

Adopt Olympus if you are doing research on task routing or multi-model orchestration and you can afford a single shared environment plus the specialist stack, including CUDA extension builds and two gated Hugging Face licences for TRELLIS.2-4B. Do not adopt it if you need a supported product, a licence you can read in the repository, or CPU-only inference. Before committing, verify three things: whether the repository declares a licence at all, whether your GPU can build the 3D backends in scripts/install_3d.sh, and whether Hunyuan3D-2 fallback quality is acceptable for your <3D_gen_image> requests.

Frequently asked questions

Official sources

1. Issues
2. Project website
3. README
4. yuanze-lin/Olympus on GitHub