# ORION: an end-to-end driving framework that reasons before it steers

> ORION is the ICCV 2025 code from Xiaomi EV and Huazhong University of Science and Technology, an end-to-end autonomous driving framework that puts a language model in the loop and then generates trajectories from it. The repository ships inference, training, open-loop and closed-loop evaluation, a checkpoint and the Chat-B2D dataset, and it expects an A100 or a 32GB GPU to run.

**xiaomi-mlab/Orion** — [ICCV 2025] Official code of "ORION: A Holistic End-to-End Autonomous Driving Framework by Vision-Language Instructed Action Generation"

- Repository: https://github.com/xiaomi-mlab/Orion
- Stars: 668 · Forks: 75
- Language: Python
- License: Apache-2.0
- Published: 2026-10-21 · Updated: 2026-10-21 · Language: en
- Canonical page: https://hysenlabs.com/projects/xiaomi-mlab-orion

## The problem the paper attacks: reasoning space versus action space

The abstract states the failure mode precisely. End-to-end autonomous driving methods still struggle to make correct decisions in interactive closed-loop evaluation, and the reason given is limited causal reasoning capability. The current approach is to use the understanding and reasoning abilities of Vision-Language Models to resolve this, and the open problem the authors identify is that few VLMs for end-to-end methods perform well in closed-loop evaluation, because of the gap between the semantic reasoning space and the purely numerical trajectory output in the action space.

That framing is the whole point of the name. The name expands to an end-to-end autonomous driving framework by Vision-Language instructed actiON generation, with the letters in the acronym picked out of that phrase.

The architecture is three named parts, and the mapping from the paper's problem statement to the design is direct. A QT-Former aggregates long-term history context, so the temporal problem is handled by a dedicated module rather than by hoping the language model remembers. A Large Language Model handles driving scenario reasoning, which is the semantic half. A generative planner handles precision trajectory prediction, which is the numerical half. Then the paper adds that ORION further aligns the reasoning space and the action space to implement a unified end-to-end optimization for both visual question-answering and planning tasks.

The reported result is 77.74 Driving Score and 54.62 percent Success Rate on the Bench2Drive challenge datasets, which the abstract says outperforms state-of-the-art methods by 14.28 DS and 19.61 percent SR. Those are the authors' own numbers on their own benchmark, from the paper, and the repository also carries a results table with per-baseline rows, config links, checkpoint downloads and evaluation JSON files, which is the more useful artefact for anyone wanting to compare.

The project is from Xiaomi EV and Huazhong University of Science and Technology, was accepted at ICCV 2025, the paper is on arXiv as 2503.19755, and the licence is Apache-2.0.

## Setting up the environment, and the pinned dependency list

The getting started section is a single copy-and-paste block, and the pins in it are the important part:

```bash
git clone https://github.com/xiaomi-mlab/Orion.git
cd ./ORION
conda create -n orion python=3.8 -y
conda activate orion
pip install torch==2.4.1+cu118 torchvision==0.19.1+cu118 torchaudio==2.4.1 --index-url https://download.pytorch.org/whl/cu118
pip install -v -e .
pip install -r requirements.txt
```

So Python 3.8 in a conda environment named orion, PyTorch 2.4.1 with CUDA 11.18 wheels installed from the PyTorch index, an editable install of the repository itself, and then the requirements file.

That requirements file is long, and it mixes the driving stack with a lot of evaluation tooling. The driving-relevant entries are torch at 2.4.1 with the cu118 build, torchvision at 0.19.1, flash-attn at 0.2.8, transformers pinned to exactly 4.31.0, peft, diffusers at 0.32.2, numpy at 1.23.0 and shapely at 1.8.5.post1. The transformers pin being exact rather than a range is the kind of detail that decides whether the install works at all, because an unpinned transformers will pull a version whose API the model code does not match.

The evaluation tooling is separate and equally heavy: nuscenes-devkit, cityscapesscripts, open3d, scikit-learn, scikit-image, torchmetrics, motmetrics, imagecorruptions, casadi, einops and trimesh, plus a pytest and linting set. There are also platform-conditional entries, with pycocotools on Linux, pycocotools-windows on Windows, and regex only on win32, which is a sign the authors run the evaluation on more than one platform.

The setup.py in the repository shows that the package compiles CUDA extensions, with a make_cuda_ext helper that switches between CUDAExtension and CppExtension depending on whether torch.cuda.is_available(), and that is a detail worth knowing because a machine without a GPU will still attempt a build, just without the CUDA path.

## Weights: a pretrained Q-Former and an OmniDrive vision projector

The preparation step is short and consists of creating a directory for checkpoints:

```bash
cd /path/to/ORION
mkdir ckpts
```

ORION uses pretrained 2D llm weights and a vision encoder with projector weights, and the README states that the vision encoder and projector weights are extracted from the ckpts/pretrain_qformer/ directory, which is pretrained by using llava data. The links point at a Hugging Face repository for the 2D llm weights and at an OmniDrive release for the encoder and projector, with the OmniDrive repository named as the source project for those weights.

The naming is confusing if you have not worked in this area, because the directory is called pretrain_qformer but holds the vision encoder and projector weights, and it is the LLM weights that live in the separate Hugging Face link. Both have to be in place before inference or training will run, and neither is in the repository.

The benchmark data is a separate dependency too. The README points to the Bench2Drive dataset preparation guide in the Bench2DriveZoo repository, and separately notes that the Chat-B2D dataset is provided on Hugging Face to help reproduce the results.

The tree shows how the code is organised: an adzoo/ directory holds the experiment configurations and shell scripts, team_code/ holds the close-loop team agent code, mmcv/ is a vendored copy of the mmcv library, assets/ contains results and images, and there are data/ and docs/ directories. The vendored mmcv and the presence of a team_code directory both point at a codebase derived from the Bench2DriveZoo line of work rather than one built from scratch.

## Three-stage training, and what the stages are for

Training follows a three-stage process, and the README gives the commands for each. Stage one is pre-training on the Chat-B2D data, and the first step is to put the dataset under the data directory by unzipping it:

```bash
unzip Chat-B2D.zip -d data/
```

Then the stage one training runs through a distributed launch script with a config and a GPU count:

```bash
./adzoo/orion/orion_dist_train.sh adzoo/orion/configs/orion_stage1_train.py $GPUS
```

The script takes the number of GPUs as a shell argument, which is why the same pattern appears in every command in the README.

After stage one finishes, stages two and three run with a different config:

```bash
./adzoo/orion/orion_dist_train.sh adzoo/orion/configs/orion_stage2(3)_train.py $GPUS
```

The README includes a reminder to change the load_from setting in the config file, which tells you the stages are chained through checkpoints rather than being independent runs.

The dataset is described as being used for pre-training, and the Chat-B2D name points at the chat-flavoured data generation the framework is built around, since the whole design is a vision-language model reasoning about a driving scene. The names of the two later stages are not spelled out in the README excerpt, which is a genuine gap for someone planning a training budget: you can see stage one is a distinct phase from stages two and three, but not what each one optimises.

The currently supported features list at least tells you what exists today: the inference framework, open-loop evaluation, close-loop evaluation, the ORION checkpoint, the Chat-B2D dataset and the training framework, all marked as done.

## Open-loop evaluation, chain of thought, and the memory you need

Three evaluation paths are documented, and they differ in what they compute and what they cost.

The default open-loop evaluation runs the stage three inference config:

```bash
./adzoo/orion/orion_dist_eval.sh adzoo/orion/configs/orion_stage3_infer.py [--PATH_CHECKPOINTS] 1
```

The chain-of-thought variant swaps in a different config, and the README warns that it might be quite slow:

```bash
./adzoo/orion/orion_dist_eval.sh adzoo/orion/configs/orion_stage3_cot.py [--PATH_CHECKPOINTS] 1
```

That warning is the point. A chain-of-thought pass is asking the language model to lay out its reasoning before producing the trajectory, which is the more informative mode for understanding what the model decided, and the more expensive one to run over a benchmark.

The hardware guidance is explicit and worth reading before you queue anything. The README recommends inference on an NVIDIA A100 or other GPUs with more than 32GB of memory, in FP32 as the default. It then notes that ORION can also perform FP16 inference and achieve almost the same performance, and recommends fp16 inference on a GPU with more than 17GB of memory. There is a config for that too:

```bash
./adzoo/orion/orion_dist_eval.sh adzoo/orion/configs/orion_stage3_fp16.py [--PATH_CHECKPOINTS] 1
```

So the memory question reduces to a choice of precision, and the difference between the two recommendations is 32GB against 17GB. For anyone without an A100, the FP16 path is the one that is actually reachable, and the claim that it costs almost nothing in accuracy is what makes it worth taking.

The FP16 path also shows up in the dependency list, since flash-attn at 0.2.8 is an attention implementation that works in half precision, and the setup compiles CUDA extensions when a GPU is present.

## Closed-loop evaluation, and the one setting you must not skip

Closed-loop evaluation is the metric the paper's headline numbers use, and it is not self-contained. The README points at the Bench2Drive repository for the evaluation tools and for preparing CARLA, and at the Bench2Drive evaluation tools documentation for how to run them.

That dependency is significant in practice. Closed-loop evaluation means driving in a simulator, which means a CARLA installation, the Bench2Drive tooling, and a team agent configured for your checkpoint. The README flags the configuration step with a note: you may first verify the correctness of the team agent, and you need to set GPU_RANK, TEAM_AGENT and TEAM_CONFIG in the eval scripts.

The configuration is a single value in the environment, with the checkpoint path appended:

```bash
TEAM_CONFIG=adzoo/orion/configs/orion_stage3_agent.py+[CHECKPOINT_PATH]
```

The plus sign concatenation, with the checkpoint path in square brackets, is a detail that will bite anyone who copies it literally.

The result table in the README is worth reading as a whole rather than for the ORION row. It lists baseline methods such as UniAD-Tiny and UniAD-Base with their L2 error at 2 seconds, Driving Score and Success Rate, and for each one a config link, a checkpoint download, and an evaluation JSON file under assets/results. The baselines come from the official Bench2DriveZoo results, and having the evaluation JSON committed is the part that makes the comparison reproducible rather than merely asserted.

## What the follow-up work tells you about the project's direction

The news section has one entry that is not about ORION itself, and it is the most forward-looking thing in the README. On 2026-06-20 the follow-up work MindDrive, a closed-loop reinforcement learning VLA model with fewer parameters and higher performance, was accepted by ECCV 2026, with a paper and a separate code repository.

The other entries are a timeline of what was released when. The ArXiv paper appeared on 2025-03-26, the inference code and checkpoint on 2025-04-10, the ICCV 2025 acceptance and the promise of training code and dataset on 2025-06-26, and the training code and dataset release on 2025-08-13.

Read as a sequence, that is a project that shipped a working checkpoint first, then the training recipe and the dataset, which is a sensible order for anyone reproducing it: you could evaluate ORION from April 2025 without training anything.

The MindDrive direction is worth noting for a different reason. Moving from a language-model-driven generator to a closed-loop reinforcement learning VLA with fewer parameters is a move away from the very mechanism that makes ORION interesting, and the fact that the same group published it suggests the ORION approach was a stepping stone rather than the end state.

For an adopter, the practical reading is that this is a research codebase with a documented release history, an open dataset, committed evaluation JSON, and an active successor. The last push to the repository was on 2026-06-22, the project is not archived, and it publishes no GitHub releases, so the code and the paper are the only artefacts you get.

## Conclusion

Adopt ORION if you are working on end-to-end driving research and want the language-model reasoning stage, the generative planner and the Chat-B2D dataset in one released codebase with committed evaluation JSON, so that your comparisons land on the same benchmark artefacts rather than on reimplemented numbers. Do not adopt it as an inference stack for a vehicle, since the pipeline needs 17GB of GPU memory for the FP16 path and 32GB for the default FP32 one, and the closed-loop evaluation path additionally needs a CARLA setup and the Bench2Drive tooling that this repository does not include. Before you commit, install the pinned torch 2.4.1 and transformers 4.31.0 builds first, since the requirements file pins transformers to an exact version and an unpinned install will not build, and decide whether you need the chain-of-thought configuration, which the README warns is considerably slower than the default inference.

## FAQ

### What is ORION and what does it combine?

It is an end-to-end autonomous driving framework from Xiaomi EV and Huazhong University of Science and Technology, accepted at ICCV 2025. It combines a QT-Former that aggregates long-term history context, a large language model for driving scenario reasoning, and a generative planner for trajectory prediction, aligning the reasoning space with the action space for joint VQA and planning optimisation.

### What results does ORION report on Bench2Drive?

The paper reports 77.74 Driving Score and 54.62 percent Success Rate on the challenge Bench2Drive datasets, which the abstract says beats state-of-the-art methods by 14.28 DS and 19.61 percent SR. The README also carries a results table with baseline rows, config links, checkpoint downloads and committed evaluation JSON files.

### What hardware do I need to run ORION inference?

The README recommends an NVIDIA A100 or a GPU with more than 32GB of memory for the default FP32 inference, and notes that FP16 inference achieves almost the same performance on a GPU with more than 17GB of memory, using a separate stage three config for the FP16 path.

### How do I train ORION?

Training is a three-stage process. You unzip the Chat-B2D dataset into the data directory, run the stage one pre-training with the orion_dist_train.sh script and the stage1 config, then run the stage two and three configs after changing the load_from setting in the config file. The scripts take the number of GPUs as a shell argument.

### What do I need before closed-loop evaluation of ORION?

Closed-loop evaluation is not self-contained. You clone the Bench2Drive evaluation tools and prepare CARLA, set GPU_RANK, TEAM_AGENT and TEAM_CONFIG in the eval scripts, and set TEAM_CONFIG to the stage three agent config with the checkpoint path appended. The results for UniAD and VAD come from the official Bench2DriveZoo numbers.

## Sources

- [Issues](https://github.com/xiaomi-mlab/Orion/issues)
- [License: Apache-2.0](https://github.com/xiaomi-mlab/Orion/blob/main/LICENSE)
- [README](https://github.com/xiaomi-mlab/Orion/blob/main/README.md)
- [xiaomi-mlab/Orion on GitHub](https://github.com/xiaomi-mlab/Orion)

---

Hysen Labs editorial analysis, written from the project's own repository and release notes. Cite the canonical page: https://hysenlabs.com/projects/xiaomi-mlab-orion