ReCogDrive: a vision-language model plus a diffusion planner for driving

Why a language model should not output driving trajectories

ReCogDrive is a Reinforced Cognitive framework for end-to-end autonomous driving, from Huazhong University of Science and Technology and Xiaomi EV, published at ICLR 2026, with the paper on arXiv and a project page linked from the README.

The motivation is a specific critique of how vision-language models have been applied to driving. Recent studies have explored leveraging the world knowledge and cognitive capabilities of VLMs to address the long-tail problem in end-to-end autonomous driving, but existing methods typically formulate trajectory planning as a language modeling task, where physical actions are output in the language space. The abstract names the three consequences: format-violating outputs, infeasible actions, and slow inference speeds.

That is a precise engineering complaint rather than a philosophical one. A model that emits a trajectory as text has to serialise it, and the text may be malformed. A model that emits it as coordinates has to satisfy the vehicle's kinematics, and a sampled sequence can be kinematically impossible. And autoregressive generation of a long coordinate sequence is slow, because every coordinate is a token.

The proposed answer has two halves. Driving understanding and planning are unified by integrating an autoregressive model with a diffusion planner, so the language model supplies the reasoning and the diffusion model supplies the continuous trajectory. Then, on top of that, a reinforcement learning stage is added for safety.

The topic tags place the work in the diffusion policy, reinforcement learning and vision-language-action space, and the README points to a follow-up project, UniDriveVLA, that extends the same direction towards unified driving understanding, reasoning and action generation.

Generation, refinement and quality control as a data pipeline

The first contribution is a hierarchical data pipeline, and the design rationale is stated as mimicking the sequential cognitive process of human drivers. It has three stages: generation, refinement, and quality control.

The point of the pipeline is to instill human driving cognition into the VLM, and the sequencing matters. A raw driving scene is not the same as a driving scene described by somebody who has driven it, and the three stages correspond to writing a description, improving it, and checking it.

The second contribution addresses the language-action mismatch directly. The VLM's learned driving priors are injected into a diffusion planner, which then efficiently generates continuous and stable trajectories. Diffusion is doing the work that token-by-token generation was bad at, and the priors are doing the work that a generic diffusion planner without a world model would be bad at.

Together, the two pieces are what the name describes. The cognitive part is the data pipeline and the VLM, and the action part is the diffusion planner, and the framework is the coupling between them.

For a reader evaluating the project, the useful way to look at the three stages is as a curriculum: the same model is trained in three passes, and each pass adds a capability, which is also how the released checkpoints are named.

DiffGRPO, the third training stage, is where safety is bought

The third contribution is a Diffusion Group Relative Policy Optimization stage, called DiffGRPO, which reinforces the planner for enhanced safety and comfort.

That is a reinforcement learning stage applied to a diffusion planner, and the group relative part is the piece worth noting: the method compares outcomes within a group of sampled trajectories rather than against an absolute baseline, which is a practical choice when the reward is a composite of collision avoidance and comfort rather than a single number.

The training structure is visible in the checkpoint table, because the checkpoints are named by stage. The ReCogDrive-Base-VLM checkpoint is 2B and covers stage 1, the ReCogDrive-Base-IL checkpoint is 2B plus 35M and covers stages 1 and 2, and the ReCogDrive-Base-RL checkpoint is 2B plus 35M and covers stages 1, 2 and 3. The Large variants are the same three stages at 8B plus 35M.

The 35M figure is the planner, and the 2B or 8B figure is the language model that feeds it, which tells you the architecture at a glance: a comparatively small action network driven by a large reasoning model.

The abstract claims state-of-the-art performance on the NAVSIM and Bench2Drive benchmarks, and adds that qualitative results across diverse driving scenarios and on DriveBench highlight the model's scene comprehension. The last claim is the one to be careful about, since DriveBench is listed in the updates checklist as still unreleased.

What the released checkpoints actually score

The checkpoint table is the most concrete part of the README, because it gives a single number per model and the stage that produced it. On NAVSIM, with the score column labelled PDMS:

The Base-VLM model at 2B, stage 1, scores 84.1. The Base-IL model at 2B plus 35M, stages 1 and 2, scores 86.5. The Base-RL model at 2B plus 35M, stages 1, 2 and 3, scores 90.8. The Large-VLM model at 8B, stage 1, scores 86.4. The Large-IL model at 8B plus 35M, scores 86.5. The Large-RL model at 8B plus 35M, scores 90.4.

Two things stand out. The reinforcement stage is worth several points on both scales, and it is worth more to the small model than to the large one, since the 2B configuration gains 4.3 points from stage 1 to stage 3 while the 8B configuration gains 4.0. And the large model at stage 1 already matches the small model at stage 2, which says the extra language model capacity buys reasoning that the imitation stage would otherwise have to learn from data.

Every one of the six checkpoints has a weight download link, and they are published on Hugging Face under the owl10 account, with a weights collection and a datasets collection for the pretraining data.

For the Bench2Drive benchmark the README has a second table for closed-loop and multi-ability testing results in CARLA, and the table is cut off in the version available here, so the CARLA numbers are not something this page can report.

Driving pretraining data, and a reproducibility fix worth knowing about

The data release is substantial and is not only about driving frames. The August 2025 news entry says the project released all driving pretraining QA, including 12 driving datasets and their own annotated NavSim data, and that the scoring, filtering and evaluation for open-source data were rewritten.

That last clause is the interesting one for anybody planning to train on the released corpus. A pretraining set that is scored, filtered and evaluated with a consistent pipeline is usable, and one that is a dump of frames with inconsistent processing is not, so rewriting that layer was a substantial part of the release rather than a detail.

The datasets are published on Hugging Face under owl10/ReCogDrive_Pretraining, and the project pages link both the weights collection and the datasets collection.

There is also a maintenance note from September 2025 that is worth reading before you compare any numbers you reproduce yourself. The project fixed an unintended NumPy issue that had previously caused inconsistencies in the training metric cache, and says the code now ensures reproducible and consistent results, with thanks to the discussion in issue 10 for raising it. A metric cache that depends on library behaviour rather than on your seed is a classic reproducibility trap, and the project fixed it in the code rather than in the documentation.

The updates checklist shows the paper, the full models and the training and evaluation framework, the full driving pretraining datasets and the updated paper as done, and the release of the Bench2Drive, DriveLM, NAVSIM2.0 and Drivebench evaluation frameworks as still outstanding.

Getting the environment, the data and the training scripts lined up

The Getting Started section is three links, and each one is a prerequisite rather than a step you can skip.

First, the NAVSIM datasets, downloaded by following the official instruction in the NAVSIM repository, and the NAVSIM benchmark is where the PDMS numbers come from, so the data and the evaluator have to match what the results were produced with. Second, the preparation of the ReCogDrive environment, documented in docs/Installation.md, and the repository carries an environment.yml and a requirements.txt for it. Third, the training and evaluation documentation in docs/Train_Eval.md.

The requirements file shows how heavy the stack is. It pulls nuplan-devkit from a git ref, and then a large scientific set: numpy 1.26.4, opencv-python, pandas, geopandas, rasterio, scipy, shapely, ray, sympy, hydra-core for configs, bokeh and selenium for the nuBoard dashboard, matplotlib for rendering, and pyarrow, SQLAlchemy 1.4.27, pytorch-lightning, torch, torchvision, diffusers, timm and tensorboard for the model side.

The repository layout mirrors that. The internvl_chat/ directory holds the vision-language model code, navsim/ holds the benchmark code and the tutorial/ and vqa_evaluation/ directories hold the evaluation work, with scripts/ and download/ for the surrounding tooling. The setup.py is a copy of the NAVSIM packaging, with the name navsim, version 1.1.0, the University of Tuebingen as the author and a python_requires of 3.9.

Where ReCogDrive is the wrong tool, and the licence detail to check

Four limits are worth being clear about.

The first is that this is a research codebase, not a vehicle stack. It trains and evaluates a learned policy on the NAVSIM and Bench2Drive benchmarks. There is no perception stack, no sensor pipeline, no real time loop, no safety controller, and the evaluation harnesses for Bench2Drive, DriveLM, NAVSIM2.0 and DriveBench are still listed as not yet released. A team that needs a car to drive somewhere today is looking at something else.

The second is the compute and the model sizes. The checkpoints are 2B and 8B language models with a 35M diffusion planner, and the dependency set includes nuplan, ray, hydra and PyTorch Lightning, so the training run is a multi GPU research job rather than a weekend project.

The third is the alternative in approach, and it is a real design choice rather than a missing feature. The framework that ReCogDrive argues against is the modular stack, where perception produces structured state, a planner reasons over it, and a controller enforces the vehicle's limits, with explicit safety rules layered on top. ReCogDrive replaces that with a learned policy that reasons in language space and acts through a diffusion planner. The gain is the long-tail reasoning that a rule based planner cannot express. The loss is the guarantee: there is no intermediate representation you can inspect, and the safety argument moves from a rule you can read to a reward you tuned.

The fourth is the licence, and it is inconsistent. The project licence is Apache 2.0. The packaging inside the tree, the setup.py inherited from NAVSIM, declares license apache-2.0 but also carries a classifier reading License :: Free for non-commercial use, and the requirements pull in nuplan-devkit, which is a separate project with its own terms. This is not legal advice, but it is exactly the kind of detail that a legal review of a self-driving stack will ask about, and it is cheap to check now rather than later.

Editorial conclusion

Adopt ReCogDrive if you are doing research on end to end autonomous driving and want the published checkpoints, the driving pretraining data and a training and evaluation pipeline that reproduces the NAVSIM and Bench2Drive numbers. Do not adopt it as a production driving stack, and do not assume the licence is uniform across the tree. Verify first that the NAVSIM data and environment you install match what the scripts expect, that the stage you train corresponds to the checkpoint you wanted, and whether the non-commercial classifier in the bundled packaging metadata applies to your use, because the project licence is Apache 2.0 while the packaging inside the tree carries a different label.

Frequently asked questions

Official sources

1. Issues
2. License: Apache-2.0
3. README
4. xiaomi-research/recogdrive on GitHub