AutoVLA lets a driving model think only when the scene deserves it

One autoregressive model emits both reasoning and trajectory

AutoVLA is the official implementation of the paper AutoVLA: A Vision-Language-Action Model for End-to-End Autonomous Driving with Adaptive Reasoning and Reinforcement Fine-Tuning, accepted at NeurIPS 2025, with the paper on arXiv as 2506.13757 and a project page at autovla.github.io. The work comes from the UCLA mobility lab, with Zewei Zhou and Tianhui Cai as equal contributors and Zhiyu Huang as project leader.

The core mechanism is a unified autoregressive generative process that covers two things usually kept apart: chain-of-thought reasoning and physical action tokenization. Instead of a planner that emits a trajectory and a separate reasoner that explains it, the model produces both in one stream, and the trajectories come out as tokens rather than as regression output.

The interesting part is that the reasoning is not constant. The model dynamically switches thinking modes, and supervised fine-tuning is what gives it both of them: fast thinking produces a trajectory only, and slow thinking adds chain-of-thought reasoning on top. That design has a practical motivation, since reasoning tokens cost time and a straight motorway merge does not need an essay about it.

The evaluation spans nuPlan, nuScenes, Waymo and CARLA, in both open-loop and closed-loop settings, and the model was entered in the Waymo Vision-based End-to-end Driving Challenge in May 2025, where it ranked highly in RFS Overall and took the top RFS Spotlight score, the metric focused on the hardest scenarios.

GRPO fine-tuning is aimed at cutting reasoning where it is not needed

The second training stage is reinforcement fine-tuning based on Group Relative Policy Optimization. Its stated purpose is twofold: to enhance planning performance and to improve runtime efficiency, specifically by reducing unnecessary reasoning in straightforward scenarios.

That is the part worth sitting with. A vision-language-action model that always reasons is a model that spends most of its latency explaining itself. Making the reasoning mode conditional and then training the model to pick the cheap mode when the cheap mode is enough is a different bet from pure accuracy work, and it is the kind of bet that shows up as a latency number rather than as a leaderboard number.

The training stack in the requirements list makes the implementation guessable. pytorch-lightning with torchmetrics and tensorboard for the training loop, peft for parameter-efficient fine-tuning, hydra-core with omegaconf for configuration, and lm-eval and evaluate for measurement. The base model is the Qwen2.5-VL series, with Qwen2.5-VL-3B-Instruct used inside the AutoVLA model itself.

The project history is a sequence of dated milestones rather than a stream of commits. The paper came out in June 2025, NeurIPS acceptance followed in September 2025, the codebase was released in February 2026 and the checkpoints on Hugging Face in May 2026. The last push to the repository was on 2026-05-29, and there are no GitHub releases, so the checkpoint is the artefact to pin rather than a tag.

The devkit is a vendored navsim with three dataset pipelines

The repository includes the navsim code, so the evaluation harness does not have to be fetched separately, and the instructions to set it up refer to the upstream navsim install documentation for the version specific details. The three dataset downloads are separate, because nuPlan, Waymo and nuScenes come from different places and have different structures.

For nuPlan the download is a set of scripts:

bash navsim/download/download_maps.sh
bash navsim/download/download_trainval.sh
bash navsim/download/download_test.sh

The Waymo end-to-end driving dataset is downloaded from the Waymo open data site, and the nuScenes route needs an account on the official site and the v1.0-trainval split, with the DriveLM nuScenes annotations, the file named v1_1_train_nus.json, coming from the OpenDriveLab DriveLM repository.

The environment itself is a conda file:

conda env create -f environment.yml
conda activate autovla
pip install -e . --no-warn-conflicts
bash install.sh

and navsim, which lives in its own directory inside this repository, is installed with:

cd navsim
pip install -e . --no-warn-conflicts

The navsim environment variables then have to point at your data layout, and the exported names are specific:

export NUPLAN_MAP_VERSION="nuplan-maps-v1.0"
export NUPLAN_MAPS_ROOT="/path_to_your_nuplan/maps"
export NAVSIM_EXP_ROOT="$HOME/AutoVLA/navsim/exp"
export NAVSIM_DEVKIT_ROOT="$HOME/AutoVLA/navsim"
export OPENSCENE_DATA_ROOT="$HOME/AutoVLA/dataset/nuplan"

Getting these right is the unglamorous part of the setup, and each one is a place where a wrong path turns into a confusing failure later rather than an obvious one now.

nuScenes preprocessing gets its own environment to dodge conflicts

The nuScenes path is separated out, and the reason is stated: preprocessing requires nuscenes-devkit, which might have dependency conflicts with the main environment. The recommendation is a separate conda environment, created from a second file, environment_nusc_preprocess.yml:

# Create a separate environment for nuScenes preprocessing
conda env create -f environment_nusc_preprocess.yml
conda activate nusc_preprocess

# Run preprocessing (please revise the path settings in the bash)

Splitting an environment is usually treated as a smell, and here it is the right call, since the devkit pulls in its own stack and the alternative is a conflict that breaks the main training environment rather than the preprocessing one.

The preprocessing entry points are shell scripts, one per dataset, and the same scripts handle the annotation step:

bash scripts/run_nuplan_preprocessing.sh

The Waymo route has two, because the image data is first cached in the same format the other dataset uses to support random access:

bash scripts/run_waymo_e2e_image_extraction.sh

bash scripts/run_waymo_e2e_preprocessing.sh

A single setting, INCLUDE_COT in the script, decides whether the chain-of-thought reasoning annotation runs at all. That switch matters more than it looks, because the annotation is the expensive part, and the two modes produce different training data rather than the same data with extra fields.

Qwen2.5-VL 3B drives the model, 72B produces the reasoning labels

The pretrained models are downloaded with a single script:

bash scripts/download_qwen.sh

The project uses the Qwen2.5-VL series as the pretrained VLM in the VLA model and in the CoT annotation model, and the two roles use different sizes. The 72B model is used for CoT annotation, and for that role you can choose Qwen2.5-VL-72B-Instruct or Qwen2.5-VL-72B-Instruct-AWQ depending on your device. The AutoVLA model itself uses Qwen2.5-VL-3B-Instruct.

The division is worth noting for planning. The reasoning labels are produced by a large model, so the annotation step has a hardware floor that the training step does not, and the AWQ variant exists precisely so that the annotation can run on a device that cannot hold the full precision model.

The dependency list is pinned tightly. torch is held at 2.4.0, torchvision at 0.19.0, transformers at 4.49.0 and tokenizers at 0.21.1, with triton at 3.0.0 and accelerate at 1.5.2 alongside. nuplan-devkit is installed straight from a git reference, the nuplan-devkit-v1.2 tag on the motional repository, and the geospatial stack is fixed as well, with geopandas 1.0.1, shapely 2.0.7, rasterio 1.3.11 and pyproj 3.6.1. The package metadata declares Python 3.9 or newer and the licence field reads Academic-Software-License.

The reasoning data is the release still missing

The release plan is a checklist with one line that is not done. The paper is marked complete, as are the annotation and training code from February 2026 and the checkpoints from May 2026. The last line, the reasoning data, is marked TBD with the note that it is pending approval from the data provider.

That single gap shapes what a visitor can actually do. Without the reasoning corpus, the CoT annotation path has to be run on your own data with your own 72B model, which is the expensive path the project was designed to avoid. The INCLUDE_COT switch in the preprocessing scripts therefore has two meanings depending on who is running it: for someone with the released data it is on, and for everyone else it is a bill for the annotation compute.

The dataset badge on the project page also reads Coming Soon, which is consistent with the release plan rather than a separate signal.

Two other things to note. The licence is an academic software licence that GitHub cannot classify, so the terms live in the LICENSE file and have to be read before use, and the repository has no GitHub releases, which means there is no versioned artefact beyond the checkpoint on Hugging Face and the pinned environment files.

Editorial conclusion

AutoVLA is worth adopting for a research team working on end-to-end driving who wants a released training pipeline rather than a paper alone, since the annotation code, the preprocessing scripts, the pinned environment and the checkpoints are all public, and the navsim devkit is vendored so the evaluation loop is reproducible. It is the wrong tool for anything that needs the reasoning corpus today, because that release is marked as pending approval from the data provider, and it is the wrong tool if your hardware cannot hold a 72B model for annotation. Verify first that the pinned stack, torch 2.4.0 and transformers 4.49.0, matches your driver and CUDA version, that the academic software licence permits your intended use, and that the nuPlan map and split paths in the environment variables match the data you actually have.

Frequently asked questions

Official sources

1. Issues
2. Project website
3. README
4. ucla-mobility/AutoVLA on GitHub