Bench2Drive: Closed-Loop Driving Evaluation in CARLA

What Bench2Drive Measures

Bench2Drive addresses a specific evaluation problem in autonomous driving: measuring how an end-to-end policy behaves after its actions affect the simulated environment. It is intended for researchers working on E2E-AD, driving agents, CARLA-based evaluation, and machine-learning model comparisons. The benchmark uses closed-loop routes rather than treating a recorded scene as a static prediction problem. The README describes route-level evaluation, driving scores, success rates, Driving Efficiency, and Driving Smoothness. It also discusses closed-loop QA for VLM4AD through the related Bench2Drive-VL project, although that extension is separate from the main repository.

The benchmark is associated with the NeurIPS 2024 Datasets and Benchmarks Track and has a model repository named Bench2DriveZoo. The main repository includes evaluation and scenario-running directories, while the model code is maintained separately. This split matters for adoption. A researcher can use the benchmark and its results without assuming that every baseline agent is implemented in this repository.

Closed-Loop CARLA Architecture

The repository layout shows the main parts of the system: `leaderboard/`, `scenario_runner/`, `tools/`, and `B2DVisualize/`. The leaderboard directory contains benchmark configuration, including the Dev10 XML named in the README. `Dev10` is a 10-clip subset selected from the official 220 validation routes for quick model development and ablation studies. The benchmark is therefore built around route definitions and scenario execution rather than a standalone prediction script.

The README states that the evaluation protocol includes CARLA. It also records several changes to the evaluation code, including automatic cleaning code that can be placed in a bash script to restart evaluation after a CARLA crash. That detail identifies CARLA stability as part of the operating environment, not merely a background dependency. The benchmark evaluates a driving agent over a route and records behavior under the simulator's state transitions. The visible repository structure does not document a complete public Python API or a single package-level entry point.

Bench2Drive also exposes a dataset model for training. Version 0.0.4 adds 3D occupancy, while the benchmark supports separate route sets for training and validation. The distinction matters because the README warns against relying on open-loop nuScenes planning results. Its position is that closed-loop evaluation should assess the policy in a changing environment.

Bench2Drive 0.0.4 Data Distribution

The 0.0.4 update changes the training-data balance. The README specifies a new Think2Drive-collected training set containing 44 scenarios and 25 routes per scenario, for 1,100 routes in total. Every scenario contributes exactly 25 routes. This is different from the previous Base set, whose scenario distribution was uneven. The update also introduces a separate validation set with 44 scenarios and 5 routes per scenario, or 220 routes.

That split gives training and evaluation different route collections. It also gives researchers a reason to record which Bench2Drive version and split produced a result. Scores from the older Base distribution should not be treated as automatically comparable with scores from the 0.0.4 Think2Drive-collected set without checking the protocol and data version.

The 3D occupancy addition broadens the input modalities available to the benchmark, but the README does not specify the file format, coordinate convention, decoder, or training pipeline for that modality. Those details must be checked in the repository files before designing a model around 3D occupancy. The update notes also mention bug fixes across the dataset and Bench2DriveZoo, so an existing result should be associated with the exact commit or release state used.

Installing and Starting a CARLA Evaluation

The README links to the project website, the Hugging Face dataset, and the associated arXiv paper, but it does not give a complete install command, dependency list, Python version, or environment setup procedure. The repository's primary language is Python, and the top-level entries include Python-oriented benchmark infrastructure, but the installation instructions are not present in the README. According to the README, the intended sources for obtaining the project are the GitHub repository and the linked Hugging Face dataset.

A first real use therefore begins by obtaining the repository and the relevant benchmark files, then installing the dependencies required by the checked-out commit. The README does not confirm the exact command, so no installation command is reproduced here. After the environment and CARLA setup are available, the researcher can use `Dev10` for a quick evaluation cycle instead of starting with the full validation set. The documented configuration file is `leaderboard/data/drivetransformer_bench2drive_dev10.xml`.

The README gives this file path, but it does not provide the complete command-line invocation for the evaluation runner. The researcher should inspect the `leaderboard/` and `scenario_runner/` entry points in the selected commit before launching a route. The expected first result is an evaluation run over the selected 10 clips, with logs and route results collected according to the benchmark's evaluation code.

Evaluation Metrics and Failure Modes

Bench2Drive reports several kinds of driving information rather than a single isolated prediction number. The README names Driving Score and Success Rate, and it says that 0.0.4 retains those while adding Driving Efficiency and Driving Smoothness. In August 2024, the project also removed the minimum-speed penalty from Driving Score and increased `TickRunTime` from 2000 to 4000 to make evaluation more lenient. Those are protocol changes. Results produced under different versions should therefore record the relevant protocol changes.

The README documents several concrete failure risks. CARLA can crash, which is why automatic cleaning and restart logic was added to the evaluation toolkit. A prior issue involved an incorrect BACK CAMERA extrinsic in the UniAD and VAD team agents. The training process was described as correct, but the evaluation code was left inconsistent with previously reported results. Users are explicitly encouraged to use the correct extrinsics. This is a model-integration problem rather than a general benchmark guarantee.

The project also carries a caution about L2 error. The README cites CARLA_GARGE and argues that L2 error is not a meaningful indicator for this evaluation context. Bench2Drive should not be reduced to an offline displacement metric. Its closed-loop route execution is the relevant behavior under review.

Bench2Drive Versus Offline Planning Evaluation

The closest alternative described by the README is nuScenes open-loop planning evaluation. The two approaches differ in environment interaction. An open-loop benchmark compares a policy's output with a recorded reference sequence, while Bench2Drive sends actions into CARLA and evaluates the resulting driving behavior over a route. The README explicitly rejects relying on nuScenes open-loop planning results as persuasive evidence for driving performance.

This difference affects both setup and interpretation. Open-loop evaluation can use stored sensor data and does not require the policy to handle simulator state changes or crashes. Closed-loop evaluation introduces CARLA execution, route configuration, camera extrinsics, control latency, and the possibility that an early action changes later observations. It is therefore more faithful to the stated problem of E2E-AD behavior, but it also has more moving parts and more failure conditions.

Bench2Drive is the better fit when the research question concerns closed-loop driving, route completion, efficiency, smoothness, or simulator interaction. Offline datasets may remain useful for pretraining, supervised representation learning, or inexpensive comparisons, but they do not answer the same question. The README's position is a methodological warning, not a claim that all offline datasets are unusable.

Maintenance, Licensing, and Adoption Cost

The repository is not archived. Its last push was on 2026-08-11, which indicates recent repository activity, although it has no GitHub releases. The README records a Bench2Drive 0.0.4 update dated 2026-08-06, but the absence of GitHub releases means version availability and upgrade packaging are not documented through that release channel.

The repository metadata identifies the license as a custom license that GitHub cannot classify, and the repository contains a `LICENSE` file. The README does not reproduce the license terms, so adoption requires reviewing that file directly. The presence of custom terms is particularly relevant for organizations that redistribute datasets, derived annotations, or model outputs. This article does not interpret the license or provide legal advice.

Upgrade cost is likely to involve more than copying code. The 0.0.4 changes include new data, 3D occupancy, a separate validation set, and bug fixes across Bench2DriveZoo. Existing experiments may need retraining or at least reevaluation, especially if they used the previous uneven Base distribution. CARLA crashes and agent-specific camera extrinsic issues add operational work to evaluation campaigns.

Editorial conclusion

Researchers evaluating end-to-end driving policies in CARLA should consider Bench2Drive because it provides a closed-loop protocol and a more uniform training distribution in version 0.0.4. Users who need a documented one-command installation, a clear rollback procedure, or a non-CARLA simulator should verify those gaps first. The repository layout, dataset files, evaluation results, and Python environment are the first adoption checks.

Frequently asked questions

Official sources

1. Issues
2. README
3. Thinklab-SJTU/Bench2Drive on GitHub