carla_garage: a starter kit for the CARLA leaderboard 2.0

Why a starter kit was needed

The repository's own claim is specific and worth repeating accurately: this is the first complete starter kit for the CARLA leaderboard 2.0 where all components are open-source, including the dataset, the expert driver, the evaluation and the training code. Before it, a group entering the CARLA leaderboard had to assemble the evaluator, the scenario runner, a rule-based expert to generate training data and the training pipeline themselves, and the pieces were scattered across the leaderboard repository and various papers.

Two papers are attached to the work. Hidden Biases of End-to-End Driving Models, ICCV 2023, describes the method, and the LB2 Technical Report discusses the changes made to adapt TransFuser++ to the CARLA leaderboard 2.0. The repository also hosts a video, a talk, slides, a poster and the leaderboard 1.0 code on a separate leaderboard_1 branch.

The bias angle in the paper title is the research point. The claim is that end-to-end driving models trained with imitation learning pick up hidden biases in the expert data, so a model can score well on a benchmark while inheriting a systematic flaw in how the expert drove. That is a claim about what the benchmark measures rather than about the model's driving, and it is why having the dataset and the expert driver open matters for anyone who wants to check it.

Setting up the simulator and the conda environment

Setup is a fixed sequence: clone, install CARLA 0.9.15, build the conda environment. The README gives the commands:

git clone https://github.com/autonomousvision/carla_garage.git
cd carla_garage
git checkout leaderboard_2
chmod +x setup_carla.sh
./setup_carla.sh
conda env create -f environment.yml
conda activate garage_2

The environment name is garage_2 and the environment file is environment.yml, both checked into the repository. The CARLA version matters and is pinned in the instructions rather than left open, which is sensible for a simulator whose Python API changes between versions.

The next step is environment variables, and it is the part that trips people up. The README says you need to add several paths to PYTHONPATH on your system:

export CARLA_ROOT=/path/to/CARLA/root
export WORK_DIR=/path/to/carla_garage
export PYTHONPATH=$PYTHONPATH:${CARLA_ROOT}/PythonAPI/carla
export SCENARIO_RUNNER_ROOT=${WORK_DIR}/scenario_runner
export LEADERBOARD_ROOT=${WORK_DIR}/leaderboard
export PYTHONPATH="${CARLA_ROOT}/PythonAPI/carla/":"${SCENARIO_RUNNER_ROOT}":"${LEADERBOARD_ROOT}":${PYTHONPATH}

Four separate roots are involved, the CARLA root, the scenario runner, the leaderboard and the work directory, and the second PYTHONPATH assignment appends all of them in one go. The README notes you can put these in your shell scripts or integrate them into your IDE directly, and gives the PyCharm route in detail: Settings, then Project, then Python Interpreter, then Show all, then the garage environment added from an existing conda environment, then Show Interpreter Paths, and the absolute paths added there without the PYTHONPATH prefix.

Three leaderboards, three purposes

The repository contains modified copies of the leaderboard for different jobs, and the difference matters more than it might look. The leaderboard directory holds the original leaderboard with minor changes, described as extra seeding options and an upgrade to Python 3.10. The leaderboard_autopilot directory is a modified version that stores extra information which helps the privileged expert driver solve the scenarios, and it is used for data collection. The Bench2Drive directory holds the leaderboard as adapted by the Bench2Drive team for short evaluation routes.

So there is a standard evaluation harness, a data-generation harness and a third-party team's variant for a different route set. Having all three in one tree means the same scenario_runner code and the same CARLA API version are used across generation, training and evaluation, which removes a whole class of mismatch where a model is trained on data generated by one version of the harness and evaluated by another.

The repository layout makes the same point at file level. There are scenario_runner and scenario_runner_autopilot directories, a leaderboard directory, a Bench2Drive directory, team_code holding the agent, a data directory, a docs directory and a tools directory. There are also shell drivers at the top level for running evaluation and data collection on a Slurm cluster, named run_evaluation_slurm_tfpp.sh and run_evaluation_slurm_plant.sh for the two models, 0_run_collect_dataset_slurm.sh for dataset collection, and Python entry points evaluate_routes_slurm_tfpp.py, evaluate_routes_slurm_plant.py and evaluate_routes_slurm_pdm_lite.py. The presence of a Slurm workflow with a max_num_jobs.txt indicates the intended scale of use is a cluster with many parallel route workers.

Running a local evaluation on a debug route

Evaluation requires a running CARLA server, which you start from the CARLA root:

cd /path/to/CARLA/root
./CarlaUE4.sh

Afterwards you run leaderboard/leaderboard/leaderboard_evaluator_local.py as the main Python file. Four options shape the run. The --agent-config option points at a folder holding a config.json and one or more model_0030.pth files, and if several model files are present their predictions are combined into an ensemble. The --agent option is set to team_code/sensor_agent.py. The --routes option takes a route file, for example leaderboard/data/debug.xml. And --checkpoint is set to a path such as /path/to/results/result.json.

For a first run, the debug route file is the sensible choice, since a full leaderboard route set is a long evaluation. Two environment variables control extras. Setting export SAVE_PATH=/path/to/results saves additional logs or visualizations, and setting export DEBUG_CHALLENGE=1 turns on the visualisation generation with the output path set by SAVE_PATH, for example /path/to/visualization.

The visualisation output is the debugging tool that makes this kit usable by a newcomer, because a closed-loop driving score on its own tells you very little about why a route failed.

Pretrained weights and the licensing split

The pretrained models are distributed as a zip on S3 and are licensed under CC BY 4.0, which is a different licence from the repository itself. The repository code is MIT, and the weights are Creative Commons Attribution 4.0. That split is worth noting for anyone planning a release, because a model card or a derived artefact based on those weights carries an attribution obligation beyond MIT's minimal requirement to keep the copyright notice.

The README is careful about what the weights are. These are not the exact model weights used in the CVPR CARLA 2024 challenge, but re-trained models with similar performance. The team provides one set of models with three different training seeds, trained on all towns including Town13, for the Bench2Drive and the CARLA leaderboard 2.0 test routes benchmarks. Models for the validation benchmark are promised for later.

Each folder in the archive has three parts: an args.txt with the training hyperparameters, a config.json with all hyperparameters, which is loaded automatically, and model_0030_0.pth files with the weights, where the trailing number indicates the seed or training repetition. The _0 in the file name is therefore the seed, which is how you pick one of the three repetitions rather than an accident of packaging.

Bench2Drive as a training benchmark

Bench2Drive is a CARLA benchmark proposed in the paper Bench2Drive: Towards Multi-Ability Benchmarking of Closed-Loop End-to-End Autonomous Driving. It consists of 220 very short routes, around 150 metres each, split across all towns, with one safety critical scenario in each route.

The README makes an observation about it that changes how you should read its scores. Since it uses all towns for training, the methods have seen the test towns during training, so it can be considered a training benchmark, reminiscent of level 4 driving. That is a fair criticism of the benchmark as a measure of generalisation, and it is the reason the team provides separate evaluation paths for Bench2Drive and for the leaderboard 2.0 test routes.

The benchmark also comes with a training dataset generated by the Think2Drive expert, but the README says the team does not use it here, observing that TransFuser++ trained with data from their own expert performs better. That is a small but telling statement about the hidden bias theme: the identity of the expert that generated the data is itself a variable that changes the resulting model, which is exactly the kind of dependence the Hidden Biases paper is about.

Where this kit stops being the right tool

Three limits are structural. The first is the hardware and simulator dependency. Every path through this repository needs a CARLA 0.9.15 server running with a GPU behind it, plus a conda environment and a set of environment variables configured. There is no lighter mode, so a machine without a GPU, or a reviewer who cannot run a simulator, cannot reproduce anything here, and the evaluation itself is slow enough that the Slurm scripts exist to parallelise it across a cluster.

The second is the scope of the code itself. The leaderboard directories are modified copies of upstream code, with minor changes such as extra seeding options, and the README also ships a file named potential_changes_to_leaderboard.txt, which suggests the modifications are still being tracked. Modified forks of an upstream harness are normal in this research area and are the right call for reproducibility, but they also mean that improvements to the upstream leaderboard do not flow in automatically.

The third is that the leaderboard version matters. The default branch is leaderboard_2, and the leaderboard 1.0 code is on a separate branch, so a project targeting the older benchmark has to work from a branch that is no longer the default and will receive less attention. There are no GitHub releases, so a user pins a commit rather than a version tag, and the last push to the repository was on 2026-10-07, which is recent but leaves no published release history to reason about.

Editorial conclusion

carla_garage suits a research group that wants to reproduce closed-loop end-to-end driving results on CARLA leaderboard 2.0 or Bench2Drive without rebuilding the harness, since the evaluation code, the privileged expert driver, the dataset generation pipeline and the training scripts are all present and the pretrained weights are downloadable. It is the wrong tool if you are not prepared to run the CARLA simulator itself, because every route here depends on a CARLA server and a GPU, and it is the wrong tool for a project that needs leaderboard 1.0, which lives on a separate branch. Verify first that your CARLA install is 0.9.15, that the model weights you download are CC BY 4.0 licensed, and note that the default branch is leaderboard_2 with the last push on 2026-10-07 and no GitHub releases.

Frequently asked questions

Official sources

1. autonomousvision/carla_garage on GitHub
2. Issues
3. License: MIT
4. README