DriveGAN_code

DriveGAN is the code release for a CVPR 2021 paper that learns a controllable driving simulator from unannotated video, with stage based training and a dataset built on the Carla simulator.

DriveGAN is the code release for a CVPR 2021 paper on a controllable, high quality neural simulator for driving scenes, presented as an oral with authors Seung Wook Kim, Jonah Philion, Antonio Torralba, and Sanja Fidler. The abstract makes the case that realistic simulators matter for robotics, and that hand crafting them does not scale. Instead the work learns, straight from unannotated frames and their action pairs, how a dynamic environment responds to an action, directly in pixel space.

Controllability comes from disentangling components without supervision. Besides steering, DriveGAN exposes controls for sampling scene features like the weather and the location of non player objects. Because the simulator is fully differentiable, it can re simulate a recorded video sequence, letting an agent drive through the same scene again while taking different actions. The model was trained on several datasets, including 160 hours of real driving footage, and the paper claims it beats earlier data driven simulators.

The README targets Ubuntu 18.04 with Python 3.6.9, and expects other close Python 3 versions to work. The first step is cloning the repository. Business inquiries are routed to the NVIDIA Research licensing page.

The project provides a dataset derived from the Carla simulator, distributed under a Creative Commons BY-NC 4.0 license. All files sit on a Google Drive link given in the README, and the extracted folders are renamed from their 6405 prefixed names to data1 through data6 before training.

Training happens in two stages. Stage 1 is a VAE-GAN: download the archives, extract them, train until the validation loss converges, then encode the dataset with the learned model. Stage 2 is the dynamics engine, which can start from the encoded data and a checkpoint. If training is not on the menu, the README offers a shortcut: download the trained simulator and the VAE-GAN checkpoint and skip straight to play. A web page drives the model, with extra buttons for changing content, and refreshing the page samples a new scene. The codebase and trained models sit under the Nvidia Source Code License, with the VAE-GAN code adapted from stylegan2-pytorch and LPIPS imported from PerceptualSimilarity.

DriveGAN's release repo covers the whole path from paper to running model: environment setup, a Carla derived dataset, two training stages, and checkpoints for playing without training, with separate licenses for code and data.

• Source 1 · Project repository
• Source 2 · README