UniScene generates occupancy, video and LiDAR from one bird's-eye-view layout

One layout, three data forms instead of three separate generators

Generating synthetic driving data usually means writing one generator per output type. Occupancy prediction gets a model, video synthesis gets another, LiDAR simulation gets a third, and each one starts from the same coarse description of the scene while learning the layout-to-data mapping from scratch.

UniScene attacks that duplication directly. The stated contribution is a unified framework that produces semantic occupancy, multi-view video, and LiDAR point clouds, and the paper abstract calls it the first unified framework for those three data forms in driving scenes. The starting point is a customised BEV layout, which is a top-down, low-detail sketch of where things are in the scene. From that one input the system produces dense annotated output.

The audience is narrow and clear: autonomous driving teams that need high-fidelity, controllable, annotated training data and currently lack enough of it from real driving. The abstract also claims downstream benefit, stating that generating this way helps driving tasks that consume the data. The project page links to an arXiv paper (2412.05435), a PDF copy in the assets folder, a demo animation, and a Hugging Face dataset page holding the released weights.

Occupancy first, then video and LiDAR conditioned on it

The architecture is hierarchical and the order matters. The first stage generates semantic occupancy from the scene layout, and the paper treats that occupancy as a meta scene representation carrying both semantic and geometric information. The second stage takes the generated occupancy as its condition and produces video or LiDAR from it. The stated benefit of that ordering is that it lowers the generation burden, especially for complex scenes, and leaves a detailed intermediate representation that later stages can lean on.

Two transfer strategies carry the second stage. Gaussian-based Joint Rendering handles the video side, and Prior-guided Sparse Modeling handles the LiDAR side. Both are described in the abstract as novel transfer strategies, and both are the kind of mechanism you would look for in the code rather than in the docs, since the write-up does not break down how each strategy works internally.

The repository layout mirrors the pipeline. There are three top-level directories named occupancy_gen, video_gen and lidar_gen, alongside data_process for annotations and third_party for vendored code. The occupancy generator has its own readme, and the project notes that its output pc-range is set to [-40, -40, -1, 40, 40, 5.4] when comparing against the Occ3D baseline, while the ground truth labels use a different range of [-50, -50, -5, 50, 50, 3].

The environment asks for an A100, CUDA 12.1 and Python 3.9

This is a paper release, so the setup expectations are research expectations. The project recommends creating the environment with poetry through a script, and states that the setup has been tested on NVIDIA A100 and A800 hardware with CUDA 12.1 and Python 3.9. The pyproject.toml pins the interpreter range accordingly, requiring Python 3.9 or newer but below 3.10, and declares package-mode = false with version 0.1.0.

The dependency list reads like a research environment rather than an application. PyTorch is pinned to 2.5.1 and xformers is required, alongside deepspeed 0.10.1 and pytorch-lightning 2.0.1 for training. The perception side pulls mmdet 3.3.0, mmdet3d 1.4.0, mmcv 2.1.0 and mmengine 0.8.4, and diffusers 0.27.2 appears for the diffusion side of video generation. spconv-cu120 and kornia handle sparse convolution and image operations, and there is a streamlit dependency, which suggests a small demo interface ships with the code.

A step-by-step installation guide lives in install.md at the repository root, and the environment file environment.yml is committed alongside poetry.lock. Expect a large download and a long install, and expect the CUDA 12.1 specific packages to be the friction point if your driver stack differs.

Pulling pretrained weights with the huggingface-cli

Weights are released, and the documented way to fetch them is the huggingface-cli. The documentation gives the tool install and download commands:

pip install -U huggingface_hub 
huggingface-cli download --resume-download Arlolo0/UniScene_path  --local-dir $local_path

The command downloads from a repository named Arlolo0/UniScene_path into a local directory you set in the shell. The news section records that code and pre-trained weights were released in March 2025.

Three model families are listed with download links, and the naming tells you how the pipeline splits. The occupancy model offers an Occ-VAE checkpoint and an Occ-DiT checkpoint, the LiDAR model offers a single occ2lidar.pth file, and the video model offers a video_pretrained.safetensors file. The occupancy pair corresponds to the two-stage occupancy design, while the downstream LiDAR and video models are single files that take occupancy as input.

Each family is offered on both OneDrive and Hugging Face. The Hugging Face links point at files such as Occupancy_Generation_ckpt_AE_eval_epoch_196.pth, Occupancy_Generation_ckpt_DiT_0600000.pt, occ2lidar.pth and video_pretrained.safetensors. The presence of an alternative OneDrive mirror per model is useful when Hugging Face bandwidth is a problem, since these are multi-gigabyte files.

NuScenes trainval and the ground truth occupancy step

The data pipeline is the largest hidden cost. The project works on NuScenes and instructs you to download all splits of Trainval in the Full dataset v1.0, following the official nuscenes instructions, and place them under ./data/nuscenes. The expected layout is a directory with samples, sweeps and v1.0-trainval inside it.

On top of that, interpolated 12Hz annotation and mmdet3d meta data must be downloaded as nuscenes_interp_12Hz_infos_*.pkl files into data/nuscenes_mmdet3d-12Hz/. This step is not optional in practice. The 12Hz annotations come from a separate share point rather than from the official NuScenes download, so it is a separate download with a separate file naming pattern.

Optional ground truth 3D occupancy labels with LiDAR and 3D bbox labels are produced using NKSR reconstruction, aligned with the pc-range of [-50, -50, -5, 50, 50, 3] to match the OpenOccupancy convention. The generation command runs from the data_process directory:

cd ./data_process && python generate_occ.py --config_path  config-200.yaml --save_path $GT_nksr_occupancy_200/

That produces ground truth semantic occupancy at 200x200x16 resolution. The documentation also gives a command for the 800x800x64 resolution, and notes that the occupancy model itself outputs the [-40, -40, -1, 40, 40, 5.4] range used for Occ3D baseline comparisons. Anyone planning to compare numbers against published results needs both ranges to line up with the right baseline.

Where this differs from a layout-to-video model

The comparison point most readers will have in mind is a layout-conditioned video generator that maps a BEV sketch straight to multi-view camera footage. The architectural difference is what sits in the middle. In a direct generator, the video model must learn the mapping from a sparse, low-detail layout all the way to pixels, and it has to infer occluded geometry and object interiors on its own.

UniScene inserts a supervised intermediate. Occupancy is a dense voxel grid with class labels, so it forces the first stage to commit to scene structure and semantics before any rendering happens. The video stage then renders from a representation that already contains geometry, which is why the paper uses Gaussian-based Joint Rendering for that step, and why the LiDAR step, Prior-guided Sparse Modeling, can concentrate on sparsity rather than on the full scene.

For a downstream user this changes what a failed generation means. With a direct generator, a bad output tells you little. With UniScene, you can inspect the intermediate occupancy and tell whether the failure came from layout understanding or from rendering. That diagnostic value is real, and it is also why the occupancy model is the piece to test first.

A paper artefact with no releases, a permissive licence and a wide update gap

The release is generous in one respect and thin in others. The licence is Apache-2.0, which allows commercial use, modification and redistribution in a closed source product, with the usual obligations of keeping the licence and any NOTICE content intact. There is no CLA requirement stated in what is documented here, though the repository does include a pre-commit-config.yaml and a third_party directory, which usually means vendored components with their own licences you would need to check individually.

On maintenance, the last push was on 2026-05-24, and the repository has no GitHub releases at all. There is no versioned checkpoint release to pin against, so a user has to track the Hugging Face files and the git state together. The project is also visibly moving: the news section points to a second version of the work, UniScenev2, to a large-scale occupancy dataset called Nuplan-Occ, and to a separate world model project named OmniNWM, all dated March 2025. Those links tell you where the author's attention went after the CVPR 2025 paper.

Practical consequence: UniScene is best treated as a reproducible artefact for the paper's experiments rather than as a maintained library. Fork it, pin the commit, and keep your own notes on the config files you changed.

Editorial conclusion

UniScene belongs in the hands of an autonomous driving research group that needs correlated occupancy, video and LiDAR training data from one controllable layout and cannot get it from driving logs. It is the wrong choice for a product team that needs a stable data pipeline, since the release is a paper artefact with no tagged releases, a last push on 2026-05-24, and an install path that assumes an NVIDIA A100 or A800 with CUDA 12.1. Before adopting it, follow install.md, run the data_process/generate_occ.py step on the NuScenes trainval split, and confirm your own GPU has the memory for the occupancy model, because the documentation reports results only on that hardware.

Frequently asked questions

Official sources

1. Arlo0o/UniScene-Unified-Occupancy-centric-Driving-Scene-Generation on GitHub
2. Issues
3. License: Apache-2.0
4. README