CityDreamer: generating unbounded 3D cities from satellite and street data

Three generators, not one model

CityDreamer's architecture is compositional, and the composition shows up in the artefacts you have to download. The pretrained models come as three separate checkpoints: an Unbounded Layout Generator, a Background Stuff Generator, and a Building Instance Generator. At inference time both the demo and the command line tool load them by default from three specific files, one per role.

The layout model decides the city plan, the background model produces the non-building content, and the building model places individual building instances. The training section makes the same split explicit for the layout component, which consists of two networks, a VQVAE and a Sampler. The VQVAE is trained first, and the Sampler training run then points at the checkpoint the first run produced.

That structure is the reason the project can claim unbounded output. Because the layout is generated rather than fixed, the scene does not have the extent limit of a finite captured dataset, and the sampler's output drives a rendering that continues past the boundary of any single source image. The cost of that design is that you are running three learned models per frame rather than one, and the paper's own framing calls it a compositional generative model of unbounded 3D cities rather than a scene generator in the usual single-model sense.

The installation expects CUDA 11.7 and a compile step

The project assumes you already have CUDA and PyTorch in your Python or Anaconda environment, and it states the tested configuration: PyTorch 1.13.1 with CUDA 11.7 in Python 3.8. The suggested PyTorch install pins the CUDA 11.7 build:

pip install torch==1.13.1+cu117 torchvision==0.14.1+cu117 --extra-index-url https://download.pytorch.org/whl/cu117

The source install then sets a home directory variable that later steps depend on, which is the first thing to notice because every later command references it:

git clone https://github.com/hzxie/city-dreamer
cd city-dreamer
CITY_DREAMER_HOME=`pwd`
pip install -r requirements.txt

The third stage is the one that catches people out. CUDA extensions live under extensions/ and are compiled and installed by looping over each subdirectory, so every subdirectory in that tree becomes a pip install of its own:

cd $CITY_DREAMER_HOME/extensions
for e in `ls -d */`
do
  cd $CITY_DREAMER_HOME/extensions/$e
  pip install .
done

The dependency list in requirements.txt includes mayavi and xvfbwrapper alongside the usual torch, torchvision, opencv-python, numpy and scipy, plus wandb and tensorboard for experiment tracking and a git install of perlin-numpy. The presence of a visualisation stack in the requirements means a headless environment needs the xvfb wrapper to behave, which is worth checking before you start a build.

Patching is how the project fits a 24GB GPU

Both the demo and the command line tool take runtime arguments called --patch_height and --patch_width, which divide images into patches of a given size. The documented setting for a single NVIDIA RTX 3090 with 24GB of VRAM is a patch size of 5 for both dimensions, and the instruction is to adjust the values to match your own GPU's memory.

This is the central memory strategy of the project, and it has a direct consequence for output quality. Dividing an image into patches means the model never processes the full frame at once, which is what makes a heavy generative pipeline runnable on a single high-end consumer card rather than a multi-GPU node. The trade-off is spatial consistency: the model reasons about local tiles, and the boundaries between them are where artefacts appear, so a smaller patch for a smaller GPU is not a free adjustment.

Note the GPU requirement in context. A 24GB card is described as the reference configuration, not a minimum, and the training commands go considerably beyond that with four GPUs for the VQVAE, two for the Sampler and eight for the background generator. A reader considering only inference is looking at the RTX 3090 figure; a reader considering training is looking at a multi-node budget, and the two are different conversations.

Two ways to run inference, one browser and one video file

Inference has two entry points that share the same checkpoints and the same patch arguments. The iterative demo is started with:

python3 demo/run.py

and you then open http://localhost:3186 in your browser. The demo folder holds run.py, an index.html and a static/ directory, and the flask and flask-executor dependencies in requirements.txt are what serve it. Because the demo is interactive, you can steer the generation and watch the city extend, which is the natural way to understand what the layout and building models are doing.

The command line interface runs the same pipeline without a browser:

python3 scripts/inference.py

and writes the result to `output/rendering.mp4`. That output path is worth noting because it sits in the same output/ directory as the three default checkpoints, so the directory that holds your weights also collects your results.

Both scripts accept a different checkpoint location through runtime arguments if you do not want to rely on the default output/ paths, and the same arguments apply to the dataset directories in training. This is a research codebase with one main entry point at the root, run.py, and separate scripts/ for dataset preparation and inference, which is the layout you would expect from a project that grew around a paper.

Reproducing the training data needs about 1TB of disk

The two datasets are published separately. The OSM dataset supplies the city layout, the GoogleEarth dataset supplies the appearance, and by default all scripts load them from ./data/osm and ./data/ges. Both can be pointed elsewhere with runtime arguments.

The obstacle is the instance segmentation annotation for the GoogleEarth dataset, which is not shipped ready to use. It has to be generated in two stages, and the documentation puts the disk requirement at approximately 1TB.

The first stage generates semantic segmentation using SEEM, the Segment-Everything-Everywhere-All-At-Once model, which is cloned at a pinned v1.0 tag and moved next to the CityDreamer checkout. The commands then edit SEEM's requirements file to strip the PyTorch 2.1.0 dependency, with a comment noting that PyTorch 1.13.1 also works for SEEM, and install its dependency files before returning to the CityDreamer tree to run the segmentation script:

cd $CITY_DREAMER_HOME
python3 scripts/footage_segmentation.py

The second stage runs the instance segmentation step:

cd $CITY_DREAMER_HOME
python3 scripts/dataset_generator.py

The SEEM step is a separate research dependency with its own version constraints, which is the least controlled part of the whole pipeline. A 1TB disk requirement for a 2024 dataset is also the kind of number that has to be re-checked against what the download actually contains today.

Training is staged across three generator runs

The training recipes are given as numbered torchrun invocations, and the numbering tells you the dependency order. Step 0x01 trains the VQVAE of the layout generator with four GPUs, and step 0x02 trains the Sampler with two GPUs, pointing at the checkpoint the VQVAE run left behind under output/checkpoints. Step 0x03 trains the background generator with eight GPUs.

The background generator additionally needs a configuration edit before it runs, and the README gives the exact four lines that must match, which set the building mode flag to False and scale the reconstruction, perceptual and GAN loss factors:

cfg.NETWORK.GANCRAFT.BUILDING_MODE               = False
cfg.TRAIN.GANCRAFT.REC_LOSS_FACTOR               = 10
cfg.TRAIN.GANCRAFT.PERCEPTUAL_LOSS_FACTOR        = 10
cfg.TRAIN.GANCRAFT.GAN_LOSS_FACTOR               = 0.5

The alignment with spaces in those lines is part of the file as given, which is a small sign that the author expects them to be pasted rather than retyped. The overall picture is a staged pipeline where each stage consumes the previous stage's output, run with torchrun at a fixed GPU count, tracked with wandb and tensorboard.

The consequence for anyone planning a training run is that this is not a single-command job. It is a sequence with checkpoints between stages, a configuration change before the third stage, and eight GPUs at the widest point of the sequence.

A custom licence and a research project that grew a following

The repository is not under a licence identifier GitHub can classify, so the LICENSE file at the root is the one to read. This is the same situation as a vendor repository with bespoke terms, and the practical step is to check the file before you use the code or the released weights in anything that ships, rather than relying on a badge.

On the research side, the project has continued past its paper. The changelog records the training code released on 2024-06-10, the testing code on 2024-03-28, the Hugging Face demo on 2024-03-03, the OSM and GoogleEarth datasets on 2024-02-27, and the repository created on 2023-08-15. A 2025-09-01 entry points to a follow-up work, CityDreamer4D, accepted to TPAMI with its own code repository.

That follow-up is the clearest signal about where this work went. The city-dreamer repository is the 2024 model, it has no GitHub releases, and the last push was on 2026-09-07. The project is not archived, so the code remains usable and the trajectory is toward the 4D version. Anyone deciding today between the two should look at what CityDreamer4D needs, because that is where the authors put their subsequent effort.

Editorial conclusion

Adopt CityDreamer if you need unbounded procedural 3D city scenes for research, simulation or visualisation, and if you have the hardware to match, since the documented setup expects CUDA 11.7 with PyTorch 1.13.1, a 24GB GPU at the documented patch setting, and about 1TB of disk to reproduce the training annotations. Do not adopt it for a lightweight asset pipeline or a CI job, because the pipeline compiles CUDA extensions, expects multi-GPU training, and ships no tagged releases, so nothing pins a known-good state. Before you commit, download the three checkpoints into the output/ directory as sampler.pth, gancraft-bg.pth and gancraft-fg.pth, run python3 demo/run.py to confirm the browser demo at http://localhost:3186 works, and read the LICENSE file in the repository root, since the project uses a custom licence that GitHub cannot classify.

Frequently asked questions

Official sources

1. hzxie/CityDreamer on GitHub
2. Issues
3. Project website
4. README