Skyfall-GS: Building Explorable 3D Urban Scenes from Satellite Imagery

What Skyfall-GS actually solves, and for whom

The stated problem is not rendering. It is data. The paper abstract in the README argues that generalizable 3D urban scene generators are held back by the absence of large-scale, high-quality real-world 3D scans for training, and that satellite imagery offers coarse geometry at a scale no ground survey matches. Skyfall-GS takes that route: satellite imagery supplies the geometry, an open-domain diffusion model supplies close-up appearance, and the result is meant to be an explorable 3D Gaussian splatting scene without any 3D annotation step.

The audience is narrow. This is a research release tied to an ECCV 2026 paper, with a project page, an arXiv link and Hugging Face datasets and PLY models. Someone who wants a viewer or a hosted service will not find one here. Someone who wants to reproduce a paper, run ablations on the JAX and NYC datasets, or build on the two-stage training code is the intended user. The repository is Python, licensed Apache-2.0, and the last push was on 2026-06-20, so it is a snapshot of a paper release rather than a project with a maintenance cadence you can plan around.

Two training stages and where the diffusion model enters

The pipeline is split into Stage 1, reconstruction, and Stage 2, synthesis with Iterative Dataset Update (IDU). Stage 1 trains a Gaussian splatting scene from the satellite-derived inputs, with arguments that expose the geometry side of the trade-off: a pseudo-depth loss enabled by --lambda_pseudo_depth, densification controlled by --densify_until_iter and --densify_grad_threshold, and opacity regularisation via --lambda_opacity and --opacity_reset_interval. Stage 2 loads the Stage 1 checkpoint through --start_checkpoint and refines it with the iterative dataset update strategy, which the README describes as progressively enhancing geometric completeness and photorealistic textures.

The data flow is visible in the expected directory layout. A dataset directory holds images/, optional masks/ in both .npy and .png form, transforms_train.json and transforms_test.json for camera parameters, and points3D.txt for the initial point cloud. The masks are optional: the README states that when they are absent, all non-black pixels are treated as valid. That default matters, because satellite tiles frequently contain black borders, and the mask convention is the only mechanism the documentation gives for excluding them.

What the README does not show is the diffusion side in any operational detail. The requirements file pins diffusers==0.30.1, transformers==4.46.3 and accelerate==1.0.1, which is consistent with a diffusion component, but no stage-2 command line, model identifier or sampling configuration appears in the excerpt. Treat the synthesis stage as documented by the paper rather than by the repository.

Installing Skyfall-GS with Conda and CUDA 12.8

The README gives a linear install. Clone with submodules, create a Python 3.10 Conda environment, install the CUDA toolkit, then the Python requirements, then three submodules that provide the rasterizer, the kNN helper and the fused SSIM loss. The submodule flag matters because the rasterizer lives in submodules/ and the repository root lists .gitmodules.

git clone --recurse-submodules https://github.com/jayin92/Skyfall-GS.git
cd Skyfall-GS
conda create -y -n skyfall-gs python=3.10
conda activate skyfall-gs
conda install cuda-toolkit=12.8 cuda-nvcc=12.8 -c nvidia
pip install -r requirements.txt
pip install --force-reinstall torch torchvision torchaudio
pip install submodules/diff-gaussian-rasterization-depth
pip install submodules/simple-knn
pip install submodules/fused-ssim

The force-reinstall of torch is the step to watch. It runs after requirements.txt, so it overrides whatever torch version the requirements resolved to. The requirements file itself pins several packages exactly (accelerate==1.0.1, diffusers==0.30.1, huggingface-hub==0.33.4, transformers==4.46.3, tokenizers==0.20.3) but does not pin torch, so the CUDA build you end up with is whatever the default index serves for your platform at install time. If the submodule builds fail afterwards, that mismatch is the first thing to check.

A first real run needs data. Download the JAX and NYC archives from Hugging Face or Google Drive, unzip them into data/, and confirm the layout matches the documented structure before training.

unzip datasets_JAX.zip
unzip datasets_NYC.zip
python train.py -s ./data/datasets_JAX/JAX_068/ -m ./outputs/JAX/JAX_068 --eval --port 6209 --kernel_size 0.1 --resolution 1 --sh_degree 1 --appearance_enabled --lambda_depth 0 --lambda_opacity 10 --densify_until_iter 21000 --densify_grad_threshold 0.0001 --lambda_pseudo_depth 0.5 --start_sample_pseudo 1000 --end_sample_pseudo 21000 --size_threshold 20 --scaling_lr 0.001 --rotation_lr 0.001 --opacity_reset_interval 3000 --sample_pseudo_interval 10

That is the Stage 1 command from the README, reproduced with the flags it lists. The --port 6209 value is the one the README uses; it is a fixed choice in the example, not a documented default. Outputs land under the -m path, and the Stage 2 example expects a checkpoint named chkpnt30000.pth inside it.

Custom data is where the README stops helping

Training on your own imagery is listed as supported from two sources, COLMAP reconstructions and satellite imagery, but the README defers the actual preprocessing to a separate SatelliteSfM repository. That is a real gap. The format requirements are stated (images/, masks/, transforms_train.json, transforms_test.json, points3D.txt), yet the steps that produce a valid transforms file or a usable points3D.txt from raw satellite captures are not in this repository's documentation. If you cannot already produce that structure, Skyfall-GS is the wrong entry point and SatelliteSfM is the one to read first.

The second limitation is hardware. The install pins cuda-toolkit=12.8 and cuda-nvcc=12.8, and the training stack compiles three CUDA submodules. There is no CPU fallback described, no prebuilt wheel path, and no Docker image in the repository listing. On a machine without a matching CUDA toolchain, the install stops at the submodule builds.

The third is that nothing in the README describes what happens when a run diverges. There is no rollback procedure, no checkpoint retention policy beyond the single chkpnt30000.pth referenced in the Stage 2 command, and no guidance on resuming. For a two-stage pipeline where Stage 2 depends on a Stage 1 artifact, that is the operational detail most likely to cost you a day.

How it differs from a COLMAP plus 3DGS workflow

The obvious comparison is the standard photogrammetry route: run COLMAP over overlapping photographs to recover poses and a sparse point cloud, then train a Gaussian splatting model on those images. That pipeline assumes you can photograph the scene. It gives you accurate geometry where the photographs cover it and nothing where they do not, and it does not scale to a city block you cannot walk.

Skyfall-GS inverts the input. Satellite imagery covers the whole area at once but resolves facades poorly, so the framework leans on a diffusion model to invent close-up appearance and on the curriculum-driven refinement to keep that invented detail consistent across views. The README claims improved cross-view consistent geometry and more realistic textures compared to state-of-the-art approaches, which is the paper's claim, not something this article can verify. The trade-off is the usual one for generative completion: you get coverage and plausible texture, and you accept that fine detail is synthesised rather than measured. A survey-grade reconstruction and a Skyfall-GS scene are not interchangeable, and the choice depends on whether missing data is acceptable.

Licence, dependencies and the cost of keeping it running

The repository is Apache-2.0, but the root listing also contains LICENSE_inria.md alongside the main LICENSE file. The Inria licence is associated with the original 3D Gaussian Splatting rasterizer, and the submodule installed here is diff-gaussian-rasterization-depth, a fork of that code. Two licence files in one repository means the terms are not uniform across the tree, and anyone redistributing a build should read both rather than assuming Apache-2.0 covers everything. That is a description of the files present, not legal advice.

Upgrade cost is dominated by the pinned dependencies. Requirements pin accelerate, diffusers, huggingface-hub, transformers and tokenizers to exact versions, and the install then force-reinstalls torch outside that pinning. Moving any one of those forward means re-checking the others, and the repository shows no release tags, so there is no versioned upgrade path to follow. The last push was on 2026-06-20; there is no evidence of a maintenance schedule, so plan on tracking changes yourself rather than waiting for a release.

Editorial conclusion

Adopt Skyfall-GS if you are a graphics or remote-sensing researcher with a CUDA 12.8 machine, a Conda workflow, and a reason to compare against COLMAP-plus-3DGS baselines on the JAX or NYC datasets. Do not adopt it if you need a stable product, a supported API, or a documented path for custom data, because the README points custom preprocessing at a separate SatelliteSfM repository and documents neither rollback nor upgrade steps. Verify three things before committing: that your GPU can run the CUDA 12.8 toolkit the installation asks for, that the submodules diff-gaussian-rasterization-depth, simple-knn and fused-ssim compile on your driver, and that the Apache-2.0 licence plus the separate LICENSE_inria.md file match how you intend to use the rasterizer.

Frequently asked questions

Official sources

1. Issues
2. jayin92/Skyfall-GS on GitHub
3. License: Apache-2.0
4. Project website
5. README