MonoScene: single-image 3D semantic scene completion, and what it costs to run

The gap MonoScene fills: dense occupancy from one camera

LiDAR gives you a sparse ring of returns. MonoScene's premise is that a single RGB image should be enough to produce a dense voxel grid where every cell carries a semantic label, including cells the camera never directly observed. The repository describes this as 3D semantic occupancy prediction, and the teaser assets show the target output on SemanticKITTI, KITTI-360 and NYUv2. The intended audience is narrow: researchers and engineers working on camera-only perception who need a published baseline they can train and evaluate, not a drop-in perception module for a product. The README points to a Hugging Face demo for anyone who wants to see the output before installing anything, which is the cheapest way to judge whether the prediction quality is relevant to your use case.

Architecture as the repository reveals it: relation matrices and a preprocessing stage

The most concrete architectural signal in the README is not a layer diagram but a build step. Both dataset pipelines run a preprocess.py script whose stated purpose is to generate labels at a lower scale, which are then used to compute the ground truth relation matrices. That tells you the training objective depends on a precomputed, downscaled representation of the ground truth, not on the raw voxel labels alone. If you modify the label definition or the voxel resolution, you inherit the preprocessing script as part of your change. The code is organized under monoscene/data/ with per-dataset subpackages for semantic_kitti and NYU, and the training entry point lives at monoscene/scripts/train_monoscene.py. Configuration is handled through a key-value interface on the command line, with keys like dataset, enable_log, kitti_root, kitti_preprocess_root, kitti_logdir, n_gpus and batch_size. The README does not document the model internals, so anyone needing to understand the network itself has to read the paper (arXiv 2112.00726) or the source.

Installation is a pinned-stack exercise, not a pip install

The README is explicit about the environment: Python 3.7, PyTorch 1.7.1, torchvision 0.8.2, torchaudio 0.7.2 and CUDA 10.2. The install sequence is six steps. Create the environment with conda create -y -n monoscene python=3.7, activate it, install the PyTorch trio from the pytorch channel, run pip install -r requirements.txt, then conda install -c bioconda tbb=2020.2, then pip install torchmetrics==0.6.0, and finally pip install -e ./ from inside the repository. Two of those steps are downgrades or out-of-band installs, which is a signal that the dependency graph was resolved against an older ecosystem and never relaxed. The README also notes that installation errors reported in issue 18 were fixed in June 2022, so the documented path is the corrected one. The bioconda channel for tbb is worth flagging: it pulls from a channel most vision projects do not use, and it is the step most likely to behave differently on a machine with an existing conda configuration.

Datasets, environment variables and the exact commands

For SemanticKITTI you download the Semantic Scene Completion dataset v1.1 (700 MB of voxel data), plus the KITTI Odometry calibration files and the 65 GB color archive. The README specifies the expected folder shape: a dataset directory containing poses and sequences. You then export KITTI_PREPROCESS and KITTI_ROOT, and run python monoscene/data/semantic_kitti/preprocess.py kitti_root=$KITTI_ROOT kitti_preprocess_root=$KITTI_PREPROCESS. Training follows with python monoscene/scripts/train_monoscene.py dataset=kitti enable_log=true kitti_root=$KITTI_ROOT kitti_preprocess_root=$KITTI_PREPROCESS kitti_logdir=$KITTI_LOG n_gpus=4 batch_size=4, which the README describes as four GPUs with one item per GPU. For NYUv2 the flow is the same shape with different keys: NYU_ROOT, NYU_PREPROCESS, a preprocess script under monoscene/data/NYU/, and a dataset=nyu training run. KITTI-360 is inference only. You download either the 128 GB train and val perspective images or the 1.5 GB test set, export KITTI_360_ROOT, and no preprocessing or training command is given. Pretrained checkpoints for SemanticKITTI and NYUv2 are hosted at rocq.inria.fr and are meant to be placed in /path/to/MonoScene/trained_models.

Where MonoScene stops being the right tool

The four-GPU, batch-size-4 training configuration is the clearest constraint. That is one sample per GPU, which means the effective batch is small and the memory footprint per sample is large. If you have a single consumer GPU, the README offers no reduced configuration, no gradient accumulation recipe and no smaller variant. The pinned stack is the second constraint: PyTorch 1.7.1 and CUDA 10.2 predate most current hardware and driver combinations, and torchmetrics is held at 0.6.0, so any evaluation code you write against a modern torchmetrics release will not run in this environment. Third, KITTI-360 support is inference only, so the repository cannot be used to train on that dataset without work the README does not describe. Finally, the README does not state inference latency, memory use or accuracy numbers in the text, so any performance expectation has to come from the paper or from your own run.

Alternatives and the difference in input assumption

The README itself lists related camera-only 3D occupancy prediction projects and links to PaSCo, the authors' later work on urban 3D panoptic scene completion with uncertainty awareness, and to OccAny, described as generalized unconstrained urban 3D occupancy. The meaningful difference is scope rather than architecture: MonoScene targets semantic scene completion from a single image, PaSCo extends the task to panoptic scene completion and attaches uncertainty estimates to the predictions, and OccAny is framed around generalization across urban settings. If your problem is panoptic labelling or you need a confidence signal per voxel, MonoScene does not provide either, and the authors' own later repositories are the natural starting point. If you need occupancy from LiDAR rather than a camera, MonoScene is addressing a different input modality entirely and the comparison is not meaningful.

Licence and the cost of keeping this running

MonoScene is released under Apache-2.0, which permits commercial use and modification provided the licence and notices are retained. That is the permissive end of the spectrum, but it applies to this repository only. The datasets are separate: SemanticKITTI, KITTI Odometry, KITTI-360 and NYUv2 each carry their own terms, and the 65 GB and 128 GB downloads have their own access conditions. Check those independently; nothing here is legal advice. Maintenance cost is the more practical concern. The last release is v0.1 from December 2021, and the dependency pins have not moved since. The repository is not archived, so it is still being touched, but the README's news entries are announcements of successor projects rather than upgrades to this one. Expect to own the environment: if a CUDA 10.2 toolchain is not available to you, adapting the code to a current PyTorch is your work, not the maintainers'.

Editorial conclusion

Adopt MonoScene if you need a reference implementation of monocular 3D semantic scene completion and can reproduce its pinned stack: Python 3.7, PyTorch 1.7.1, torchvision 0.8.2, torchaudio 0.7.2, cudatoolkit 10.2, tbb 2020.2 and torchmetrics 0.6.0. Do not adopt it if you need a current PyTorch release, a library with a stable API, or anything other than inference on KITTI-360, which is the only dataset the README covers for that purpose. Before committing, verify that your SemanticKITTI download matches the v1.1 layout the README expects, since the preprocessing script depends on it, and check that the published checkpoints still load under your torchmetrics version.

Official sources

1. astra-vision/MonoScene on GitHub
2. License: Apache-2.0
3. Project website
4. README
5. Releases