NeuralRecon-W: from a COLMAP workspace to a reconstructed mesh

A conference paper delivered as a pipeline

NeuralRecon-W is the code for Neural 3D Reconstruction in the Wild, published in the SIGGRAPH 2022 conference proceedings, and the project links a project page and the paper itself.

The author list is a useful guide to the lineage of the work, with Jiaming Sun, Xi Chen, Qianqian Wang, Zhengqi Li, Hadar Averbuch-Elor, Xiaowei Zhou, and Noah Snavely, spread across the groups that have shaped neural rendering over the last few years.

What is in the repository is a checklist with every box ticked. The training code, described as the reconstruction code, is there. So is the toolkit and pipeline for reproducing the evaluation results on the proposed Heritage-Recon dataset. And so is the configuration for reconstructing generic outdoor and indoor scenes, which is the part that turns a paper into a tool.

That third item is the one that matters for a reader who is not reproducing the paper. The evaluation pipeline only works on the four scenes the dataset contains, while the generic configuration is what you would use for a building you photographed yourself.

The repository layout follows the structure of a PyTorch Lightning project. There is a training entry point, a models directory, a losses file, a metrics file, a rendering directory, an evaluations directory, a lightning_modules directory, a datasets directory, a config directory, a scripts directory of shell entry points, a tools directory for data preparation, and a utils directory. The environment is described in a conda environment file at the root, and the requirements file pins every library.

The project also says that parts of the code are derived from two other public implementations, nerf_pl and NeuS, and thanks their authors, which is worth knowing before you read the loss functions and wonder why they look familiar.

The environment, and the model you have to fetch first

The installation is a conda environment and one script, and the script is the part that is easy to miss.

conda env create -f environment.yaml
conda activate neuconw
scripts/download_sem_model.sh

The third line is not optional. The pipeline includes semantic segmentation, the download script fetches the model that the semantic map generation depends on, and without it the data preparation step cannot run. It is the kind of dependency that a readme lists in a single line and that costs an afternoon to discover when a script fails with a missing weight file.

The dependency list in the repository is more informative than the environment file, because it shows what the pipeline actually does. The training side is built on PyTorch Lightning, with the metrics library, the einops rearrangement helpers, the Kornia computer vision library, and a perceptual loss implementation, which is the usual set for a neural rendering loss. The data side uses OpenCV, imageio with its ffmpeg backend, h5py for the cached arrays, pandas, matplotlib, and the plotting library for the phototourism style visualisations.

The mesh side has its own group of libraries, which is what the extraction and the evaluation need: a marching cubes implementation, the COLLADA file format library, the trimesh geometry library, a windowing toolkit, a graph library, scikit-image, a renderer, and a distributed computing library that one of the utility scripts needs for reprojection filtering.

The point cloud side adds the plyfile reader and Open3D, and the last group is the segmentation stack, with a compiled extension installed from a wheel index built for one specific CUDA and Torch combination, plus the segmentation package itself.

Heritage-Recon and the four scenes it contains

The evaluation dataset is called Heritage-Recon, and it is distributed as a folder that you download and place under a data directory in the repository.

mkdir data && cd data
gdown --id 1eZvmk4GQkrRKUNZpagZEIY_z8Lsdw94v

The download uses a command line tool for Google Drive, because the dataset is hosted there and the folder is too large for a single link that a browser can fetch conveniently.

Four scenes make up the dataset, and their names appear in the ray cache generation step, which loops over them and calls the data generation script for each one.

for SCENE_NAME in brandenburg_gate lincoln_memorial palacio_de_bellas_artes pantheon_exterior; do
  scripts/data_generation.sh data/heritage-recon/${SCENE_NAME}
done

The result of that step is a ray cache: for each scene, a split directory holding the ray tensors and the colour tensors, with a metadata file for each and a separate split subdirectory per chunk. The tensors are stored in h5 files, which is why that library is in the requirements, and the split layout is what lets the training loop read the data in pieces rather than loading a whole scene into memory.

That caching step is the reason the pipeline has a preparation stage at all. The training does not read images from disk and re-derive the camera rays on the fly, it reads a precomputed cache of rays and their matching colours, and the cache is built once per scene from the reconstructed camera poses.

For a reader deciding whether this is worth running, the four scenes tell you the scale of the problem the paper addresses. They are large, well-known outdoor heritage sites, photographed from the ground, which is the hard case for a method that has to work in the wild rather than on a controlled capture rig.

Training, extracting a mesh, and scoring it

Training a scene is a single script call with four arguments.

scripts/train.sh $EXP_NAME config/train_${SCENE_NAME}.yaml $NUM_GPU $NUM_NODE

The experiment name is yours to choose, the configuration names the scene, and the last two are the number of GPUs and the number of nodes, so the same entry point covers a single machine and a multi node run. A configuration file per scene lives in the config directory, and there is a separate one for indoor scenes.

Extracting a mesh from a checkpoint is the next script, and the last argument is the evaluation level.

scripts/sdf_extract.sh $EXP_NAME config/train_${SCENE_NAME}.yaml $CKPT_PATH 10

The extracted meshes are written to a results directory inside the project, and then the evaluation pipeline scores a mesh against the ground truth for that scene.

scripts/eval_pipeline.sh $SCENE_NAME $MESH_PATH

The evaluation results are written next to the mesh that was evaluated, which is a small convenience that matters when you are comparing a dozen checkpoints and want the numbers to travel with the geometry they describe.

The pipeline is therefore four scripts with clear boundaries between them: prepare the data, train, extract, evaluate. Each one takes explicit paths, and each one leaves an artefact on disk, so a run can be interrupted at any of the three boundaries and resumed from the last one.

Reconstructing a scene you captured yourself

The generic path starts from a COLMAP workspace, which is the structure a structure from motion pipeline produces, and the project accepts that as the input format.

The automatic route is a single script.

scripts/preprocess_data.sh

The manual route exists because the automatic one picks the reconstruction region for you, and sometimes it picks the wrong one. The manual path has three steps, and the first generates the semantic maps with a tool in the tools directory.

python tools/prepare_data/prepare_semantic_maps.py --root_dir $WORKSPACE_PATH --gpu 0

The second step writes a scene metadata file into the workspace, and it is the file that decides the scale and the extent of the reconstruction. The scene is normalised into a unit sphere, so the file needs an origin and a radius, and the documentation suggests taking the structure from motion key points to work out the centre and the extent, with a bounding box set to cover the region of interest.

{
    name: brandenburg_gate, # scene name
    origin: [ 0.568699, -0.0935532, 6.28958 ], 
    radius: 4.6,
    eval_bbx: [[-14.95992661, -1.97035599, -16.59869957],[48.60944366, 30.66258621, 12.81980324]],
    voxel_size: 0.25,
    min_track_length: 10,
    # The following configuration is only used in evaluation, can be ignored for your own scene
    sfm2gt: [[1, 0, 0, 0],
            [ 0, 1, 0, 0],
            [ 0, 0, 1, 0],
            [ 0, 0, 0, 1]],
}

The last entry is a transform matrix used only in the evaluation, and the file says so, which means you can ignore it for your own scene.

The third step generates the cache, with the same script the dataset reproduction uses, and the resulting workspace has a particular shape: the scene name as a directory, a table file, the cache directory with its splits, the metadata file, the dense reconstruction with its sparse model files, and the semantic maps as images.

Training then uses the generic configuration, with the dataset root pointing at the workspace, and there is one switch to remember for indoor scenes, the inside outside setting in the signed distance configuration, which the documentation points at the indoor configuration file for.

The dependency pins, and what they imply

The requirements file is the most consequential document in the repository, and reading it tells you when this code was written and what it will cost you to run it.

The training stack is pinned to a PyTorch Lightning release from the 1.4 series, with the metrics library at 0.7, the segmentation extension built from a wheel index for CUDA 11.3 and Torch 1.11, and the perception loss at 0.1.3. The image and mesh libraries are of the same vintage, with the mesh library at 3.9, the point cloud library at 0.12, and the OpenCV build at 4.5.

Two consequences follow. The first is that the segmentation extension is not a source build but a prebuilt wheel for one specific combination of graphics driver version and Torch version, and if your machine does not match that combination you are compiling a compiled extension from source against a Torch that the project was never tested with. The second is that the whole stack predates the current releases of most of its libraries by several years, so the environment is only reproducible if you can find the old packages, which is what the conda environment file is for.

That is not a criticism of the project. A conference artefact has to be pinned to the versions the paper was produced with, and a floating dependency list would make the numbers in the paper unreproducible. It is a fact about what you are adopting: this is a snapshot of a 2022 pipeline, and the modern equivalent of using it is to reimplement it against current libraries rather than to expect the environment to build itself.

The other practical point is the scale. The training script takes a number of GPUs and a number of nodes, which is a signal about the hardware the paper assumed, and the ray cache exists to keep the data pipeline from becoming the bottleneck at that scale.

Editorial conclusion

NeuralRecon-W fits a researcher who wants to reproduce the numbers in the paper, or to apply the same pipeline to a scene they have captured themselves with a COLMAP workflow, and who has a GPU environment prepared for the pinned versions. It does not fit a team that wants a current, maintained reconstruction stack, because the dependency set is from the PyTorch 1.11 era and includes a compiled extension built for one specific CUDA and Torch combination. Verify first that the mmcv wheel index in the requirements matches your CUDA and Torch versions, since a mismatch there is the first thing that breaks, that you have the semantic segmentation model the download script fetches before the training will start, and that the scene you want to reconstruct has a COLMAP workspace with a sensible bounding box, because that box decides what gets reconstructed.

Frequently asked questions

Official sources

1. Issues
2. README
3. zju3dv/NeuralRecon-W on GitHub