NVlabs/nvdiffrec: Recovering Meshes, Materials and Lighting From Photos

What nvdiffrec optimizes, and for whom

nvdiffrec is the official code for the CVPR 2022 oral paper "Extracting Triangular 3D Models, Materials, and Lighting From Images". The problem it addresses is the gap between neural scene representations and the asset formats that downstream pipelines actually consume. A radiance field gives you a function you can query for colour along a ray; it does not give you a triangle mesh with per-vertex attributes, a set of material parameters and an environment map that a renderer can reload. nvdiffrec performs joint optimization of topology, materials and lighting from multi-view image observations, so the outputs are the three things named in the title rather than a density field.

The intended audience is narrow and the README is explicit about the hardware assumption: the approach is designed for high-end NVIDIA GPUs with large amounts of memory, and running on mid-range GPUs means reducing the batch size parameter in the .json files. This is research code released to reproduce a paper, not a product with a support contract. If you are evaluating it as a component in a production asset pipeline, the licence and the hardware floor matter more than the reconstruction quality.

Differentiable rendering, marching tetrahedra and the config-driven pipeline

The repository layout tells you how the system is divided. train.py is the entry point. configs/ holds JSON files that define each experiment. geometry/ and render/ hold the mesh representation and the differentiable rasterization code. data/ and dataset/ hold dataset handling and the download and preprocessing scripts. docker/ holds the image build and run scripts.

The differentiable renderer is not written from scratch. The README states that for differentiable marching tetrahedra the project adapted code from NVIDIA's Kaolin library, and that the renderutils library depends on nvdiffrast, which is installed from its own repository as a PyTorch extension rather than from PyPI. A later news entry, dated 2023-10-20, describes a version of renderutils written in slangpy that uses slang's autodiff instead of CUDA extensions with manually crafted forward and backward passes, with the same runtime performance as before; that version lives in the slang branch rather than on main. A second news entry, dated 2023-09-15, adds FlexiCubes isosurfacing support, with configs/bob_flexi.json given as the usage example and the FlexiCubes documentation as the reference.

The practical consequence of this architecture is that the interesting knobs are in JSON, not in command-line flags. Geometry resolution, batch size and the lighting assumptions are all config-level decisions, and switching between the included examples is a matter of pointing train.py at a different file. The trade-off is that the codebase carries a build step: the CUDA extensions must compile against your toolkit before anything runs, and that build is where most first attempts stall.

Installing nvdiffrec and running the genus 1 example

The README gives a one-time setup for Windows and states the requirements as Python 3.6+, VS2019+, Cuda 11.3+ and PyTorch 1.10+, tested in Anaconda3 with Python 3.9 and PyTorch 1.10. The example below uses CUDA 11.6 and creates the environment named dmodel. The CUDA toolkit is required to build the PyTorch extensions, and the PyTorch version you pick has to match the toolkit you installed.

conda create -n dmodel python=3.9
activate dmodel
conda install pytorch torchvision torchaudio cudatoolkit=11.6 -c pytorch -c conda-forge
pip install ninja imageio PyOpenGL glfw xatlas gdown
pip install git+https://github.com/NVlabs/nvdiffrast/
pip install --global-option="--no-networks" git+https://github.com/NVlabs/tiny-cuda-nn#subdirectory=bindings/torch
imageio_download_bin freeimage

Note that nvdiffrast and tiny-cuda-nn are installed straight from their git repositories, and that the tiny-cuda-nn install passes --no-networks to skip building the neural network bindings. In every new command prompt you reactivate the environment with `activate dmodel`.

The first real use is the genus 1 reconstruction example. From the repository root:

python train.py --config configs/bob.json

The README states that results are stored in the out folder, and that the Bob and Spot models were created and released into the public domain by Keenan Crane. If you want to watch progress while it trains, the README documents a display flag that it marks as supported on Windows only:

python train.py --config configs/bob.json --display-interval 20

The configs directory also ships spot.json for geometry, materials and lighting from image observations, spot_fixlight.json for the case where the environment lighting is assumed known, spot_metal.json for joint learning of materials and high frequency environment lighting to demonstrate split-sum, and the nerf_*.json and nerd_*.json configs that reproduce the paper's main results. The dataset configs need third-party data, which the README says can be fetched by running the download script from the data directory:

activate dmodel
cd data
python download_datasets.py

The README notes that individual licences apply to each dataset, and documents manual fallbacks if the script fails: unzipping the NeRF synthetic dataset archive into the nvdiffrec/data folder for the nerf_*.json configs, and cloning the ethiopianHead and moldGoldCape repositories into nvdiffrec/data/nerd and running scale_images.py to rescale them to 512 x 512 for the nerd_*.json configs.

Docker and multi-GPU, and what the README does not promise

For server use the README documents building an image from the docker directory and running it with GPU access:

cd docker
./make_image.sh nvdiffrec:v1

An interactive container is started with `docker run --gpus device=0 -it --rm -v /raid:/raid -it nvdiffrec:v1 bash`, and a detached training run with `docker run --gpus device=1 -d -v /raid:/raid -w=[path to the code] nvdiffrec:v1 python train.py --config configs/bob.json`. The volume mount and the working directory are left to the operator, which is a reasonable choice for research code and an inconvenience if you expect a turnkey image.

Multi-GPU training is documented as Linux only and the README labels it experimental, stating that all results in the paper were generated using a single GPU. The launcher is PyTorch DDP:

torchrun --nproc_per_node=4 train.py --config configs/bob.json

The honest reading of that note is that the four-GPU path is not the path the authors validated. If your reconstruction quality matters, the single-GPU configuration is the one the paper's numbers correspond to. Nothing in the README describes checkpoint resumption, distributed checkpointing, or how to recover a run that dies partway through, so plan for long single-shot jobs rather than restartable ones.

Where nvdiffrec is the wrong tool

The clearest limitation is the hardware floor. The README states the approach is designed for high-end NVIDIA GPUs with large amounts of memory, and the only remedy it offers for smaller cards is reducing the batch size parameter in the .json files. That is a memory knob, not a quality guarantee; the documentation does not say what reconstruction quality you get at reduced batch size, and there is no published guidance on how far down the batch size can go before results degrade.

Licensing is the second constraint, and it is not a footnote. The work is made available under the NVIDIA Source Code License, and the repository's licence field is reported as NOASSERTION rather than a recognized SPDX identifier. The README directs business inquiries to NVIDIA Research Licensing. If your evaluation criteria include an OSI-approved licence, this project does not meet them, and the datasets pull in their own separate terms on top. Nothing here is legal advice; read LICENSE.txt and the individual dataset licences against your own use case.

The third limitation is scope. nvdiffrec reconstructs one object under controlled multi-view capture, which is what the paper is about. It is not a general photogrammetry suite, it does not document a path for large scenes, and it does not document a CPU fallback. If you need to reconstruct a room, a city block, or a scene from casually captured phone video, this is not the tool the README describes.

nvdiffrec against instant-ngp and other neural reconstruction tools

The obvious alternative in the same research lineage is instant-ngp from NVIDIA, which is a frequent companion term in searches around this project. The difference in approach is the output representation and the optimization target. instant-ngp trains a multiresolution hash grid to represent a radiance field and renders it through a fully fused MLP; the artifact is a trained network and a fast renderer. nvdiffrec instead optimizes an explicit surface representation through differentiable marching tetrahedra and a differentiable rasterizer, and the artifact is a triangle mesh with material parameters and an environment map. Both consume multi-view images and both lean on NVIDIA GPU acceleration, but only one of them hands you geometry in a format a conventional renderer or DCC tool can open.

A second comparison worth making is between nvdiffrec's two isosurfacing paths. The original differentiable marching tetrahedra code is adapted from Kaolin. The 2023-09-15 news entry adds FlexiCubes, with configs/bob_flexi.json as the example config and the FlexiCubes repository as the documentation. If you are starting fresh, the FlexiCubes config is the more recent of the two options the README documents, though the README gives no comparison of the two on quality or runtime. Similarly, the slang branch replaces the CUDA extensions with a slangpy renderutils that relies on slang autodiff. The README claims the same runtime performance as before and says the code is substantially simpler; it does not claim better results. Choosing between main and slang is therefore a maintenance question about which build you would rather debug, not a quality question.

Editorial conclusion

Adopt nvdiffrec if you have a high-end NVIDIA GPU, a CUDA 11.3+ toolchain and a reason to reproduce or extend the CVPR 2022 method rather than ship a product, and keep the slang branch in mind if you want the renderutils rewrite. Do not adopt it if you need a permissive licence, a CPU-only or AMD path, or a supported library with versioned releases; the README states the code is designed for high-end NVIDIA GPUs and that mid-range cards require reducing the batch size in the .json configs. Before committing, verify three things in your own checkout: that the PyTorch extensions build against your CUDA toolkit, that your GPU has enough memory for the batch size in the config you intend to run, and that the NVIDIA Source Code License and the separate dataset licences fit how you plan to use the outputs.

Frequently asked questions

Official sources

1. Issues
2. NVlabs/nvdiffrec on GitHub
3. README