nvdiffrec: inverse rendering that outputs a triangle mesh, not a radiance field

The output is a mesh with materials, not a neural field

Most image-to-3D work in the years around this paper answers a rendering question: given a set of photographs, synthesize a new view. nvdiffrec answers a content question. It recovers a triangular mesh, a set of material parameters, and an environment map, all optimized jointly against the input images. The repository description states this directly: the code extracts triangular 3D models, materials and lighting from images. The intended user is therefore not someone building a viewer. It is someone who needs an asset, a piece of geometry with a surface description that a conventional renderer can consume.

That distinction drives everything else about the project. A radiance field is a function you sample; it has no canonical topology and no texture atlas. A mesh with materials is a file. The README's own examples are telling: spot.json and bob.json are reconstructions of public-domain models by Keenan Crane, and the repository ships start-of-training and end-of-training images so you can see the initial guess and the final result side by side. The framing is asset recovery, and the paper title says so.

The cost of that framing is that the optimization has to make discrete decisions. Choosing triangles is not a smooth operation, and the project's answer to that is the interesting part of its design.

Marching tetrahedra as the differentiable topology step

The README credits NVIDIA's Kaolin library for the differentiable marching tetrahedra code that nvdiffrec adapted. This is the mechanism that lets topology be optimized rather than fixed. A density or signed-distance field is defined over a grid, the isosurface is extracted, and the extraction is written so that gradients flow back to the field values. Because the surface is produced by an isosurfacing step rather than by deforming a template mesh, the optimizer can in principle add or remove geometry during training instead of only moving vertices around.

Rendering then happens through nvdiffrast, also an NVIDIA project, installed straight from git in the setup instructions. The rendered image is compared against the observed photographs, and the loss propagates back through the rasterizer into the geometry, the material parameters and the lighting. That joint loop is the whole point of the paper: separating geometry estimation from material estimation tends to bake lighting into albedo, and the README's config list shows the authors care about this. spot.json learns geometry, materials and lighting together. spot_fixlight.json assumes known environment lighting. spot_metal.json targets joint learning of materials and high-frequency environment lighting, and the README describes it as an example to showcase split-sum, the approximation used to make the specular integral tractable.

The repository has since added an alternative isosurfacing path. A September 2023 note announces support for the FlexiCubes technique, with configs/bob_flexi.json as the usage example and a pointer to the separate FlexiCubes documentation. So there are now two ways to get a surface out of the field, and the choice is expressed as a config file rather than as a code fork.

Getting it running means building CUDA extensions first

The installation section is the least forgiving part of the README, and it is worth reading before anything else. Requirements are Python 3.6+, Visual Studio 2019 or newer, CUDA 11.3+ and PyTorch 1.10+. The authors state they tested in Anaconda3 with Python 3.9 and PyTorch 1.10, which tells you the combination that is known to work.

The one-time Windows setup creates a conda environment, installs a CUDA-matched PyTorch, then installs ninja, imageio, PyOpenGL, glfw, xatlas and gdown. Two of the next lines are the ones that fail in practice for people: nvdiffrast is installed from git, and tiny-cuda-nn is installed from git with the --global-option="--no-networks" flag and the bindings/torch subdirectory. Both build against the local CUDA toolkit. If your toolkit version and your PyTorch wheel disagree, this is where you find out. A final imageio_download_bin freeimage call fetches a binary the image pipeline needs.

Training itself is a single command. python train.py --config configs/bob.json runs the simple genus 1 example. Results land in the out folder. The README notes that --display-interval 20 visualizes training progress and that this is only supported on Windows. Multi-GPU is available through torchrun with --nproc_per_node, but the README flags it as Linux only and experimental, adding that all results in the paper were generated on a single GPU. Treat the multi-GPU path as a convenience, not as a validated configuration.

Datasets are handled by data/download_datasets.py, which downloads and pre-processes the NeRF synthetic and NeRD datasets used by the nerf_*.json and nerd_*.json configs. The README also documents the manual route, including cloning the Ethiopian Head and Mold Gold Cape NeRD datasets and running scale_images.py to rescale them to 512 by 512. For servers, a docker directory contains make_image.sh, and the README gives both an interactive docker run line with --gpus device=0 and a detached variant.

Memory is the binding constraint

The README is unusually blunt about hardware: the approach is designed for high-end NVIDIA GPUs with large amounts of memory, and to run on mid-range GPUs you should reduce the batch size parameter in the .json files. That sentence is doing a lot of work. It means the default configs are not portable, that the failure mode on a smaller card is an out-of-memory error rather than a graceful degradation, and that the batch size key in each config is the dial you turn. There is no stated guidance on how far you can reduce it before the optimization stops converging, which is the kind of thing you can only determine by running it.

The second constraint is the build chain. Because nvdiffrast and tiny-cuda-nn are compiled from source against your CUDA toolkit, the project inherits every version-compatibility problem of that ecosystem. A machine with a mismatched toolkit, or one where you cannot install a compiler, is not a machine where this runs. There is no published wheel that removes this step.

The third is the licence. The repository is classified NOASSERTION on the hosting side, and the README points to the NVIDIA Source Code License with a copyright notice for NVIDIA Corporation and a business-inquiries link. Source-available is not the same as permissive. If you are evaluating this for a commercial pipeline, the licence text is the first document to read, not the last. Nothing here is legal advice, and the terms are not summarized in the README beyond the link.

A fourth limitation is quieter: the release history. There are no retrieved releases, and the most recent activity in the README is a 2023 note about a slang branch and a FlexiCubes config. The slang branch is described as a rewrite of renderutils using slangpy's autodiff instead of hand-written CUDA forward and backward passes, with the same runtime performance claimed and substantially simpler code. That branch is where the maintenance story lives, and it is a branch, not main.

How this differs from NeRF-style view synthesis

The obvious comparison is NeRF, and the README makes it for you by using the NeRF synthetic dataset for the nerf_*.json configs. The difference in approach is structural rather than incremental. NeRF fits a coordinate network that maps position and direction to color and density, then renders by ray marching. The output is the network. To get geometry you have to extract it as a post-process, and materials are not part of the model at all.

nvdiffrec optimizes an explicit surface representation through differentiable isosurfacing and rasterizes it with nvdiffrast. The material and lighting parameters are first-class variables in the optimization, which is why spot.json can recover all three jointly and why spot_metal.json can target high-frequency environment lighting with a split-sum approximation. If your downstream consumer is a game engine or an offline renderer, that difference is the entire value proposition: you get a mesh and a texture, not a checkpoint you still have to convert.

What you give up is the part NeRF is good at. A radiance field handles view-dependent effects and fuzzy volumetric content without needing a watertight surface, and it does not require the isosurface extraction step to succeed. If your scene is smoke, fur, or anything without a well-defined boundary, forcing it through marching tetrahedra is the wrong shape of tool. The same applies if you only ever need to render novel views from the same pipeline. The mesh is then overhead, not output.

It is also worth noting that nvdiffrec's own evaluation depends on third-party data with its own terms. The README says individual licenses apply to each dataset, which means a reproduction run is not a single-licence exercise.

What to check before you commit a project to it

Start with the build. Run the one-time setup on the exact machine and CUDA version you intend to use, and confirm that both git installs complete. If nvdiffrast or tiny-cuda-nn fails to compile, nothing downstream matters, and the README offers no fallback path.

Then run the smallest config. python train.py --config configs/bob.json is the genus 1 example and the least demanding entry point in the repository. Watch memory. If it does not fit, the README's instruction is to lower the batch size in the JSON, so open configs/bob.json and find that key before assuming the project is out of reach.

Then decide which surface extraction you want. The default path uses the marching tetrahedra code adapted from Kaolin. The FlexiCubes path is exercised by configs/bob_flexi.json. Picking between them is a config change, but the two are documented in different places, and the FlexiCubes details live in a separate repository.

Finally, read the licence and the dataset terms together. The NVIDIA Source Code License governs the code; the NeRF synthetic and NeRD datasets carry their own conditions, and the NeRD material is described in the README as real-world photogrammetry with manually annotated segmentation masks, which is a different quality regime from the synthetic set. If your use is commercial, the business-inquiries link in the README is the route the authors point to. The slang branch is the other thing to keep an eye on if you care about how much hand-written CUDA you are maintaining, since it replaces those forward and backward passes with slangpy autodiff while claiming the same runtime performance.

Editorial conclusion

Adopt nvdiffrec if your deliverable is a triangular mesh with per-texel materials and an environment map, and you have a high-end NVIDIA GPU plus the CUDA toolchain to build nvdiffrast and tiny-cuda-nn. Do not adopt it if you only need novel-view synthesis, if you cannot accept the NVIDIA Source Code License, or if you are working on a machine where building PyTorch CUDA extensions is not an option. Before committing, verify three things on your own hardware: that the extension build completes under your CUDA version, that the batch size in the config you intend to use fits your GPU memory, and that the licence terms cover your intended use.

Official sources

1. Issues
2. NVlabs/nvdiffrec on GitHub
3. README