Neural Body renders a moving human from a structured latent code

Two papers, one repository

The project page lists two publications, and the distinction is worth keeping straight. The conference paper is Neural Body: Implicit Neural Representations with Structured Latent Codes for Novel View Synthesis of Dynamic Humans, by Sida Peng, Yuanqing Zhang, Yinghao Xu, Qianqian Wang, Qing Shuai, Hujun Bao and Xiaowei Zhou, at CVPR 2021, with the arXiv version and a video linked. The journal version is Implicit Neural Representations with Structured Latent Codes for Human Body Modeling, in IEEE Transactions on Pattern Analysis and Machine Intelligence, accepted in November 2023, with Chen Geng added to the author list and the project description updated accordingly. The repository description calls the work a CVPR 2021 best paper candidate, and the two publications mean the code in the repository is the journal version rather than exactly what was released at the conference, which matters if you are reproducing a specific table of numbers.

The datasets are the durable part

Most of what makes this repository useful today is the data, and the news section shows it being extended for years. In May 2021 the authors updated the ZJU-MoCap dataset with better fitted SMPL parameters, using EasyMocap, and released a website for visualising the capture. The fitted body parameters are what let a method that represents a human with a body model line up with real footage, and the difference between a mediocre fit and a good one is visible in every rendered frame. Two more datasets arrived in January 2024, the Mobile-Stage dataset and SyntheticHuman++, both distributed through request forms. The People-Snapshot dataset is the other half of the story, and some of its processing is already done for you, with a script for processing more videos of it and a second for visualising its SMPL parameters. The authors also saved their own rendering results on ZJU-MoCap together with a document describing the training and test protocols, and did the same for competing methods, so a comparison against other approaches is possible without rerunning them.

Two ways to install it

The installation section is short because the detail is in the files it points at. For a manual install you follow INSTALL.md, and there is a separate Docker route described in docker/README.md, which the README credits to Zhaoyi Wan for providing the implementation. A container matters more here than it usually does, because the dependency set includes native packages for geometry and rendering, and a system without the right OpenCV and Open3D builds is a long way from a working environment. Running on a custom dataset is a third path, documented separately under tools/custom, and the People-Snapshot and ZJU-MoCap sections each point back at INSTALL.md for downloading the data and at a shared drive for the pretrained models, which are published as checkpoint files named after the sequence and the configuration.

Visualisation is where the results show up

The visualisation commands are the most useful thing to read first, because they show the shape of the configuration system. The script visualize.sh records the exact command lines for the female-3-casual example, and the first one renders novel views of a single frame with 144 render views:

python run.py --type visualize --cfg_file configs/snapshot_exp/snapshot_f3c.yaml exp_name female3c vis_novel_view True num_render_views 144

The second switches to views of a dynamic human with a fixed camera, using the vis_novel_pose flag, and the third generates meshes with the vis_mesh flag, with a separate tool for rendering a specific mesh by index. The results land in two directories under the data path, one for the rendered novel views and one for the performed sequences. Every run takes a configuration file, so the experiment is defined by YAML rather than by code, which is what lets the authors ship one command line per figure in the paper.

Training, distributed training and Tensorboard

The training section follows the same shape, with the command lines recorded in train.sh and the female-3-casual sequence as the example. A single GPU run is a call to train_net.py with a configuration file, an experiment name and the resume flag set to false:

python train_net.py --cfg_file configs/snapshot_exp/snapshot_f3c.yaml exp_name female3c resume False

Distributed training uses the torch distributed launcher with four processes per node and an explicit GPU list, which is the older style of launcher that predates the run interface, and there is a variant that trains with a white background, selected with the white_bkgd flag, for the datasets where the background is not part of the capture. Training logs go to a directory that Tensorboard reads:

tensorboard --logdir data/record/if_nerf

The record directory name comes from the original implementation, and it is a small reminder of where the code comes from.

The SMPL parameter trap

The section on potential problems with the provided SMPL parameters is the one to read before you trust a result, and it is unusually candid. The newly fitted parameters live in a directory called new_params, while the released pretrained models are trained on the previously fitted parameters, which live in a directory called params, so a model and its parameters have to be matched. Second, the SMPL parameters of ZJU-MoCap have a different definition from the one used by MPI's smplx, and if you want to extract vertices from the provided parameters you should use the script in zju_smpl/extract_vertices.py, with the reason for the choice described in the EasyMocap documentation for users of the smpl and smplx conventions. The last line is the reassuring one, that it is okay to train Neural Body with SMPL parameters fitted by smplx, so the definition difference is a nuisance for the released checkpoints rather than a limit on the method.

Dependencies, layout and the state of the code

The requirements file is short and tells you the era of the code, with open3d, PyYAML, tqdm, tensorboardX, termcolor, scikit-image, opencv-contrib-python, opencv-python below version 4, imageio, trimesh, plyfile, PyMCubes, pyglet and chumpy. The chumpy dependency and the OpenCV pin are the two that will cause friction on a current system, since chumpy is a package that no longer installs cleanly on recent Python versions. The repository layout is conventional for a research project, with configs/ for the experiment definitions, tools/ for the custom dataset and processing scripts, lib/ for the library code, docker/ for the container setup, a zju_smpl/ directory for the body model helpers, and the entry points at the top level in run.py, train_net.py, eval_whole_img.sh, test.sh, train.sh and visualize.sh. The licence is recorded as a custom one that GitHub cannot classify, so the LICENSE file in the repository is the document to read, and the repository is not archived with the last push on 2026-09-15 and no tagged releases published.

Editorial conclusion

Neural Body fits a researcher who wants to reproduce or extend a view synthesis method on human motion, since the datasets, the fitted body parameters and the pretrained checkpoints are all published, and the training and evaluation scripts are documented with worked examples. It does not fit a modern PyTorch project, because the requirements are pinned to versions from the PyTorch 1.x era, including a chumpy dependency and opencv below 4, so it will need work to install today. Before you start, read the section on the SMPL parameters, because the released models were trained on the earlier fitted parameters and a mismatch there quietly ruins results, and note the release state, where the repository publishes no tagged releases and the last push was on 2026-09-15.

Frequently asked questions

Official sources

1. Issues
2. README
3. zju3dv/neuralbody on GitHub