animatable_nerf: the code behind Animatable Neural Fields

What the repository actually contains: two papers and one codebase

animatable_nerf is the code for Animatable Implicit Neural Representations for Creating Realistic Avatars from Videos, published as TPAMI 2024, with a predecessor paper, Animatable Neural Radiance Fields for Modeling Dynamic Human Bodies, at ICCV 2021. The authors are Sida Peng, Zhen Xu, Junting Dong, Qianqian Wang, Shangzhan Zhang, Qing Shuai, Hujun Bao and Xiaowei Zhou.

The naming history in the repository's news section explains the layout, and it is worth following because the code names follow it. In July 2022 the authors released an extended version of Animatable NeRF, and evaluated three different versions of Animatable Neural Fields: vanilla Animatable NeRF, a version where the neural blend weight field is replaced with a displacement field, and a version where the canonical NeRF model is replaced with a neural surface field, which outputs an SDF instead of a volume density and also uses a displacement field. They also provide an evaluation framework for reconstruction quality comparison. The Animatable SDF implementation was later renamed Animatable Neural Fields, and the paper was accepted to TPAMI in January 2024.

So the repository is a research codebase with an experiment harness rather than a library with a stable API. The scripts are named after experiments, the config files are named after subjects, and the presence of the evaluation framework tells you the code was written to produce comparable numbers across variants rather than to be imported by someone else.

The people it is for are researchers in neural rendering who want to reproduce the paper's results, extend the method, or compare against it. The people it is not for are application developers looking for an avatar pipeline they can call, because the entry point is a script driven by YAML configuration and command line flags rather than a package with a documented interface.

The project has no GitHub releases, the default branch is `master`, and the last push to the tree was on 2026-09-15. The licence is recorded as a custom licence that GitHub cannot classify, with the LICENSE file as the authority, which is worth checking before you build on it.

The pipeline as the commands reveal it: canonical, deformation, blend weights

The command lines are the clearest description of the method's structure, and they are all of the form run.py with a config file, an experiment name and a set of boolean switches.

The evaluation on training human poses is a single run.

    python run.py --type evaluate --cfg_file configs/aninerf_s9p.yaml exp_name aninerf_s9p resume True

The evaluation on unseen human poses adds three switches, and the names give the method away: `aninerf_animation`, `init_aninerf` and `test_novel_pose`.

    python run.py --type evaluate --cfg_file configs/aninerf_s9p.yaml exp_name aninerf_s9p_full resume True aninerf_animation True init_aninerf aninerf_s9p test_novel_pose True

The `init_aninerf` switch points at an initialisation experiment name, which in the training instructions is `aninerf_s9p`, the model trained on the observed poses. `aninerf_animation` selects the variant trained on the blend weight fields of unseen poses, and `test_novel_pose` selects the novel pose test. So the architecture is a canonical model, a deformation field, and a blend weight field that decides how much of each canonical part is used, and the training runs in two stages: first the model on observed poses, then the animation-specific blend weight fields on top of it.

That is the whole of the animatable part of the method, and the extended version in the news section is where the two alternatives live. Replacing the blend weight field with a displacement field gives one variant, and replacing the canonical NeRF with a neural surface field gives the other, with SDF output instead of volume density.

The two-stage training in the repository follows the same order.

    # training
    python train_net.py --cfg_file configs/aninerf_s9p.yaml exp_name aninerf_s9p resume False

    # training the blend weight fields of unseen human poses
    python train_net.py --cfg_file configs/aninerf_s9p.yaml exp_name aninerf_s9p_full resume False aninerf_animation True init_aninerf aninerf_s9p

And a training run writes logs that Tensorboard reads.

    tensorboard --logdir data/record/deform

The config file does the describing. A run on subject `S9` uses `configs/aninerf_s9p.yaml`, and the config carries the subject, the experiment and the model options, which is why the same command shape works for every subject in the dataset.

The visualisation paths: novel views, pose sequences and meshes

The visualisation commands are a separate type, and each one answers a different question about the reconstructed avatar.

Novel views of a frame ask whether the model generalises across cameras. The `vis_novel_view` switch with `begin_ith_frame 0` renders the zero-th frame from viewpoints the model was not trained on.

    python run.py --type visualize --cfg_file configs/aninerf_s9p.yaml exp_name aninerf_s9p resume True vis_novel_view True begin_ith_frame 0

A pose sequence asks whether the animation generalises across poses. The `vis_pose_sequence` switch, combined with a camera index in the `test_view` argument, renders a dynamic human through a sequence of poses from one camera. The example in the documentation uses the third camera on Human3.6M and the zero-th camera on ZJU-MoCap.

Mesh extraction is the third path, and it is the one that turns the implicit model into something a graphics tool can open. The `vis_posed_mesh` switch generates meshes rather than images.

    # generate meshes
    python run.py --type visualize --cfg_file configs/aninerf_s9p.yaml exp_name aninerf_s9p vis_posed_mesh True

The `exp_name` and the output paths in the commands are the other half of the structure. Results land in directories named by experiment, so the visualisation on `S9` ends up at `$ROOT/data/novel_view/aninerf_s9p` and `$ROOT/data/novel_pose/aninerf_s9p`, and the training logs are under `data/record/deform`.

That directory convention is the practical part of using this code. Every artefact is keyed by experiment name, so the same script can produce many subjects and many variants without collisions, and a failed run leaves a directory you can inspect rather than a mess of loose files.

The data problem: Human3.6M cannot be redistributed, ZJU-MoCap needs an agreement

Two datasets appear in the instructions and they are handled very differently, and the difference is about licensing rather than about convenience.

Human3.6M cannot be redistributed. The README is explicit that since the license of the Human3.6M dataset does not allow the authors to distribute its data, they cannot release the processed Human3.6M dataset publicly, and it says to email if you are interested in the processed data. The commands therefore assume you have supplied the data yourself, which is why the test instructions start with placing the pretrained models at specific paths under `$ROOT/data/trained_model/deform/`.

ZJU-MoCap is open to a request. The instructions say to fill in the agreement linked from the project page and email the first author with a copy to another maintainer to request the download link.

Both datasets are downloadable with a request, and the pretrained models for both are on a shared Google Drive link. So the practical setup is: obtain the dataset by whatever route its licence requires, unpack it into the expected layout, drop the pretrained weights into the paths the commands name, and then the scripts run.

Two further datasets were released in January 2024, the Mobile-Stage dataset and the SyntheticHuman++ dataset, each behind its own request form. The SyntheticHuman++ name suggests synthetic capture rather than motion capture, which is a different way to produce training data, and the Mobile-Stage name suggests a smaller staged capture setup. Neither is described in the excerpt of the README available here beyond the request forms, so their contents and licences have to be read from the forms themselves.

The repository also records, for the Human3.6M comparison, a public link to quantitative results that includes the results of other methods: Neural Body, D-NeRF, Multi-view Neural Human Rendering and Deferred Neural Human Rendering. That file was published in October 2021 to make the comparison with Animatable NeRF easier on Human3.6M, and it is a separate download from the code.

Building the native extensions: kd tree and mesh extraction

The setup.py in the repository root compiles two C and Cython extensions, and it is worth reading because the names tell you what the pipeline needs and why.

The first is pykdtree, described in the source as a kd tree, built from two C sources with the C99 standard, optimisation at level three and OpenMP enabled, linked against the OpenMP runtime. A kd tree is the structure you use for nearest neighbour queries over a point cloud, which is exactly what a deformation field that has to look up canonical space points for every sample does.

The second is mise, described as efficient mesh extraction, built from a single Cython source. Since one of the three evaluated variants replaces the canonical NeRF with a neural surface field that outputs an SDF, you need a way to get a mesh out of an implicit surface, and that is what this extension provides.

The build uses Cython's cythonize over the extension list and PyTorch's BuildExtension as the build command, with the setuptools setup call at the end. The import of the CUDA extension class from torch.utils.cpp_extension tells you the project expects a CUDA toolchain to be present, even though the two extensions listed here are CPU ones.

The requirements file is a mix of pinned and ranged versions, and the pins are old. PyYAML at 5.3.1, tqdm at 4.28.1, tensorboardX at 1.2, termcolor at 1.1.0, scikit-image at 0.14.2, imageio at 2.3.0, trimesh at 3.8.15, plyfile at 0.6, PyMCubes at 0.1.0, pyglet at 1.4.0b1, and chumpy with no version at all. OpenCV appears twice, as opencv-contrib-python and as opencv-python constrained to below version 4, both at least 3.4.2.17.

Two of those deserve a second look. chumpy is the differentiable rendering library the NeRF literature depended on, and it is unmaintained and notoriously hard to install on modern Python, so a fresh environment is likely to need work there. PyMCubes at 0.1.0 is the marching cubes implementation behind the mesh extraction, and the mise extension is the faster path.

The installation instructions are not in the README. It points to INSTALL.md for manual installation, and the repository root has a `requirements.txt`, a `setup.py`, an `extension.sh` and a `lib/` directory where the extension sources live, so the installation path is the standard compile-the-extensions route.

Where this codebase stops, and what to do instead

Three limitations are visible in the repository itself.

The first is that this is a paper's code, and it should be read as a record of an experiment rather than as a maintained tool. There are no releases, so there is no version to pin, and the environment described by the requirements file belongs to the period of the publication, with a chumpy dependency and PyYAML 5.3.1. Reproducing the published numbers on a current machine is a setup problem before it is a research problem.

The second is the data. Human3.6M cannot be redistributed and the processed version must be requested, and ZJU-MoCap and the two 2024 datasets all sit behind forms and emails. That is normal for the field and it is still the largest practical obstacle to running this code, and it applies equally to the alternatives.

The third is scope. The repository covers a human avatar pipeline with a canonical model, a deformation field and a blend weight field, and a mesh extraction path. There is no rendering application, no viewer, no editor, no service, and no packaging for someone who just wants an avatar. The commands write files into directories named after experiments and expect the reader to know what to do with them afterwards.

The alternative approach, and it is a real one, is to treat the method as a paper and use the released artefacts. The pretrained models are published on a shared Google Drive link, the results for the Human3.6M comparison are published separately including the numbers for other methods, and the Mobile-Stage and SyntheticHuman++ datasets are available on request. If your goal is to see the method's output, downloading the models and the data is faster than getting the training pipeline to run, and the difference in approach is the difference between running an experiment and reproducing one.

For someone who does want to modify the method, the repository is unusually good on one axis: three evaluated variants, a common evaluation framework for reconstruction quality, and a per-experiment directory convention that makes the comparison reproducible.

Editorial conclusion

Adopt animatable_nerf if you are working on animatable human avatars and want to reproduce the TPAMI 2024 results or swap one of the three evaluated variants, since the repository carries a common evaluation framework and a per experiment directory layout that makes the comparison reproducible. Do not adopt it as an avatar pipeline you can run in production, because the entry points are experiment scripts over YAML configs, there are no releases, and the pinned dependencies including chumpy belong to an older Python ecosystem. Before you start, request the dataset through the route its licence requires, since Human3.6M cannot be redistributed and ZJU-MoCap is behind an agreement, and read `INSTALL.md` for the extension build, because `setup.py` compiles both pykdtree and the mise mesh extraction module.

Frequently asked questions

Official sources

1. Issues
2. README
3. zju3dv/animatable_nerf on GitHub