SplattingAvatar puts Gaussians on a mesh so an avatar can move

Why the Gaussians have to ride on a mesh

SplattingAvatar is the official implementation of a CVPR 2024 paper on realistic real-time human avatars with mesh-embedded Gaussian splatting, and the mechanism it adds is stated in one short section. The embedding points of 3D Gaussian Splatting on the triangle mesh are updated by the walking on triangle scheme, and the implementation lives in a phongsurface module written in C++ and bound with pybind11. That single idea is the contribution. Standard 3D Gaussian Splatting builds a set of anisotropic primitives from static camera images, and a human avatar built that way sits in a fixed pose. Putting the Gaussians on the mesh instead means the avatar's body geometry drives the splats, so when the body moves the rendering moves with it, which is what makes a real-time avatar possible rather than a static capture. The paper, the arXiv entry and the project page are all linked from the README, along with a demonstration video and a framework diagram.

The environment is pinned to an older PyTorch

The getting started section is a conda recipe and a long dependency list, and the pinning is not incidental. You create an environment with Python 3.9, activate it, and install PyTorch 1.13.1 with CUDA 11.7, with the comment that this version is the one tested:

conda create -n splatting python=3.9
conda activate splatting

# pytorch 1.13.1+cu117 is tested
pip install torch==1.13.1 torchvision torchaudio functorch --extra-index-url https://download.pytorch.org/whl/cu117

Then pytorch3d is built from source, and the remaining dependencies are installed in groups: tqdm, omegaconf, opencv-python and libigl, then trimesh, plyfile, imageio, chumpy and lpips, then packaging and pybind11, and finally numpy pinned to 1.23.1. The libigl and chumpy dependencies are the ones that tell you the geometry side is real, and lpips is the perceptual metric used for evaluation. A PyTorch 1.13 build on Python 3.9 is old, and on a current machine that means either an older CUDA toolkit or a container. The reason is in the pinned versions, since pytorch3d and the Gaussian rasterisation submodules both need a specific PyTorch ABI.

Submodules, a custom C++ module and two body models

The repository has to be cloned recursively so that Gaussian Splatting's submodules are present, and then the two of them are installed as Python packages:

git clone --recursive https://github.com/initialneil/SplattingAvatar.git
cd SplattingAvatar

cd submodules/diff-gaussian-rasterization
pip install .

cd ../submodules/simple-knn
pip install .

On top of that you install a separate module called simple_phongsurf for the walking on triangles, built with pip install -e . from the model/simple_phongsurf directory, and that editable install is what makes the C++ surface code rebuild when you change it. Then the body models arrive. You download the FLAME model, choose the FLAME 2020 version, unzip it and copy generic_model.pkl into the model/imavatar/FLAME2020 directory, and you download the SMPL model at version 1.0.0 for Python 2.7 with ten shape principal components, then move the female and male model files into the model/smplx_utils/smplx_models/smpl directory. Two licensed body models, an editable C++ module and two compiled submodules is the real cost of running this repository, and the README is upfront about each step.

Training, watching it live, and evaluating

The training entry point takes a config file and a subject directory:

python train_splatting_avatar.py --config configs/splatting_avatar.yaml --dat_dir <path/to/subject>

You can select a GPU by setting CUDA_VISIBLE_DEVICES before the call, and the network GUI is disabled by passing an ip of none, which matters if you are running headless or over SSH. The README also explains how to watch the training live with the SIBR remote Gaussian viewer from 3DGS, with the caveat that the viewer needs a standard 3DGS output directory to start, and that you can generate one by running the original 3DGS on any original dataset, and that it is worth changing the FPS setting to Trackball in the viewer. Evaluation is a separate script that takes the same config and a model path, pointing at the last checkpoint directory. The separation of train and eval scripts is ordinary research practice, and the fact that the config is shared between them means the evaluation runs the same model and data settings as the training did.

Full body avatars, and overriding a config at the command line

The full body work is on PeopleSnapshot, and the mechanism for using it is the config override mechanism, which is the most reusable thing in the README. You override a config by passing two config files separated by a semicolon, the second overriding the first, so the PeopleSnapshot run takes the base splatting_avatar.yaml and the instant_avatar.yaml that matches the InstantAvatar format. The evaluation then takes a point cloud directory inside the checkpoint, and there is a third script for animating to a novel pose from a file. The data for PeopleSnapshot comes from a separate download of the parameter files and four sequences, with the images and masks unzipped into the right folders, and a preprocessing script that you run over them first. The override mechanism itself is a plain command line convention, and the reason it matters is that the PeopleSnapshot experiment is a different dataset and a different body model setup without being a fork of the training code.

GPU memory is the constraint you will hit first

The GPU requirement section is short and gives the numbers. The experiments were conducted on a single NVIDIA RTX 3090 with 24GB of memory, and training with less GPU memory can be achieved by setting a maximum number of Gaussians, either in the config where the model section has a max_n_gauss key set to 300000 or less as needed, or from the command line by appending model.max_n_gauss=300000 to the training invocation. The knob is on the number of primitives rather than on the resolution or the batch size, which is the natural choice for a Gaussian splatting system, since the memory cost of the splat set scales with the count and with the rasterisation buffers. A 24GB card is also the current practical floor for this kind of work, so anyone without one is working in a constrained mode from the start. The data itself is the preprocessed set of ten subjects used in the paper, and the preprocessing followed the IMavatar project with the segmentation step replaced by RobustVideoMatting.

Editorial conclusion

Adopt SplattingAvatar if you are reproducing the CVPR 2024 paper on mesh-embedded Gaussian splatting for real-time avatars, and you want the reference implementation rather than your own. Do not adopt it as a turnkey avatar system, since the pipeline depends on separately licensed FLAME and SMPL body models, a pinned PyTorch 1.13 build, and data downloaded from another project. Verify first that you have a CUDA GPU with enough memory, since the experiments used a single RTX 3090 with 24GB, and that the subject data and checkpoints you use match the ten subjects the paper covers.

Frequently asked questions

Official sources

1. initialneil/SplattingAvatar on GitHub
2. Issues
3. README