EasyVolcap: a PyTorch toolkit for neural volumetric video

Volumetric video is a different problem from scene capture

EasyVolcap is a PyTorch library for accelerating neural volumetric video research, particularly in areas of volumetric video capturing, reconstruction and rendering. The paper it accompanies was presented at SIGGRAPH Asia 2023 in Sydney, and the technical report and paper are linked from the README, along with an example dataset and a pretrained model.

Volumetric video is worth separating from ordinary neural rendering because the capture problem is different. The subject is a person or an actor moving through space, captured by multiple cameras, and the research questions are about how to represent and render a dynamic subject rather than a static scene. The examples in the README run on a small multi-view video dataset with several of the implemented algorithms, named as Instant-NGP+T, 3DGS+T and ENeRFi, which is ENeRF Improved.

The most visible output of the project is 4K4D, a real-time 4D view synthesis algorithm developed using EasyVolcap, which was made public in October 2023 and accepted to CVPR 2024. That is a useful signal about the codebase, because a research framework that produced a peer-reviewed real-time method has been exercised hard by its own authors.

The changelog lives in docs/misc/changelog.md, and the last entry in the news list is from August 2024, so the project's own release history is documented in the tree rather than only in the README.

Install it editable, and watch the PyTorch build

The installation section is careful about the order of operations, and the reason is that the library needs PyTorch with CUDA extensions, which means the PyTorch build has to be right before the library is installed on top of it.

The core install is one command, an editable install with the dependencies from requirements.txt:

# Editable install, with dependencies from requirements.txt
pip install -v -e . 

And then the warning that catches people out. On Windows or older versions of Linux you might end up with a CPU-only PyTorch installation by only running that command, because only the CPU version for Windows is available on PyPI. The README links to the light-the-torch project for the background and gives the CUDA-enabled alternative:

pip install -v -e . -f https://download.pytorch.org/whl/torch_stable.html

with a further suggestion to try installing PyTorch directly with pip install torch and the cu118 or cu121 index URLs if that still does not give you a CUDA-enabled build.

The development install is separate and heavier. It requires a valid CUDA building environment with PyTorch already installed, and the two commands are run in tandem so that PyTorch and EasyVolcap are handled before the CUDA extensions are compiled:

# Editable install, with dependencies from requirements-devel.txt
pip install -v -e . # will install from requirements.txt
pip install -v -r requirements-devel.txt

The project also lists an environment.yml in the repository root, and the requirements file notes that using it makes the torch requirement a no-op, since the environment already pinned it.

Updating means re-running the editable install

There is a small operational wrinkle in the updating section, and it is the kind of thing that costs an afternoon if you do not know about it.

Aside from running git pull, you might also need to reregister the command lines and code path by running pip install -e . --no-build-isolation --no-deps again. The command line tools in this project are entry points into the installed package, so a git pull that changes the code does not by itself update what the command line resolves to.

The README gives a concrete example, which is the clearest way to see why this matters. When updating to 4K4D, you are required to rerun the editable install command to use that repository instead of this one. In other words, the installed package decides which algorithm implementation the command line calls, and switching to a sibling repository means repointing the install.

For a user who installs once and works for a month, this does not matter. For someone following the research over a year, with four or five algorithm repositories to swap between, it is the difference between a working checkout and a stale entry point, and the fix is one command.

The example dataset, and preparing it with COLMAP

The examples section is concrete enough to be followed, which is a good sign for a research library.

The example dataset is a small subset of the ENeRF-Outdoor dataset released by the same team, downloadable from a Google Drive link, and the README says the full dataset has its own guide. The files are unzipped into a data/enerf_outdoor directory, and the listing shows what you should see, the images directory for the actor1_4_subseq sequence, and the intri.yml and extri.yml calibration files.

The scripts set two shell variables to make the rest of the commands readable:

expname=actor1_4_subseq
data_root=data/enerf_outdoor/actor1_4_subseq

And the documentation has a static.md page with a complete example of preparing the dataset using COLMAP and running the three models, Instant-NGP+T, 3DGS+T and ENeRFi, with EasyVolcap. That preparation step is the part that usually goes undocumented in research code, and having it written down is worth a lot to someone reproducing the results.

For ENeRFi specifically, the pretrained model for the DTU dataset is downloaded separately, renamed to latest.npz and placed in data/trained_model/enerfi_dtu.

The config-driven command line, and the speed dial

The rendering commands show how the library is meant to be used, and the shape is the point. Every run is a config file list plus key overrides on the command line.

# Render ENeRFi with pretrained model
evc-test -c configs/exps/enerfi/enerfi_${expname}.yaml,configs/specs/spiral.yaml,configs/specs/ibr.yaml runner_cfg.visualizer_cfg.save_tag=${expname} exp_name=enerfi_dtu

There are two entry points. evc-test renders and saves, and evc-gui renders with a GUI, and the same config list drives both. The specs files are the interesting part, because each one is a quality and speed setting, and the README shows the dial turning.

The default gives 2.5 FPS on a 3060. Setting n_planes to 32,8 and n_samples to 4,1 gives 3.6 FPS, and adding the fp16 spec does the same. The best speed setting, fp16 with the reduced planes and samples, reaches 5.0 FPS on the same card. The README labels them as slightly worse quality but faster, fine quality but faster, and worst quality and fastest, and the fact that the trade-off is exposed as configuration rather than baked into the code is what makes it usable.

The library also supports WebSocket-based server-side rendering, which is appended as another spec in the config list, server.yaml, and the viewer then runs in a client of your choice, tested on Windows and macOS. For a multi-GPU lab, that is the difference between rendering on the workstation and rendering on the server while the workstation just displays.

Where EasyVolcap is the wrong choice

The first wrong-tool case is a machine without a working CUDA setup. The README is explicit that a plain install on Windows or an older Linux gives you a CPU-only PyTorch, and the project expects a CUDA build environment for development, so a laptop without a supported GPU is not a development machine for this library.

The second is a project doing ordinary scene reconstruction. The scope is volumetric video, human subjects and multi-view capture, and a NeRF or a Gaussian splatting project on a static scene will find the dataloaders and the example assumptions are not the ones it needs.

The third is a team that wants a stable, pinned dependency. The package version is 0.0.0, the dependencies are dynamic, and the changelog's last entry in the news list is from August 2024. The code moves with the research, which is what you want if you are doing the research and not what you want if you are building on it.

There is also a data size consideration that the project mentions in passing. The setuptools configuration notes that automatic finding of packages is too slow because of a symlink to large datasets, which is a reminder that this is a repository with gigabytes of sample data referenced, and the dataset for your own capture will be larger still.

The package layout, and what the licence covers

The package metadata describes a library with a clear internal architecture, and the explicit package list is more informative than a directory tree would be.

There is easyvolcap.engine with its handlers, easyvolcap.dataloaders with its datasets, easyvolcap.models with subpackages for networks, and inside that for regressors, embedders, samplers, renderers and supervisors, and easyvolcap.runners with evaluators and visualizers. Then easyvolcap.utils with its shaders, easyvolcap.scripts, easyvolcap.configs and easyvolcap.assets with meshes, images and fonts.

That structure is a research codebase arranged by role rather than by feature, and the models subpackages map onto the concepts of the field, an embedder turns a scene into a representation, a regressor predicts view-dependent effects, a sampler generates samples, a renderer turns samples into pixels and a supervisor supervises the training. The assets package carrying meshes, images and fonts is the detail that tells you the renderer is self-contained, with everything it needs to draw text and geometry in the box.

The licence is a file named license in the repository root, and the project metadata points at it rather than at a standard identifier, so read it before using the code in a commercial product. The project requires Python 3.8 or newer, the author is Zhen Xu, and the last push was on 2026-09-15 with no GitHub releases, which fits a research library whose versions live in the repository history rather than in a registry.

Editorial conclusion

Adopt EasyVolcap if you are researching neural rendering of human subjects and want several algorithms, dataloaders and evaluators in one PyTorch codebase with a config-driven command line. Do not adopt it for production rendering on a machine without CUDA, since the README warns that a plain install gives you a CPU-only PyTorch, and do not treat it as a general NeRF library, since the scope is volumetric video rather than scene reconstruction. Before you install, read the CUDA installation notes, and if you plan to swap in the 4K4D algorithm, rerun the editable install afterwards, because the command line registration points at the repository you installed.

Frequently asked questions

Official sources

1. Issues
2. README
3. zju3dv/EasyVolcap on GitHub