Vista4D reshoots a video from new camera paths through a 4D point cloud

What video reshooting means here

Vista4D is described as a video reshooting framework which synthesizes the dynamic scene represented by an input source video from novel camera trajectories and viewpoints. The name is doing real work: the input is a video of a scene that moved, and the output is that scene rendered again from somewhere else, as though the camera had been there.

The technical problem the project attacks is a distribution shift. The framework bridges the gap between training and inference for point-cloud-grounded video reshooting, and it does so by tolerating point cloud artifacts from imprecise 4D reconstruction of real-world videos. The approach is to train on noisy, reconstructed multiview videos, so that the model expects the messy geometry it will actually receive at inference time rather than a clean synthetic world.

The scene representation is a 4D point cloud with temporally persistent static points, which the README credits with explicitly preserving scene content and improving camera control.

The claimed applications follow from that representation. Dynamic scene expansion treats a casual video capture of a scene as a background reference, 4D scene recomposition is point cloud editing, and long video inference with memory extends the output beyond the length of the input.

The audience is a research or graphics engineering team. This is a paper artefact with weights, not a product, and the repository is organised around reproducing and extending a published result.

The release timeline and where the artefacts live

The updates section gives a dated sequence that is worth reading as a project history. On 2026/02/21 Vista4D was accepted to CVPR 2026. On 2026/04/09 it was selected as a Highlight paper. On 2026/04/23 the inference code, model weights and evaluation dataset were released. On 2026/05/02 the project announced that WanGP v11.52 had added Vista4D to their repo.

So the code came out roughly two months after acceptance and the Highlight designation came a fortnight before the release, which tells you the artefacts are recent.

The weights and the data are hosted on Hugging Face, with one repository for the model and a dataset repository for the evaluation data. The paper itself is on arXiv, and there is a project page.

The WanGP integration is the interesting entry for anyone without the hardware to run this directly. The announcement says WanGP v11.52 has added Vista4D, and that you can check it out if you want to run Vista4D with lower VRAM and optionally fewer timesteps. That is a third-party integration path around a demanding reference setup, and it changes the practical cost of trying the method.

The people behind the work come from Eyeline Labs, Netflix, Columbia University, UCLA, Stony Brook University and the University of Oxford, and the README notes that some of the work was done during an internship at Eyeline Labs.

The repository has no GitHub releases, so the artefacts live on Hugging Face and the code lives on the main branch.

Conda, CUDA 12.8 and a pinned transformers release

The environment setup is documented step by step, starting with a Conda environment:

conda create --name vista4d python=3.12
conda activate vista4d

The project uses CUDA 12.8 for installation and compilation. If the default CUDA version on your system is not 12.8, or you are unsure, the README asks you to install a self-contained toolkit into the environment:

conda install -c nvidia cuda-toolkit=12.8
conda install -c conda-forge gxx_linux-64  # Ensure a compatible C++ compiler
export CUDA_HOME=$CONDA_PREFIX  # Required for the environment setup
export LD_LIBRARY_PATH=$CONDA_PREFIX/lib:$LD_LIBRARY_PATH

PyTorch comes next, from a specific wheel index:

pip3 install torch==2.10.0 torchvision==0.25.0 --index-url https://download.pytorch.org/whl/cu128

The README states that 2.10.0 is the version the authors used during development, while noting that others may work. The rest of the dependencies come from the requirements file:

pip3 install -r requirements.txt

That requirements file is where a hard constraint hides. The transformers version is pinned at 4.57.6 with a comment that transformers 5 will break DA3, the Depth Anything 3 dependency. That is the kind of note that saves a debugging session, and it also means the environment cannot be moved to the latest transformers without work.

Flash Attention, XFuser and the architecture list you must set

Two installations follow, and the second one carries the most dangerous failure mode in the whole setup.

Flash Attention and XFuser, the latter for unified sequence parallel, are installed like this:

export FLASH_ATTENTION_FORCE_BUILD="TRUE"  # Optional: If you want to force a compilation from source
pip3 install flash-attn==2.8.3 --no-build-isolation  # This might take a while if you are compiling from source
pip3 install "xfuser[flash-attn]==0.4.5"

The expandable section about SLURM login nodes and multiple GPU types explains why the architecture list matters. If you are installing on a GPU-less login node, or you plan to run the environment across different GPU architectures, for example compiling on an A100 but running on an H100, you must explicitly specify the target architectures. The README is blunt about the consequence of getting this wrong: failing to do it causes the compiler to skip building the necessary kernels, resulting in a silent fallback to standard math attention and causing out of memory errors when doing Vista4D inference.

A silent fallback that ends in an out of memory error is a bad combination, because the error points at the model rather than at the build. The fix is a single variable:

export TORCH_CUDA_ARCH_LIST="8.0;8.9;9.0+PTX"

with a comment listing the architecture guide, 8.0 for A100 and A30, 8.6 for RTX 3090, A40 and A6000, 8.9 for RTX 4090, RTX 6000 Ada and L40, 9.0 for H100 and H200, and 9.0+PTX as the JIT fallback for Blackwell GPUs. The guide also asks you to select only the architectures you need to keep compile times reasonable.

Reconstruction and segmentation run before inference

The pipeline has a preprocessing stage that is not optional, and it depends on third party models the repository carries as directories.

For 4D reconstruction you can pick from Depth Anything 3 or Pi3, and for dynamic mask segmentation the project uses Segment Anything 3, which requires you to request checkpoint access on their Hugging Face repo. That access request means you authenticate before anything runs:

hf auth login

and then paste your Hugging Face access token as instructed.

The preprocessing run itself is driven by a shell script under scripts/preprocess:

EXAMPLE=couple-newspaper RECON_METHOD=pi3 bash scripts/preprocess/example_recon_and_seg_single.sh

The two environment variables pick the example and the reconstruction method, which tells you that reconstruction choice is a runtime decision rather than something baked into the pipeline.

The repository layout shows how much of the stack is third party. There are directories for depth_anything_3, pi3, sam3 and diffsynth alongside the project's own checkpoints, scripts, utils, eval_data, cam_ui and media directories.

The ordering matters for anyone planning compute. Reconstruction and segmentation have to happen before the reshooting inference, and the segmentation stage depends on gated checkpoints you may have to wait for.

The camera UI and the two downstream applications

The repository ships more than the core reshooting script.

A camera UI is included, described as being for all of the above, video reshooting and the applications alike. The requirements list for that UI component names fastapi, uvicorn with the standard extra, and viser, which is a fast way to serve a browser based control surface. A camera path is the natural control for this kind of system, and a web UI is a reasonable way to draw one interactively rather than encoding it as coordinates.

Two applications have their own inference code. 4D scene recomposition is point cloud editing, and dynamic scene expansion has its own inference path as well.

The requirements file is organised by component, which is worth reading as a dependency map. A DiffSynth-Studio section covers accelerate, einops, ftfy, modelscope, imageio with ffmpeg, pandas, peft, protobuf, safetensors, sentencepiece and the pinned transformers. A Vista4D section adds matplotlib, opencv-python-headless and openexr. The Depth Anything 3 section carries the minimal dependencies for its use, including e3nn, evo, moviepy at 1.0.3, omegaconf, pycolmap, plyfile, trimesh and xformers, and the Segment Anything 3 section adds decord, iopath, kernels, pycocotools and timm.

That structure tells you the project treats its dependencies as components with a defined surface, not as an undifferentiated pile.

What you get, what you do not, and the maintenance picture

The scope is inference. The repository lists inference code for video reshooting, instructions to download and use the model weights, instructions to run Vista4D on the evaluation dataset, inference code and a UI for 4D scene recomposition, inference code for dynamic scene expansion, and the camera UI. Training is described in the overview as the thing the model does, with training on noisy reconstructed multiview videos, but no training code is listed here.

The limitations are therefore concrete. If you want to retrain the method or adapt it to a new capture setting, this repository does not give you the training path. If you want to run it, you need CUDA 12.8, a supported GPU architecture, a gated SAM 3 checkpoint, and enough VRAM that the WanGP integration exists because the reference path is heavy.

The dependency pin is the other soft spot. transformers is fixed at 4.57.6 because of DA3, and the CUDA toolchain is pinned to 12.8, so this environment does not drift forward on its own.

On maintenance, the last push was on 2026-06-02 and the repository is not archived, with no GitHub releases published on GitHub. The licence is Apache-2.0, which is the expected choice for an academic release that carries third party components, and it lets you use the repository code commercially with attribution and a patent grant, though the licences of the Hugging Face weights and evaluation data should be checked separately.

One last practical note for anyone planning around this code: the model weights and the evaluation dataset live on Hugging Face rather than in the repository, so an offline or restricted-network environment has to plan for that separately.

Editorial conclusion

Vista4D suits a research team with CUDA capable GPUs that wants to run published video reshooting results, reproduce the evaluation dataset runs, or explore camera control over a reconstructed dynamic scene. It is the wrong tool if you need training code, a consumer-facing application, or a machine without a recent GPU. Verify first that your GPU architecture appears in the TORCH_CUDA_ARCH_LIST you compile for, since a silent fallback to standard math attention produces out of memory errors rather than a clear failure, and check that the Hugging Face checkpoints you need are accessible before you start the preprocessing stage.

Frequently asked questions

Official sources

1. Eyeline-Labs/Vista4D on GitHub
2. Issues
3. License: Apache-2.0
4. Project website
5. README