4DAnyone: Turning One Casual Monocular Video Into Multi-View Footage for 4DGS

The gap 4DAnyone is trying to fill between a phone video and a 4D asset

Dynamic scene reconstruction needs multi-view footage. Getting it usually means a camera rig, a calibrated capture stage, or a subject who can hold still while you circle them. 4DAnyone takes the opposite input: one casual monocular video, the kind you can shoot handheld, and generates the missing viewpoints synthetically. The README states the goal plainly, that the project turns a casual monocular video into multi-view videos, enabling downstream 4DGS reconstruction. The intended user is therefore not someone who wants a finished 4D model. It is someone building that pipeline, who has a source clip and needs synchronized, view-consistent frames at known camera positions before a Gaussian splatting stage can run. The repository ships under Apache-2.0 and is written in Python, with a third_party/GVHMR submodule that has to be initialized before anything runs. The paper is listed as SIGGRAPH Asia 2026, and the arXiv identifier in the README is 2608.20335.

What actually happens between the source clip and the dense output folder

The output layout in the README is the clearest description of the data flow. A run produces metadata.json with run settings, timings and resources, and cameras.json with intrinsics and poses for the N target views. Alongside those sit a gvhmr directory holding motion.json and motion.safetensors, a skeletons directory with one pose-conditioning video per target view, and a videos directory split into sparse and dense subfolders. The sparse folder in the example holds six files (00, 04, 09, 12, 14, 19), while dense holds every target view from 00.mp4 through the last index. That split suggests a two-stage generation in which a smaller set of views is produced first and the dense set is derived with those as reference, though the README does not spell out the mechanism. The motion recovery stage is explicitly reusable: the documentation says completed outputs are never overwritten, and that after a failed or interrupted run you can rerun with the same --output_dir to reuse completed motion recovery and restart generation. On multi-GPU machines, the gpu_ids argument defaults to all visible GPUs and is used for parallel pose and VAE view stages as well as target denoising.

Camera layout is the main knob, and it is expressed in yaw and pitch

Rather than exposing a fixed set of presets, 4DAnyone parameterizes the target rig. views_per_layer sets how many evenly spaced cameras sit on each pitch ring, and the README states it must be divisible by 4 or 6. layer_pitches takes a list of pitch angles in degrees, one per layer, with positive values placing cameras above the subject; total views are views_per_layer multiplied by the number of layers. start_yaw sets the horizontal angle of the first view, with yaw 0 defined as the front view, and yaw_span sets the horizontal range each layer covers. The four documented configurations build on those flags: a 6-view full orbit for a first test, a 24-view full orbit described as suitable for 4DGS reconstruction, a 48-view layout using --views_per_layer 16 --layer_pitches '[-10,15,35]' across three pitch rings, and a 24-view frontal arc using --views_per_layer 12 --layer_pitches '[0,30]' --start_yaw -90 --yaw_span 180. The 48-view case is the one the README ties to free-viewpoint 4DGS rendering, which is a meaningful hint: if you only need a fixed set of angles, the smaller layouts cost far less generation time.

Installation and the two commands that matter

Setup is a clone, a submodule update, a conda environment on Python 3.11, and a pip install from requirements.txt. The submodule step is not optional if you want motion recovery. FlashAttention-3 or SageAttention can be installed for faster inference, and the README says the installed backend is enabled automatically, so there is no flag to set. Models and examples download on first use, or you can fetch them ahead of time with python scripts/download_smplx.py, python scripts/download_model.py and python scripts/download_example.py. Inference is a single entry point, inference.py, with --video_path and --output_dir as the required pair. By default it uses 4DAnyone-Turbo, the distilled variant described as doing four-step denoising; --enable_turbo controls that, and 4DAnyone-Base uses the standard denoising schedule. The project reports a 5.58x denoising speedup for Turbo over Base and a 1.42x end-to-end speedup for the full 24-view pipeline, both from its own release notes. Run python inference.py --help for the complete argument list, since the README only covers the key ones.

The input constraints are strict, and they are the real adoption barrier

The Custom Data section is short and uncompromising. The source video must show a single person in a full-body or upper-body shot, with no large camera movements, clear footage, 1080p or higher resolution, a 9:16 portrait aspect ratio, and at least 121 frames. Every one of those is a filter. Footage shot in landscape is out. Clips with a moving camera are out, which rules out most handheld walking shots even though the project markets itself around casual video. Two people in frame is out. A clip shorter than roughly five seconds at 25 fps is out. The 121-frame figure also matches the performance claim of 27 seconds per 121-frame video on a single RTX 4090, so the benchmark and the input requirement are the same length. On memory, the README reports 22 GB of peak CUDA memory after a 2026-09-05 change that brought it below 24 GB, following an earlier reduction below 32 GB. That number is a project measurement on its own configuration; nothing in the repository lets you predict it for a different view count, and the 48-view layout is not covered by the quoted figure.

How this differs from pose-free dynamic view synthesis

The obvious comparison is a pose-free dynamic view synthesis method, the class of model that takes a monocular video and renders novel views directly from learned scene representations without ever materializing an explicit multi-camera rig. The difference is architectural, not just quantitative. 4DAnyone writes cameras.json with intrinsics and poses for N target views and emits actual per-view video files, so the output is a synthetic multi-camera capture that any downstream reconstruction tool can consume as if it had been filmed. A pose-free renderer gives you pixels along a requested trajectory and stops there. That makes 4DAnyone the better fit when your goal is to feed an existing 4DGS or 3DGS pipeline that expects a conventional multi-view dataset, and the worse fit when you only need to view a scene from a new angle and have no reconstruction stage at all. It also means 4DAnyone inherits the failure modes of explicit camera placement: if the generated views at a given yaw and pitch are inconsistent with each other, the reconstruction stage will bake that inconsistency into the geometry, and the model has no mechanism to report that it happened.

Maintenance surface, licence, and the unchecked roadmap item

The dependency surface is not small. There is a conda environment pinned to Python 3.11, a requirements.txt, a git submodule for GVHMR that must be updated separately, auto-downloaded SMPL-X and model weights, and optional attention backends whose installation differs by GPU generation (FlashAttention-3 is documented for Hopper). Upgrading means re-checking all of those, and the submodule in particular can drift from the parent repository without any version pin visible in the README. The licence is Apache-2.0, which is permissive, but the README does not state the licence terms of the auto-downloaded model weights or of SMPL-X, and those carry their own conditions that are separate from the code licence. That is worth confirming before any commercial use; this is not legal advice. On scope, the roadmap shows memory and speed items checked, 3DGS reconstruction with nerfstudio checked and documented in docs/nerfstudio.md, and 4DGS reconstruction with an open-source method still unchecked. So the headline use case, 4D Gaussian splatting, is the part of the pipeline you currently have to supply yourself.

Editorial conclusion

Adopt 4DAnyone if you already have a clean, static-camera, full-body or upper-body monocular clip at 1080p or higher in 9:16 with at least 121 frames, and you want synchronized multi-view output to feed a 4DGS pipeline. Do not adopt it if your footage has large camera movement, multiple subjects, or a landscape frame, and do not expect an end-to-end 4DGS result yet: the roadmap still lists open-source 4DGS reconstruction as unchecked, with only nerfstudio-based 3DGS documented. Before committing, run the 6-view example command on one of your own clips and read metadata.json for the recorded peak memory and timings, since the 22 GB figure is a project claim you should confirm on your own hardware.

Official sources

1. ant-research/4DAnyone on GitHub
2. Issues
3. License: Apache-2.0
4. README