NeoVerse: turning a single monocular video into a 4D world model

What a 4D world model is supposed to give you

NeoVerse is described as a versatile 4D world model that is capable of 4D reconstruction, novel-trajectory video generation, and rich downstream applications.

The framing of the paper title is the interesting part: enhancing a 4D world model with in-the-wild monocular videos. A world model is a representation of a scene that supports reasoning about what the scene would look like from a viewpoint that was never filmed, or over a time that was never recorded. A 4D one carries geometry through time rather than only appearance.

The input is deliberately undemanding. In-the-wild monocular video means an ordinary video of an ordinary scene, no calibration rig, no depth sensor, no controlled capture. The output of the reconstruction stage is a representation of the scene's geometry and appearance, and the output of the generation stage is a video of that scene along a camera path you specify.

That combination is what makes the tool interesting to a working researcher. Many 4D reconstruction systems need a multi-camera rig or a LiDAR capture, which limits them to datasets someone curated. A system that reconstructs from a phone video opens the question of whether a scene captured casually can be navigated convincingly afterwards.

The project comes from NLPR and MAIS at CASIA and CreateAI, and it was accepted to CVPR 2026, selected as a highlight on 2026-04-09, and named best paper of the VideoWorldModel workshop at CVPR 2026 on 2026-05-08. The arXiv identifier is 2601.00393 and the project page is at neoverse-4d.github.io.

Two ways in: a script and a web demo

The repository offers two entry points, and the split is sensible for a research release.

The first is a simple inference script that generates novel-trajectory videos with a single command. The second is an interactive Gradio demo, a step-by-step web interface for reconstruction, trajectory design and generation. The demo is the better first contact if you want to understand the model's behaviour, since the trajectory design step is interactive, while the script is what you will use once you know what you want.

The inference script supports two trajectory input modes. The first picks from 13 built-in camera motions and fine-tunes them with parameters for angle, distance or orbit radius. The list is a full camera vocabulary: pan_left and pan_right rotate the camera horizontally in yaw, tilt_up and tilt_down rotate it vertically in pitch, move_left and move_right translate it horizontally, push_in and pull_out translate it forward and backward, boom_up and boom_down translate it vertically, orbit_left and orbit_right arc around the scene centre, and static keeps the original camera path.

The second mode is full keyframe-level control, where you provide a trajectory JSON file. The built-in motions are a convenience; the JSON is the real interface, and the repository ships a directory of example trajectory files alongside the example videos.

The README also documents a static scene path where the input is a single image rather than a video, and a sparse keyframe mode with interpolation. That last one is a reasonable way to author a path: give a few poses and let the system fill in the motion between them.

The inference script, option by option

The examples show the same shape every time, which makes the script easy to read. You give an input, a trajectory or trajectory file, a prompt, and an output path, and you can adjust the camera parameters.

A tilt up looks like this:

python inference.py \
    --input_path examples/videos/robot.mp4 \
    --trajectory tilt_up \
    --prompt "A two-arm robot assembles parts in front of a table." \
    --output_path outputs/tilt_up.mp4

A lateral move with an explicit distance:

python inference.py \
    --input_path examples/videos/tree_and_building.mp4 \
    --trajectory move_right \
    --distance 0.2 \
    --output_path outputs/move_right.mp4

A zoom is done by adjusting the focal length rather than by moving the camera, which is a detail worth knowing because the two produce visibly different footage:

python inference.py \
    --input_path examples/videos/animal.mp4 \
    --trajectory static \
    --zoom_ratio 2.0 \
    --output_path outputs/zoom_in.mp4

The trajectory file mode takes a JSON path and an optional alpha threshold, which controls how the keyframe selection behaves:

python inference.py \
    --input_path examples/videos/movie.mp4 \
    --trajectory_file examples/trajectories/orbit_left_pull_out.json \
    --alpha_threshold 0.95 \
    --output_path outputs/orbit_left_pull_out.mp4

The prompt argument is required in the sense that the generation is text-conditioned, so the motion you request and the content you ask for are separate inputs. That is the main difference from a plain video-to-video model: the trajectory is not implied by the source clip, it is specified.

The environment, and the two CUDA paths

The installation instructions are specific about the hardware, and they are the part of the documentation most likely to cost you a day if you skip it.

The project has been tested on CUDA 12.1 with PyTorch 2.3.1 and on CUDA 12.8 with PyTorch 2.7.1, and the install script gives both paths side by side. The environment itself is a conda environment on Python 3.10:

git clone https://github.com/IamCreateAI/NeoVerse.git
cd NeoVerse
conda create -n neoverse python=3.10 -y
conda activate neoverse

The CUDA 12.1 path then installs the pinned PyTorch wheel from the cu121 index, the requirements file and a torch-scatter wheel matched to that torch version, plus gsplat from source:

pip install torch==2.3.1 torchvision==0.18.1 --index-url https://download.pytorch.org/whl/cu121
pip install -r requirements.txt
pip install torch-scatter -f https://data.pyg.org/whl/torch-2.3.1+cu121.html
pip install --no-build-isolation git+https://github.com/nerfstudio-project/gsplat.git

The CUDA 12.8 path is the same shape with torch 2.7.1, torchvision 0.22.1 and the torch-2.7.1+cu128 wheel. The gsplat install is done with --no-build-isolation, which is a detail that matters because it lets the build see the torch you just installed instead of pulling its own.

The requirements file itself is a good map of the stack. transformers is pinned at 4.57.6, and the video and image processing side uses imageio with ffmpeg, opencv-python and decord, with moviepy at 1.0.3. Three dimensional reconstruction pulls in jaxtyping, scipy, matplotlib, evo, e3nn, addict and trimesh, and the training stack uses accelerate, deepspeed at 0.16.7, omegaconf, peft and tensorboard.

Checkpoints, and what is inside them

The model weights are not in the repository, they are downloaded, and the README gives both a Hugging Face and a ModelScope path so that neither is a single point of failure:

hf download Yuppie1204/NeoVerse --local-dir models/NeoVerse
# Or using ModelScope
modelscope download --model Yuppie1204/NeoVerse --local_dir models/NeoVerse

The expected directory structure is published, which is worth reading before you start because it tells you the scale of what is being downloaded and where each piece comes from:

models/NeoVerse/
├── diffusion_pytorch_model-0000*-of-00006.safetensors
├── diffusion_pytorch_model.safetensors.index.json
├── models_t5_umt5-xxl-enc-bf16.pth
├── reconstructor.ckpt
├── Wan2.1_VAE.pth
├── google/
│   └── ... (tokenizer files)
└── loras/
    └── Wan21_T2V_14B_lightx2v_cfg_step_distill_lora_rank64.safetensors

Several things are visible there. The main diffusion model is sharded across six safetensors files. There is a separate text encoder checkpoint, a separate reconstructor checkpoint, and a separate VAE file, and the google/ directory holds the tokenizer files. The loras/ directory contains a distilled LoRA, and the name tells you what it is: a lightx2v configuration and step distillation at rank 64, which is the acceleration mechanism behind the fast inference claim.

The checkpoint is also a statement about the architecture. The diffusion model, the text encoder and the VAE are recognisably from a large text-to-video model, and NeoVerse is built on top of that rather than training one from scratch, which is what makes the weights downloadable at all and the fine-tuning story short.

Pluggable reconstructors and the inference speed claim

The most transferable design decision in the project is the reconstructor interface. The README lists support for different 3D reconstructors, for example Depth Anything 3, through a plug-and-play interface, and the repository layout shows a diffsynth/ directory and a models/ directory that sit alongside the training code.

The reason this matters is that 3D reconstruction from monocular video is the part of the pipeline most likely to be replaced. Depth estimation in particular moves fast, and a system that hard-codes one depth network is a system that ages badly. A plug-in boundary means a newer reconstructor can be dropped in without touching the generation side.

The speed claim is also specific. The inference pipeline completes in under 30 seconds with distilled LoRA acceleration on a single A800, and the LoRA file in the checkpoint directory is the artefact that buys it. The claim is tied to one GPU model, so it is a reference point rather than a promise, and the distilled LoRA is a separate file you can leave out, which would change the timing completely.

On the training side, the code was released and you can fine-tune the NeoVerseControlBranch on your own data with multi-node ZeRO-2, which points at deepspeed for the distributed configuration. The update log records the training scripts and reproducible demo notebooks on 2026-04-14, and the demo notebooks in docs/ cover the downstream applications shown on the project page. The distinction between those two dates is worth noting: inference scripts and checkpoints came out on 2026-02-16, the arXiv paper on 2026-01-01, and the training code came later, on 2026-04-14.

Where the project is the wrong tool

The first limitation is hardware. The reference setup is a single A800, the checkpoint set is a multi-file download, and the install matrix is two specific CUDA and PyTorch combinations. A team without a datacentre GPU, or without patience for a build that installs torch-scatter and gsplat from source, will not get to the interesting part quickly. This is a research release and it behaves like one.

The second limitation is what the output is. The system generates video along a trajectory, and the README frames the demos around visual appeal, with more videos on the project website for an enhanced view experience. A generated video that looks convincing and a scene representation that is metrically or geometrically correct are different things, and the documentation here is largely about the first. If your downstream task needs camera pose you can trust, or geometry you can measure, you should establish that yourself from the reconstruction output rather than assume it from the generated frames.

The third limitation is the model lineage. The checkpoints are built on a large text-to-video diffusion model with a distilled LoRA, and the reconstruction stage depends on an external depth network. Each of those is a moving dependency, and the project has no GitHub releases, so there is no versioned artefact to pin beyond the commit you happen to check out. The last push to the repository was on 2026-05-12 and the repository is not archived, so the code is being changed.

The real alternative is to build on the open components directly. A depth estimator plus a point-based or Gaussian reconstruction method plus a video diffusion model can be assembled into a comparable pipeline with full control over each stage, and the gsplat and nerfstudio dependencies already point at that ecosystem. The difference in approach is that NeoVerse has already solved the hard part of making reconstruction and generation agree, and gives you a trajectory interface, a control branch to fine-tune and a set of released weights, which is a lot of work saved if your question is about the resulting video. If your question is about the method itself, the paper and the released training code are the more useful artefacts and the training branch is exactly where you would start modifying.

Editorial conclusion

Adopt NeoVerse if you are researching 4D reconstruction or world models and want the released checkpoints, the pluggable reconstructor interface and the training code rather than reimplementing a video diffusion pipeline from a paper. Do not adopt it as a production video service, since the reference timing is a single A800 and the model weights are a large multi-file download, and do not expect it to work from a single image without reading what the static scene path does. Before you start, confirm you have a CUDA 12.1 or 12.8 machine with enough disk for the checkpoint directory, read the requirements file for the pinned transformer and deepspeed versions, and test one of the demo notebooks in docs/ before wiring the inference script into anything.

Frequently asked questions

Official sources

1. IamCreateAI/NeoVerse on GitHub
2. Issues
3. License: Apache-2.0
4. Project website
5. README