SpatialVID: 2.7 million clips with per-frame camera poses

From 21,000 raw hours to 7,089 annotated hours

The scale argument for the dataset is a funnel, and the numbers at each stage matter for anyone planning a training run. The project collected more than 21,000 hours of raw in-the-wild video, then processed it through a hierarchical filtering pipeline into 2.7 million clips, totalling 7,089 hours of dynamic content. Roughly a third of the collected footage survived, which tells you the filtering was doing real work and that the released set is the filtered view of the world rather than a sample of everything recorded.

The second stage is the annotation pipeline, which enriches those clips with spatial and semantic information: camera poses, depth maps, dynamic masks, structured captions and serialized motion instructions. The motivation stated in the paper is that current models are constrained by the scarcity of large scale, high quality training data, and that existing datasets which do provide camera pose information are limited in scale, diversity and annotation richness, particularly for real world dynamic scenes with ground truth camera motion. Camera pose for a moving camera in a moving scene is the hard case, and that is what this dataset is aimed at.

Every clip gets its own annotation folder

The structure on disk is regular, which matters when you are writing a loader. Each video clip is paired with a dedicated annotation folder named after the video's id, and each folder contains five key files. The details of what is in those files are documented separately in the dataset's own annotation file explanation rather than in the code repository, so the data documentation and the pipeline documentation live in two different places. The annotation types listed in the paper give a good idea of the coverage: per-frame camera poses, depth, dynamic masks for separating moving foreground from static background, structured captions, and motion instructions that are serialised, which suggests they are stored in a machine readable form that a model can be conditioned on rather than only as prose for a human to read.

SpatialVID-HQ, and a gated door to the raw metadata

There are two distribution channels, and they are not the same thing. The main dataset is available on HuggingFace and on ModelScope, and a higher quality subset named SpatialVID-HQ is published on both, with depth data for it made available later in September 2025. The date sequence in the project news shows the release happening in stages, with the paper, code and SpatialVID-HQ appearing first, the dataset following, and the raw metadata arriving last.

That last part deserves attention. Raw metadata access is not open by default: it is available through a gated HuggingFace dataset, and the stated reason is to support community research better. In practice this means there is a request and a review step before you can see the unfiltered metadata for the full collection, and the filtered clips on the open channels are what you get without asking. The project also has a ModelScope organisation, which gives a second mirror and a different download option for people who prefer it.

Setting up the scoring environment by hand

The repository is the processing code rather than the data, and the manual setup path is documented step by step. The repository uses submodules, so the clone has to be recursive, and the environment is a conda environment on a pinned Python version:

   git clone --recursive https://github.com/NJU-3DV/SpatialVID.git
   cd SpatialVid
   conda create -n SpatialVID python=3.10.13
   conda activate SpatialVID
   pip install -r requirements/requirements.txt

Scoring needs a second set of packages, including a pinned GPU build of PaddlePaddle from the project's own package index:

   pip install paddlepaddle-gpu==3.0.0 -i https://www.paddlepaddle.org.cn/packages/stable/cu126/
   pip install -r requirements/requirements_scoring.txt

The documentation says to ignore the warning about the `nvidia-nccl-cu12` and `numpy` versions, because it is not a problem. FFmpeg is handled separately, with its own instructions in the motion scoring directory, and once installed the `FFMPEG_PATH` variable in the motion inference script and in the cut utility has to be pointed at your actual binary, since the default is `/usr/local/bin/ffmpeg`. The annotation side has its own requirements file and its own compile step for the camera tracking extensions.

AV1 footage needs a different OpenCV build

One dependency problem is called out in detail, and it is the kind that costs an afternoon if you hit it without warning. The project notes that if your videos are in AV1 rather than H264, you need FFmpeg installed and then conda has to be taught to support the AV1 codec, which means replacing the pip build of OpenCV with a conda-forge build:

   pip uninstall opencv-python
   conda install -c conda-forge opencv==4.11.0

If the conda environment still cannot handle AV1 after that, the documented fallback is the `--backend av` option in the scoring scripts, which switches the video reading backend to PyAV. The documentation is honest about the cost of that fallback: using PyAV for frame extraction may lead to slight inaccuracies in frame positioning. For a dataset whose value is precise geometry, that is not a small caveat, and it is the sort of detail that should decide whether you convert your footage first instead.

A container for the GPU FFmpeg setup

For anyone who would rather not assemble the environment by hand, a containerised path was added in October 2025, described as a pre-configured environment with NVIDIA GPU-accelerated FFmpeg. The repository carries a `Dockerfile.cuda` and a `docker-entrypoint.sh`, so the container path is a first class option rather than an afterthought, and it targets the same scoring and annotation tooling that the manual instructions cover. The Docker route also removes the FFmpeg path configuration step, since the image knows where its own binary is, which is precisely the step that is most often wrong in a manual install.

The rest of the repository layout matches the pipeline stages: `camera_pose_annotation/` for the tracking work, `caption/` for the captioning side, `scoring/` for evaluation, `scripts/` and `utils/` for the shared tooling, `requirements/` for the three requirement files, and a `viser/` directory for visualisation. The licence is Apache-2.0, the paper is on arXiv, the project was accepted by CVPR 2026, and the last push was on 2026-04-22 with no tagged releases in the repository, so the versions in the requirement files are the thing to pin.

Editorial conclusion

SpatialVID suits a research group training or evaluating a spatial model that needs ground truth camera motion in dynamic scenes, since the per-frame poses and depth are the thing existing datasets rarely provide, and the split between the hosted dataset and the open processing code lets you use the data without adopting the pipeline. It is a poor fit if you need the raw hours rather than the filtered clips, because only 7,089 of the more than 21,000 collected hours survive the hierarchical filtering, and it is a poor fit if you want a turnkey dataset loader, because the repository ships scoring and annotation tooling instead. Before you start, read the annotation file documentation for the five files that accompany each clip, register for the gated raw metadata dataset if you need it, and check the codec of your own footage, since the pipeline needs a different OpenCV build for AV1 and the PyAV fallback introduces slight frame positioning inaccuracies.

Frequently asked questions

Official sources

1. Issues
2. License: Apache-2.0
3. NJU-3DV/SpatialVID on GitHub
4. Project website
5. README