NVIDIA Isaac Video to Data: from iPhone video to a robot policy

Three stages that each write their artifacts to disk

Video to Data, usually shortened to V2D, is an NVIDIA Isaac project that converts human demonstration video into simulation-ready assets and physics-grounded robot training data. Its architecture is the interesting part: three composable stages, a video ingestion agent, reconstruction, and robotic grounding, each of which runs independently and writes its artifacts to disk. That design choice is deliberate. Because state lands on disk at every boundary, you can stop a run, inspect what came out, cache it, and recompose the pipeline rather than re-running everything from the raw footage. The repository root adds an end-to-end workflow that connects the components into a complete reconstruction to policy pipeline, and the documented example starts with a raw egocentric demonstration captured on an iPhone. The workflow has been verified with both floating Sharpa and Vega Sharpa embodiments, so the scope is narrow and known rather than hypothetical.

Ingestion writes a graph database and a vector database

The first stage turns a demonstration video into something queryable. It is a LangGraph-driven agentic workflow that segments the video into temporally-bounded action clips, extracts an entity-relation scene graph, and stores per-frame SigLIP-2 embeddings. The output is two files, a graph database and a vector database, referred to as `graph.db` and `vector.db`. The point of building this database rather than processing a whole video linearly is downstream selection: later stages can pick which clips to work on through natural-language retrieval instead of brute-forcing the entire recording. Inside the package the pipeline is staged as segment, then verify or refine, then entity graph, then embeddings. A LangGraph pipeline is joined by an EGAgent-style natural-language retrieval agent, and there is an optional Gradio UI for browsing. The repository also tracks its own pre-publication TODOs in a release readiness document, which is worth reading before you build a pipeline on top. Getting the agent running takes a Python virtual environment, a vLLM server and two scripts. Create the environment and install the package with all extras, which cover vLLM, the webapp, the benchmark and dev tools:

cd video_ingestion_agent
uv venv .venv && source .venv/bin/activate
uv pip install -e ".[all]"     # vLLM, webapp, benchmark, dev tools

You should end up with an activated virtual environment and the extras installed. Then start the vLLM server, which loads the VLM and takes roughly a minute:

python scripts/serve.py -c configs/ingestion.yaml

Ingest a clip of video, writing the run to a directory of your choice, and then ask for the moment you care about in plain language:

python scripts/run_ingestion.py path/to/video.mp4 \
  -c configs/ingestion.yaml --no-verify -o runs/my_run

python scripts/run_retrieval.py "Find clips where someone picks up a mug" \
  -d outputs/ -c configs/retrieval.yaml

The retrieval run should return the matching action clips from the database rather than a video file. If you prefer to browse by hand, the optional interface is one command away:

python scripts/run_webapp.py

Reconstruction runs as 18 separate container images

Reconstruction is where the video becomes geometry, and the operational cost shows here. The stage is described as containerized vision modules that turn selected RGB or stereo clips into per-frame depth, object masks, textured meshes, 6-DoF object poses, and SMPL human body parameters. The package listing gives the number as 18 containerized modules plus multi-view pipelines, each module with its own Docker image. Two multi-view entry points orchestrate the whole reconstruction from a rosbag, named `run_mv_hoi_reconstruction` and `run_mv_calibration`. Running per-module images means the host does not need the model dependencies itself, only lightweight orchestration wrappers, and it also means the isolation cost is real: image builds, container startup, and artifact shuffling between stages. The runtime column in the package table is not decoration. For most teams the Docker overhead of reconstruction is a larger obstacle than the GPU work that follows it.

Retargeting and RSL-RL training close the loop to a policy

Robotic grounding is the third stage. Human motion, for example Arctic, is retargeted onto the target robot embodiment, Sharpa in the described setup, and the reconstructed scene plus the retargeted motion then drive Isaac Lab environments trained with RSL-RL PPO to produce deployable policies. Two details matter for a reader sizing this up. First, the training runtime is pinned to NVIDIA Isaac Lab 2.3.1, and the container is expected to be built from a private registry path, written as `nvcr.io/your-org/your-team`, so the images are not published ready to pull. Second, the training output is not the end of the story. A separate package handles expert demonstration collection, LeRobot dataset conversion, GR00T N1.7 fine-tuning and closed-loop evaluation, and it needs Python 3.10 plus an external Isaac-GR00T checkout. There is also a faster route in the repository for users who want to start directly with GR00T fine-tuning: a bundled tissue-box ONNX policy, offered as an optional shortcut rather than the main path.

run_e2e.sh keeps the full pipeline inside one 48GB GPU

The end-to-end reference is more approachable than the component list suggests. `run_e2e.sh` is a reproducible launcher that orchestrates the stages and records their state for inspection and resumption, and a companion agentic skill drives the launcher underneath to guide setup and execution. The documented example continues through reconstruction, retargeting, source-expert training, demonstration collection, GR00T N1.7 fine-tuning and closed-loop evaluation. Hardware requirements are stated directly: the complete workflow can run on a single NVIDIA RTX A6000 with 48GB of VRAM, and it has also been verified on NVIDIA L40 and L40S GPUs. That is the single most useful line in the documentation for anyone evaluating the project, because it removes the guesswork about cluster size. The same documentation also points to a runnable tissue-box demo with acceptance gates and resume behaviour, so the pipeline can be exercised end to end before any of the research questions are taken on.

The v2d_cuslm stack refuses Blackwell compute capability 12.0

There is a hardware exclusion that will not show up in a casual read, and it is the clearest limitation in the project. The separate v0.2 multi-view HOI object-reconstruction pipeline is validated on RTX A6000 at SM 86 and L40S at SM 89. Blackwell GPUs with compute capability 12.0, written as `sm_120` and including the RTX PRO 6000 Blackwell, are not supported by the TensorRT and cuVSLAM versions inside `v2d_cuslm`. The reconstruction container build and the pipeline launcher both check this before starting, so the failure is a refusal at setup rather than a wrong answer later. If your lab standardises on the newest workstation cards, that is exactly the configuration that fails, and the check happens in the container build and the launcher, so you hit it early and repeatedly. The compatibility notes under the HOI GPU compatibility heading are the place to confirm your own card before planning a schedule around this pipeline.

Each package carries a different runtime, and the roots differ too

The dependency picture is split across five packages, and a reader can be surprised by how many environments a single task needs. The video ingestion agent runs as a Python venv plus a vLLM server, the reconstruction stage runs on Docker with per-module images, robotic grounding runs on Docker, the FlashCHORD accelerated reference-tracking training path with PPO and FlashSAC needs a separate Newton, Warp and JAX container, and the GR00T fine-tuning package needs Python 3.10 with an external Isaac-GR00T checkout. The documented prerequisites are Docker with GPU support, the NVIDIA Container Toolkit, Python 3.10 or later, and for `robotic_grounding` an NVIDIA driver 580.126.09 with CUDA 13.0 recommended. That is four or five isolated environments, each with its own container or virtualenv, for one pipeline. Any schedule estimate should account for building and maintaining all of them, not just the one you happen to be studying first.

Three tagged releases since August, with a custom licence

Maintenance signals here are fresh. The last push was on 2026-10-19, and the repository carries tagged releases v0.2.0 on 2026-08-14, v0.3.0 on 2026-09-17 and v0.4.0 on 2026-09-23, so there is a version stream to pin against and a fairly quick recent cadence. That is the good news. The licensing story needs reading directly: the licence field on the repository says Other, with a custom licence that GitHub cannot classify, and a LICENSE file at the root holds the actual terms. The repository also ships a SECURITY.md and separate documentation, model and project pages, plus published datasets on Hugging Face. Anyone planning to use the released datasets or outputs in a product should read the LICENSE file rather than assuming a permissive open source grant, because the repository does not declare a standard OSI licence identifier.

Editorial conclusion

Video to Data fits teams already working inside the NVIDIA Isaac stack, with Docker GPU support, a vLLM server for the ingestion agent and appetite for multi-stage container runs. It does not fit anyone wanting a single command from raw footage to a deployed policy on non-NVIDIA hardware, since the reconstruction path is tied to TensorRT, cuVSLAM and Isaac Lab 2.3.1. Before committing, check your GPU against the sm_120 exclusion in the HOI compatibility notes, confirm the robot embodiment you need is covered by the retargeting utilities, and read the LICENSE file, since the repository licence is a custom one that GitHub cannot classify.

Frequently asked questions

Official sources

1. Issues
2. nvidia-isaac/video_to_data on GitHub
3. README
4. Releases