# Zero-WAM treats a human video as the task instruction a robot never trained on

> A research release that pairs robot trajectories with generated human demonstration video so one causal policy can run unseen RoboTwin tasks from language or video context, with a 46.95% zero-shot average across seven tasks against 17.45% for LingBot-VA.

**robbyant-research/Zero-WAM** — In‑Context World‑Action Modeling from Human Videos for Open‑Ended Task Generalization

- Repository: https://github.com/robbyant-research/Zero-WAM
- Website: https://robbyant-research.github.io/Zero-WAM/
- Stars: 426 · Forks: 11
- Language: Python
- License: Apache-2.0
- Published: 2026-09-17 · Updated: 2026-09-17 · Language: en
- Canonical page: https://hysenlabs.com/projects/robbyant-research-zero-wam

## The human video is the prompt, and the robot trajectory is the answer key

Zero-WAM is built on one substitution: a human demonstration video takes the place of a task specification the robot was never trained on. A single causal policy then predicts both future robot observations and the executable actions, and it accepts either language instructions or human video as the instruction. That is what makes the generalization claim different from collecting more robot data, since the deployment-time context carries the task rather than the weights. The project states the constraint plainly: tasks were never practiced during training, and the model parameters are not updated at deployment, with no robot data collected for those unseen configurations. The video path is the unusual half. A language instruction tells the policy what to do; a video of a person doing it also carries object states, hand trajectories and timing, which is why the paper treats video and text as two views of the same in-context signal.

## HumanGen rebuilds a robot's first frame as a human demonstration

The HumanGen pipeline is where the human video comes from, and it runs in four stages. It starts from task-sampled robot trajectories that still carry their executable actions, so the source keeps the action labels. A vision language model parses task semantics and object-state changes. An image editor converts the first robot frame into a human observation, which is what makes the clip look like a person rather than a manipulator. A video generation model then synthesizes the corresponding human manipulation video. Every generated clip is filtered for semantic preservation and physical plausibility before it is paired back with the original robot trajectory as an in-context sample. Those two filter criteria are the only quality control named anywhere in the walkthrough, and they are the hinge of the whole approach: drop a clip where the human changes the wrong object and the prompt stops being a correct specification of the task.

## Task-level resampling, 6K tasks and about 400K trajectories per epoch

The robot half of the training data is built around task diversity rather than raw trajectory count, and the mechanism is resampling at the task level. Five public robotic video-action datasets are re-sampled that way, yielding more than 6K manipulation tasks and about 400K robot trajectories per training epoch, so a single pass over the corpus sees many more distinct tasks than a frame count would suggest. HumanGen sits on top of that with 74.2K human-robot in-context pairs spread over 8.6K tasks, drawn from external, in-house, simulation and real-world sources. The public pre-training subset is described as built from the same five robotic video-action datasets, and the list of their names is where the walkthrough ends: AgiBot, InternData-A1, and a third that the text cuts off after the letters Op. Two of the five names are not readable from the documentation, so anyone reconciling the corpus against a data card is working from an incomplete list.

## 43 training tasks, seven unseen tasks, and latents shipped with a manifest

Two HumanGen subsets are released with the repository: Simulation ICL for RoboTwin and Pre-training ICL for public datasets. The RoboTwin subset holds about 2,500 human-robot pairs across 50 RoboTwin 2.0 tasks, split 43 for training and seven held out for evaluation, which is the same seven that make up the headline numbers. What ships is more than video: robot videos, robot actions, human-video latents, robot-video latents and a pairing manifest, so the pairing between a generated human clip and its source trajectory is a file you can read rather than a convention you have to infer. The documented layout puts each task in its own directory and the human latents in per-run sample folders:

```text
data/HumanGen/
├── robotwin_data/<task>/{data,meta,videos,latents}
├── human_latents/robotwin/run_*/samples/*/*.pth
└── icl_configs/ICL_config_robotwin.json
```

For post-training and evaluation the minimal download filters the dataset down to those three prefixes.

## The download narrows to three prefixes so you skip the pre-training bulk

Getting the data takes the Hugging Face CLI, installed on its own before anything else:

```bash
python -m pip install "huggingface_hub[cli]"
```

Then the RoboTwin ICL subset comes down as a filtered download rather than a full repository clone:

```bash
hf download Robbyant-Research/HumanGen \
  --repo-type dataset \
  --local-dir "${PROJECT_ROOT}/data/HumanGen" \
  --include "icl_configs/*" \
  --include "robotwin_data/**" \
  --include "human_latents/robotwin/**"
```

Three include patterns, one for the ICL config file, one for the per-task robot data tree and one for the RoboTwin human latents. That split matters because the two subsets are released from the same dataset repository, so the pre-training pairs and the RoboTwin pairs sit side by side on the Hub, and the RoboTwin reviewer only needs about 2,500 of them. The local directory has to be the one the project expects, because every later command is written against `${PROJECT_ROOT}/data/HumanGen` rather than against a cache location the CLI picks.

## Two unseen-task success conditions were changed after the fact

The evaluation instructions defer to a separate file for anything that affects reproducing a number, and one of those items deserves attention before you read the table: INSTALL.md carries the exact tested environment of Python 3.10, PyTorch 2.9.0 and CUDA 12.6, the tested Robotwin revision, and the two unseen-task success-condition updates. Success conditions are the pass or fail rules of a manipulation task, and adjusting them for the two unseen tasks means the published percentages were produced under conditions that the walkthrough does not show. The install itself is a requirements file with build isolation turned off:

```bash
python -m pip install -r requirements.txt --no-build-isolation
```

RoboTwin evaluation additionally needs a working RoboTwin 2.0 installation, with the official guide at https://robotwin-platform.github.io/doc/usage/robotwin-install.html, and every command in the walkthrough is written to run from the repository root, which is why the path is exported first:

```bash
export PROJECT_ROOT=/path/to/Zero-WAM
cd "${PROJECT_ROOT}"
```

## requirements.txt and pyproject.toml pin numpy differently

The two dependency declarations in the repository disagree in a way that will decide whether your install matches the reported numbers. requirements.txt, the file the walkthrough tells you to install, pins numpy==1.26.4 along with torch==2.9.0, torchvision==0.24.0, torchaudio==2.9.0, diffusers==0.36.0 and transformers==4.55.2, and adds opencv-python and matplotlib that pyproject.toml never mentions. pyproject.toml is looser on numpy at >=1.26.4,<2, adds tokenizers>=0.21.4, and spells flash-attn where requirements.txt writes flash_attn. It also groups lerobot==0.3.3, scipy and wandb under a train extra, while requirements.txt carries them flat under a Post-training comment. The package metadata is not a release either: the project version is 0.0.0, and the distribution exists to package the wan_va module with its assets rather than to be installed from an index. Treat requirements.txt as the source of truth for evaluation runs and pyproject.toml as packaging metadata.

## wan_va ships weights and normalization stats as package data

The repository tree is small enough to read as a map of the pipeline: evaluation/, script/, tests/, assets/, the example/demo/ and example/robotwin/ directories, and one Python package, wan_va. Package discovery is restricted to wan_va and its subpackages, and the packaged data is declared explicitly as assets/*.pt plus assets/norm_stats/*.json, which tells you that action normalization statistics travel with the code rather than being computed at run time. Two checkpoints are published on the Hugging Face Hub to match those assets, zero-wam-pretrain and zero-wam-posttrain-robotwin, and the paper is at arXiv 2608.26103. Tooling is thin by design: the Makefile exposes a single format target that runs isort and then black, black is configured for a line length of 88, isort uses the black profile and mypy runs in strict mode. No make target runs the evaluation or the tests, so the entry points for both live in the script/ and evaluation/ directories rather than in the build tooling.

## Conclusion

Zero-WAM matters if you work on policy generalization and treat human demonstration video as a prompt rather than as training data, because the framing, the HumanGen pipeline and the future-chunk objective all separate from any particular robot stack. Before running an evaluation, check three things: INSTALL.md, because the two unseen-task success conditions were updated after the fact and the README says so; the tested matrix of Python 3.10, PyTorch 2.9.0 and CUDA 12.6, because requirements.txt and pyproject.toml disagree about numpy; and the scope of the release, which is a HumanGen dataset plus two checkpoints, with the walkthrough ending before it reaches the pre-training data details. A 9.00% score on stacking three blocks says the generalization is real but not yet uniform.

## FAQ

### What is Zero-WAM and what problem is it aimed at?

It is a causal video-action policy for zero-shot cross-task robotic manipulation, where a task is never practiced during training and only deployment-time context describes it. One policy accepts either language instructions or human video as the task instruction and predicts future robot observations plus executable actions.

### How does Zero-WAM generate its human video training prompts?

Starting from task-sampled robot trajectories that keep their executable actions, a VLM parses task semantics and object-state changes, an image editor turns the first robot frame into a human observation, and a video generation model synthesizes the human manipulation clip. Each clip is filtered for semantic preservation and physical plausibility before pairing back to its robot trajectory.

### How well does Zero-WAM do on the unseen RoboTwin 2.0 tasks?

It reports 46.95% plus or minus 0.72 average zero-shot success across seven unseen tasks, against 10.98% for WAN-Action and 17.45% for LingBot-VA, a 29.50 percentage point lead over LingBot-VA. The weakest task is Stack blocks three at 9.00% and the strongest is Place empty cup at 84.87%.

### What do I need installed to evaluate Zero-WAM on RoboTwin?

Python 3.10 with a CUDA-enabled PyTorch environment, installed from requirements.txt with build isolation disabled, plus a working RoboTwin 2.0 installation. INSTALL.md holds the tested matrix of PyTorch 2.9.0 with CUDA 12.6, the tested Robotwin revision, and two unseen-task success-condition updates.

### What data and checkpoints does the Zero-WAM release include?

Two HumanGen subsets, Simulation ICL for RoboTwin and Pre-training ICL for public datasets, with robot videos, robot actions, human-video latents, robot-video latents and a pairing manifest. Two checkpoints are published on the Hugging Face Hub: zero-wam-pretrain and zero-wam-posttrain-robotwin.

## Sources

- [Issues](https://github.com/robbyant-research/Zero-WAM/issues)
- [License: Apache-2.0](https://github.com/robbyant-research/Zero-WAM/blob/main/LICENSE)
- [Project website](https://robbyant-research.github.io/Zero-WAM/)
- [README](https://github.com/robbyant-research/Zero-WAM/blob/main/README.md)
- [robbyant-research/Zero-WAM on GitHub](https://github.com/robbyant-research/Zero-WAM)

---

Hysen Labs editorial analysis, written from the project's own repository and release notes. Cite the canonical page: https://hysenlabs.com/projects/robbyant-research-zero-wam