FastWAM for action prediction with or without future video

FastWAM's Action Prediction Problem

FastWAM targets a specific research question: must a world action model spend computation and inference time generating imagined future video before it predicts robot actions? The repository supports both sides of that comparison. Its first-frame Fast-WAM path predicts actions directly from the current observation, while the Optional IDM path imagines future video first and then predicts actions. The Optional IDM checkpoint can switch between those modes at evaluation time without retraining.

The package is intended for researchers working with robotic control datasets, model training, and benchmark evaluation. The documented targets are LIBERO and RoboTwin, with dataset and task configurations under `configs/`, training entry points under `scripts/`, and separate evaluation managers under `experiments/`. It is not presented as an embedded control library for an arbitrary robot.

The project also includes cached and on-the-fly T5 text embeddings, VAE encoding, image-based observations, action diffusion components, and model checkpoints. That scope is substantial. A user who only needs to replay a fixed action sequence will find that most of this package does not apply. FastWAM is most relevant when the objective is to train a policy or compare different test-time generation paths.

Future Video and First-Frame Inference Paths

The central architectural choice is visible in the distinction between IDM and first-frame evaluation. In `idm` mode, the model produces an imagined future video and uses that result for action prediction. In `first_frame` mode, it skips future video imagination and predicts actions from the current observation. The released Optional IDM checkpoint supports both modes, so the comparison does not require two separately trained checkpoints.

The pipeline also separates action generation from video display. The README states that `infer_action` returns actions and video latents directly. VAE decoding happens only when `infer_joint` requests video output. This avoids redundant work for deployments that need actions but do not need to inspect the imagined video. It is a meaningful design boundary: adding a video-only evaluation mode can require more computation than ordinary action-only inference.

FastWAM uses Hydra configuration overlays for evaluation. The command shown in the README passes a checkpoint, dataset statistics, scheduler shift, inference mode, and GPU count after the script name. Existing model components can therefore remain shared while evaluation selects a different behavior through `EVALUATION.action_infer_mode`.

LIBERO and RoboTwin Data Pipeline

Dataset selection is configuration-driven. The repository contains separate data, model, and task configurations, and the training entry point accepts `data=<config_name>`. The documented LeRobot support covers versions 2.1 and 3.0. The README states that LeRobot 3.0 uses a chunked parquet and video layout intended to scale better as datasets grow, including for data loading and dataset-statistics computation.

For the released LIBERO dataset, the project directs users to Hugging Face and provides a download command that selects the `lerobot_v30` files. The resulting local files are then used with `data=libero_2cam_lerobot_v30`. Existing LeRobot 2.1 configuration files are not described as requiring conversion.

Adapting another LeRobot 3.0 dataset requires copying `configs/data/libero_2cam_lerobot_v30.yaml`, changing `train.dataset_dirs`, and selecting the copied configuration through `data=<config_name>`. This is more controlled than assuming that every LeRobot dataset is immediately compatible. It also puts responsibility on the researcher to ensure that camera fields, observations, actions, and dataset statistics align with the selected model configuration.

Downloading a LeRobot 3.0 LIBERO Dataset

The first documented data operation downloads only the LeRobot 3.0 portion of the released dataset. The command uses `huggingface-cli`, selects the dataset repository, includes `lerobot_v30/**`, and stores the files under `./data`. After the download, the training script points `data=libero_2cam_lerobot_v30` at that configuration.

Use this sequence:

huggingface-cli download yuanty/LIBERO-fastwam \
  --repo-type dataset \
  --include "lerobot_v30/**" \
  --local-dir ./data

The local destination should contain the downloaded LeRobot 3.0 files before training is started. The repository excerpt does not provide a complete environment setup tutorial or hardware compatibility matrix. Its `pyproject.toml` requires Python 3.10 or later and pins CUDA 12.8 builds of `torch` and `torchvision`, so those declared package requirements must be accounted for separately.

Once the data is present, the documented training command is:

python scripts/train.py task=libero_uncond_2cam224_1e-4 \
  data=libero_2cam_lerobot_v30

This starts the LIBERO unconditional two-camera task configuration with the LeRobot 3.0 data configuration. The README does not give a small smoke-test dataset or a one-command local evaluation example, so this is a full research training path rather than a lightweight first-run demo.

Running Optional IDM with a Released Checkpoint

A concrete first evaluation uses the released Optional IDM checkpoint and its matching dataset-statistics file. The README instructs users to download both files from the `yuanty/fastwam` Hugging Face model repository:

huggingface-cli download yuanty/fastwam \
  libero_optional_idm_2cam224.pt \
  libero_optional_idm_2cam224_dataset_stats.json \
  --local-dir ./checkpoints/fastwam_release

The checkpoint and statistics file should remain paired. The statistics file is passed into evaluation because the action inference process needs dataset statistics, while the `.pt` file supplies model parameters.

The following command selects IDM mode and requests eight GPUs through the manager:

python experiments/libero/run_libero_manager.py \
  task=libero_optional_idm_2cam224_1e-4 \
  ckpt=./checkpoints/fastwam_release/libero_optional_idm_2cam224.pt \
  EVALUATION.dataset_stats_path=./checkpoints/fastwam_release/libero_optional_idm_2cam224_dataset_stats.json \
  EVALUATION.sigma_shift=1.0 \
  +EVALUATION.action_infer_mode=idm \
  MULTIRUN.num_gpus=8

Replacing `idm` with `first_frame` selects the direct-action path. The `+` before `EVALUATION.action_infer_mode` follows Hydra's syntax for adding a new configuration key. The manager is also documented as supporting persistent model workers, dynamic task scheduling, bad-GPU quarantine, failure recovery, and resumable results, which are relevant to multi-GPU benchmark runs. The README does not document a smaller supported GPU configuration for this exact command.

Compiled Training and Its Trade-offs

FastWAM has two documented training paths. Cached text embeddings avoid T5 encoding during training. On-the-fly T5 encoding removes the separate text-cache preprocessing step, but the README says it costs approximately 10% training throughput. The choice is therefore between preprocessing convenience and training speed.

The accelerated path combines a compiled denoising core, batched VAE encoding, and a lightweight CUDA Graph backend. Denoising compilation is enabled through `model.compile_training_denoise=true` in the DeepSpeed ZeRO 1 training launcher:

bash scripts/train_zero1.sh 8 task=libero_uncond_2cam224_1e-4 \
  model.compile_training_denoise=true

At evaluation, compiled action inference is enabled by default for LIBERO with `EVALUATION.compile_action_infer=true`, and the README says existing checkpoints remain compatible with that accelerated path. The reported end-to-end inference figures cover text and VAE encoding, not action decoding in isolation. They are attributed to NVIDIA H20 and RTX 4090 systems, so they should be read as repository-reported measurements with named hardware rather than universal guarantees.

FastWAM also changes the default scheduler shift to `1.0` for training and evaluation. The repository reports similar behavior across shifts from `1.0` to `3.0` in its experiments. For older released checkpoints, however, the README specifically instructs evaluators to set `EVALUATION.sigma_shift=5.0`. This distinction matters because copying current defaults onto an older checkpoint can change the evaluation setup.

Maintenance, Licensing, and Operational Limits

The repository is not archived, and its last push was on 2026-08-20. No GitHub releases are listed, so there is no documented tagged-release schedule from which to infer upgrade compatibility. The package version in `pyproject.toml` is `0.1.0`. Adoption should therefore be based on repository state and the paper-linked experiment setup rather than an assumed stable release cadence.

The repository declares a custom license that GitHub cannot classify. The label does not state its permissions, restrictions, redistribution terms, or patent provisions. Anyone considering integration with commercial or redistributed work must inspect the repository's `LICENSE` file directly. No legal conclusion can be drawn from the label alone.

Operationally, the declared stack is large and tightly versioned. `pyproject.toml` pins `torch==2.7.1+cu128`, `torchvision==0.22.1+cu128`, `deepspeed==0.18.7`, `transformers==4.49.0`, and other exact versions. The package targets Python 3.10 or later. This can improve reproducibility within the intended environment, but it also means version changes should be treated as deliberate changes to the training or inference environment, not routine package refreshes.

Editorial conclusion

FastWAM fits researchers comparing explicit future-video imagination with direct action prediction, especially on LIBERO or RoboTwin. It is a poor fit for users seeking a small general-purpose robotics package, an install-and-run consumer tool, or a conventional release process. Before adoption, verify the custom terms in LICENSE, inspect the full environment setup for the required GPU software, and confirm that a named checkpoint matches your camera setup, dataset version, task, and inference mode.

Frequently asked questions

Official sources

1. Issues
2. Project website
3. README
4. yuantianyuan01/FastWAM on GitHub