GRAIL: synthesising humanoid loco-manipulation data instead of collecting it

The problem is data, and the answer is to synthesise it

GRAIL is described as a fully digital data-generation pipeline for humanoid loco-manipulation, presented at CoRL 2026 as an oral presentation. The repository description calls it a fully digital data-generation pipeline that synthesizes humanoid loco-manipulation data from world models.

The mechanism is a chain, and the README spells out each link. GRAIL composes 3D assets, simulator-ready scenes, robot-proportioned characters, and video foundation model priors to synthesize metric 4D human-object interaction (HOI) trajectories. Those trajectories are then retargeted to a Unitree G1, and used to train task-general policies for pick-up, whole-body manipulation, sitting and terrain traversal.

Two words in that sentence carry the argument. Metric means the trajectories carry real physical units rather than being normalised video frames, which is what makes retargeting to a robot with different limb lengths possible at all. And task-general means the policies are not one policy per demonstration, so the training set is reusable across the four task families rather than being a set of demonstrations for one task.

The last claim in the README is the one to read with care. Using only GRAIL-generated data, the resulting egocentric visual policies transfer to real-world object pick-up and stair-climbing. That is a sim-to-real result derived entirely from synthetic motion, with no teleoperation data in the loop, and the gallery of videos in the README shows the simulated motions and the deployed robot separately.

Four pipeline stages, and a documentation page for each

The documentation index lists the stages in the order they run, and the shape of the list is the architecture.

Asset generation comes first, with a page for 3D asset generation. Then 2D HOI generation, which is where a video model is asked for human-object interaction footage, followed by 4D HOI reconstruction, which recovers metric 3D motion from that footage. After that come retargeting, which maps the recovered human motion onto the robot, and task general tracking, which is the training stage. Data export, data visualization and a web visualizer close the list, which are the stages you use to get the result out and look at it.

The pipeline names match the Python module names, so the documentation and the commands line up. The pipeline pages are gen_3d_assets, gen_2dhoi, recon_4dhoi, retargeting, tracking, data_export, visualization and web_visualizer.

Two of the stages carry the weight of the whole approach. The 2D generation stage is where a video foundation model provides the prior about how a person moves around an object, and the 4D reconstruction stage is where that prior is turned back into metric geometry, because a video model gives you pixels and a robot needs coordinates.

The task general tracking page also documents something specific. The README's news section records that on 2026-07-15 the project released policy checkpoints trained on the released GRAIL data, which is the first thing to try if you want to see the result rather than reproduce it.

Two container images, and what is deliberately not inside them

The quick start asks you to choose a Docker image, and there are two, differing in one specific way.

| `docker.io/nvgrail/grail:1.1.1` (`:latest`) | Standard GRAIL with Kling; no MiniMax-H3/SGLang runtime |
| `docker.io/nvgrail/grail:1.1.1-minimax-h3` (`:minimax-h3`) | Standard GRAIL plus an isolated MiniMax-H3/SGLang runtime |

The first is standard GRAIL with Kling, the hosted video generation service, and it carries no MiniMax-H3 or SGLang runtime. The second is standard GRAIL plus an isolated MiniMax-H3/Sol-Engine runtime, which is the local video model path added in August 2026 for the 2D HOI pipeline.

That is the real alternative in this project, and the difference is concrete. The standard image calls out to a hosted API and needs an API key; the MiniMax-H3 image runs the model locally, which is what you need when the footage cannot leave your network or when you do not want to depend on a service's availability.

The README is unusually careful about what the images contain. Both are Linux AMD64 images holding runtime environments only. They do not contain the GRAIL or Sana source, the GRAIL or MiniMax-H3 checkpoints, API keys, or user data. Keep credentials outside the image and provide them only at runtime.

That separation is a deliberate security property rather than an accident of packaging. Checkpoints and source arrive from GitHub and Hugging Face where they are versioned and licensed, while the container carries only the libraries.

The install is a clone, a pull and a container with the GPU attached

The quick start sequence is short and every line does something you will notice if it goes wrong.

git clone https://github.com/NVlabs/GRAIL.git
cd GRAIL
git submodule update --init --recursive

# The public Docker Hub images can be pulled without registry credentials.
export GRAIL_IMAGE=docker.io/nvgrail/grail:1.1.1
docker pull "$GRAIL_IMAGE"

The submodule update is not optional, because the repository carries a .gitmodules file and the pipeline depends on the code it points at.

The run then mounts your checkout into the container, and the two flags on the docker run line are the ones that matter:

docker run --gpus all -it --shm-size=16g \
    -v "$PWD":/workspace/grail \
    "$GRAIL_IMAGE"

`--gpus all` gives the container the GPU, without which the rendering stages have nothing to render on. `--shm-size=16g` enlarges the container's shared memory, which is what a data loader with many workers and large tensors needs, and the default Docker shared memory size is far too small for it. The volume mount is what makes your checkout live inside the container rather than copied into it.

Inside, the setup is three scripts and an environment:

cd /workspace/grail
bash scripts/setup/install_env_docker.sh   # validates native extensions, downloads Blender
bash scripts/setup/download_checkpoints.sh # GEM-SMPL / GEM-SOMA / FoundationPose weights
bash scripts/setup/download_comasset.sh --category cordless_drill # quick-start object
source /root/miniconda3/etc/profile.d/conda.sh
conda activate grail
[ -f .env ] && source .env                  # OPENAI_API_KEY, KLING_*, HF_TOKEN

The first script rebuilds GPU and Python specific native extensions when needed, naming nvdiffrast and the FoundationPose mycpp extension, and installs Blender into the mounted checkout. The second fetches the GEM-SMPL, GEM-SOMA and FoundationPose weights, and the third pulls a single object category to start with. The credentials live in a .env file on your side of the mount, which is the practical form of the README's instruction to keep them outside the image.

Invoke stages as modules, not as wrapper scripts

The README makes a point that will save you an afternoon if you know it in advance: pipeline stages are package entrypoints, so you invoke them with python -m grail.pipelines.* rather than the project-root wrapper scripts.

The three commands it shows cover the asset and reconstruction path:

python -m grail.pipelines.gen_terrain --type stairs --num 50 --output_dir data/syn_stairs
conda run -n hunyuan python -m grail.pipelines.gen_3d_assets \
    -i configs/gen_3d/example_objects.yaml -o data/gen_example

# 2D HOI generation (Blender + Kling video)
python -m grail.pipelines.gen_2dhoi --dataset ComAsset --category cordless_drill \
    --character kid --results_dir results --video_model_api kling-ai

# 4D HOI reconstruction
python -m grail.pipelines.recon_4dhoi --dataset ComAsset --category cordless_drill --results_dir results

The terrain generator is procedural, so asking for fifty stairs is a flag rather than an asset library. The 3D asset stage is the exception to the single-environment rule, because it runs under a separate conda environment named hunyuan, which is what the conda run wrapper is for. The 2D stage names its video backend with --video_model_api kling-ai, and the character is a parameter, so the human proportions in the video are something you choose rather than inherit.

The 4D reconstruction stage takes the same dataset and category as the generation stage and the same results directory, which shows the pipeline's contract: each stage reads what the previous one wrote, and the naming is how you find it.

The checkpoints exist, the manipulation dataset does not yet

The TODO list at the end of the README is the honest part of the release, and it has three items with two of them unchecked.

Released task-general tracking policy checkpoints is done. Provide a quick-start demo script is not, and Release GRAIL manipulation dataset is not.

That second gap is the one that matters for adoption. The README's News section says that on 2026-07-15 the project released policy checkpoints trained on the released GRAIL data, and links both the tracking guide and a Hugging Face dataset page, nvidia/PhysicalAI-Robotics-Locomanipulation-GRAIL. The checkpoints therefore exist and can be used. What the TODO list still marks as outstanding is the manipulation dataset itself, so a reader should assume the training data for manipulation tasks is not yet published in full.

The code itself was released on 2026-06-03, so the repository has been public since then and the project page, the paper on arXiv at 2606.05160 and the rendered documentation at nvlabs.github.io/GRAIL/ are the reference points for the parts the README does not spell out.

Two smaller constraints belong in the same list. The environment is Linux AMD64 for the images, so the pipeline is not set up for macOS or Windows hosts, and the Docker run line asks for a GPU, so the stages are not CPU-only paths.

A custom licence and a CUDA base image pinned to one GPU generation

The licence is the first thing to read. The repository listing records it as a custom licence that GitHub cannot classify, with a LICENSE file at the root, and the repository also carries a SECURITY.md, so there is a stated place for vulnerability reports as well.

The container base is where the hardware expectation shows up. The Dockerfile starts from nvidia/cuda:12.8.0-devel-ubuntu22.04, and the PyTorch install is commented as CUDA 12.8 for RTX 5090, with the wheels pulled from the cu128 index. That is a specific generation of consumer GPU rather than a generic requirement, and it is worth checking against the hardware you have before you plan around the pipeline.

The rest of the image is built for headless rendering rather than for a desktop. The system packages include ffmpeg, xvfb, libeigen3-dev, libboost-all-dev, the Mesa GL and EGL development packages, gosu and git-lfs, and the environment sets PYOPENGL_PLATFORM to egl and EGL_PLATFORM to surfaceless, which is how Blender renders without a display server. Miniconda is installed, a conda environment named grail is created with Python 3.10, and the entrypoint comes from tools/docker-entrypoint.sh with the working directory set to /workspace/grail.

The Python side shows the same care. The build uses setuptools and wheel, black and ruff both target py310 with a line length of 100, and the ruff configuration selects the B, E, F, I, UP and W rule sets with a long ignore list whose comments explain each exception, from bare except as legacy code to the star imports used extensively in the FoundationPose integration and the zip without explicit strict that is pervasive in machine learning code. The repository also has a .pre-commit-config.yaml.

There are no GitHub releases, the default branch is main, and the last push was on 2026-10-15, so the code is moving but there is no tagged version to pin to.

Editorial conclusion

GRAIL fits a robotics group that needs humanoid loco-manipulation demonstrations without the cost of collecting them on real hardware, and that can afford a CUDA workstation with the disk space for Blender, video models and several sets of weights. It is the wrong choice for a team that needs the manipulation dataset today, because the README still lists releasing it as an unfinished item, and the wrong choice if you need a stable API, since the project is a research release with no tagged versions and an entrypoint convention that changed between stages. Verify first which of the two images matches your video generation plan, that the checkpoints your stage needs are actually downloadable, and that your GPU can carry the rendering workload.

Frequently asked questions

Official sources

1. Issues
2. NVlabs/GRAIL on GitHub
3. Project website
4. README