tau-0-vla: a planner that searches, and a policy that executes

The planner searches, and the policy executes

The architecture is two policies, and the split is where the name hierarchical comes from.

A memory-augmented high-level policy generates the next subtask. The memory is not incidental: it updates from three camera views, so the planner carries state across steps rather than treating each observation in isolation. When the situation needs more reasoning, the policy uses world-model-guided test-time computation to search over alternatives instead of committing to the first proposal.

A generalist low-level policy then executes whichever subtask was selected, and it does so across robot embodiments. That word matters for the pretraining claim: the same low-level policy is meant to transfer between bodies rather than being retrained per robot.

The world model is the piece that makes the search concrete. It generates a goal image from the current observation and a subtask, so candidate next steps can be rendered and compared rather than only described in text.

The proposal and the world model are both released with weights, inference code and fine-tuning code, and the README asks you to use their separate environments, which is a practical signal that the two have different dependency sets rather than being a single package with two entry points.

A Qwen3.5 backbone with a flow matching action expert

The low-level policy is described in enough detail to be worth reading closely.

It combines a Qwen3.5 vision-language backbone with a Mixture-of-Transformers action expert trained through conditional flow matching. So the language and vision side is a pretrained vision-language model, and the action head is a separate expert of a different architecture, trained with a generative objective rather than by regressing actions directly.

Two numbers define the interface. The policy uses a unified 40-dimensional state and action space, which is what lets one checkpoint serve more than one robot: the embodiment difference is absorbed into a fixed-width vector instead of requiring a per-robot head. And it was trained on 40,115 hours of heterogeneous real-world robot data with multimodal co-training, meaning several data modalities were mixed during training rather than handled by separate stages.

The implementation notes confirm the shape. The models directory holds the Qwen3.5 backbone and the flow-matching action expert, and the dependency list pins a package whose gated-delta-rule kernels the Qwen3.5 backbone imports, which is a fairly strong hint that the backbone is used as-is rather than reimplemented.

Four checkpoints for four different jobs

Four model checkpoints are published, and each is scoped rather than being a variant of the same thing.

The base τ₀-VLA checkpoint is the pretrained low-level policy, and it is the one you start from when post-training for your own robot.

The LIBERO checkpoint is separate, described as a checkpoint for simulation evaluation. It fine-tunes the pretrained low-level policy for end-effector control across the Spatial, Goal, Object and Long task suites. That it is a distinct artefact rather than a flag on the base model is because the control space changes: end-effector control is not the same problem as joint control.

The Proposal checkpoint is the high-level planner, with task memory and three-camera input.

The World Model checkpoint is described as a robotics LoRA for goal-image generation, which is a compact way to ship a component that would otherwise need its own fine-tuning run.

Weights live on Hugging Face under the sii-research organisation, with the paper on arXiv and a project website alongside. The repository publishes no GitHub releases, so there is no tag history to consult and the checkpoints are the versioning surface.

The pinned stack was verified on one eight GPU node

The reference environment is Python 3.11, CUDA 12.8 and PyTorch 2.7.1, and installation is three commands:

git clone [email protected]:sii-research/tau-0-vla.git
cd tau-0-vla
bash scripts/setup.sh

The requirements file is unusually explicit about what it is for. Its header states that these are the versions the release was verified on, on a single 8xH200 node running the shipped example end to end: fine-tuning from the released checkpoint, open-loop evaluation, and HTTP serving.

That sentence is the most useful line in the file, because it tells you the scale of the environment the reference path assumes. Everything is exact-pinned, including the core model stack, the numerics, the data tools, the training stack with DeepSpeed and the linear attention kernels, and the serving pair.

One dependency is deliberately absent. `flash-attn` is installed separately by the setup script with build isolation disabled, after torch is present, because pip's default isolated build cannot see the already-installed CUDA torch package. That is a workaround for a real failure mode rather than a style choice.

A new robot starts from the template, not the example

The repository ships a worked example and a template, and the README is clear that the template is the starting point for anything new.

The example data directory contains a small AgiBot World subset in LeRobot v3.0 format, with its matching post-training recipe in a configuration directory. Training it is one command with the config and a checkpoint:

bash scripts/train.sh configs/example_agibot_world_gong/train.yaml \
    --model_name_or_path /path/to/tau-0-vla-checkpoint

For a different dataset or a different robot, the instruction is to start from the configuration template and from the adapter template under the source tree. Both are pointed at by their own README.

The adapters directory is described as holding embodiment-specific data layouts and deployment input and output, which is exactly the part that has to change per robot given the unified 40-dimensional state and action space. The data directory handles LeRobot loading, prompting, masking and normalisation, and separate documents cover the dataset format, the data pipeline and the robot adapters.

So the intended path is: read the adapter contract, copy the template, point it at your data, fine-tune from the released checkpoint.

Serving splits joint control from end-effector

There are two serving paths, and they are not interchangeable.

Hardware serving uses joint-control checkpoints. Open-loop evaluation takes a checkpoint directory as well:

python -m deploy.server --model outputs/<run_name>

python deploy/openloop.py --ckpt outputs/<run_name> --no-plot

LIBERO simulation instead uses a dedicated end-effector policy server. That is the same distinction the checkpoint table implies, expressed as an operational one: a real robot needs the joint-control path, and the benchmark needs the end-effector one.

The `--no-plot` flag has a matching entry in the requirements file, where matplotlib is described as imported lazily so that runs with the flag never need it. That is a small thing that tells you the plots are a convenience rather than part of the evaluation.

The deploy directory also holds the payload and action-order contracts, which is the document to read before writing a client. Action order in particular is the kind of convention that silently produces wrong trajectories when it is assumed rather than read.

Two dependency files, split on purpose

The manifest and the requirements file divide the dependencies, and the comments say why.

The project metadata carries what you need to import the package and run the pipeline: torch pinned to 2.7.1, transformers in a narrow 5.5 to 5.6 range, accelerate, peft, safetensors, numpy, scipy, einops, the LeRobot reader, pillow and pyyaml. Three more are declared even though they are reached through LeRobot, because they are imported directly: pyarrow by the norm-stats reader, torchcodec by the video decoder, and torchvision by the colour jitter transform.

The training and serving stack lives in the requirements file instead, because those packages pin to CUDA-specific builds that cannot be resolved from a plain index on every machine. The optional `serve` extra is the exception that keeps FastAPI and Uvicorn reachable without that file.

Two smaller details. The metadata carries a comment explaining that no OSI licence classifier is declared, because PEP 639 supersedes it with the licence expression and setuptools rejects a build declaring both. And the lint configuration selects only the import-sorting rule, with a line length of 120 and a py311 target, so the enforced style is deliberately minimal.

Four release steps in two months

The news list is short and dated, and it shows the project arriving in stages rather than all at once.

On 2026.07.27 the τ₀-VLA model, the paper, the project website and the Hugging Face entries were released. On 2026.08.19 the team announced a plan to progressively release the components of the high-level policy. On 2026.09.20 the LIBERO post-training and simulation evaluation were released. On 2026.09.21, the day this repository was last pushed, the high-level proposal and world model went out with weights, inference and fine-tuning code.

So the low-level policy, the benchmark path and the planner arrived roughly two months apart, and the full system described in the overview has been complete for less than a week as of this writing.

Licensing is split in the same spirit. The code and the model weights are under Apache-2.0, while the bundled example data follows whatever licence its own README describes, which is the arrangement to check before redistributing the dataset with your own fine-tuned weights.

Editorial conclusion

Use this repository if you are post-training a generalist manipulation policy and want a long-horizon planner above it rather than a single flat VLA, because the two halves are separately released and separately configured. Expect a heavy environment: the pinned stack was verified on a single eight-GPU H200 node, and the serving extras assume a CUDA-specific build. Before you start, decide whether you need the hardware-serving path or the LIBERO simulation path, since they use different policy servers, and read the adapter template rather than copying the bundled example configuration.

Frequently asked questions

Official sources

1. Issues
2. License: Apache-2.0
3. README
4. sii-research/tau-0-vla on GitHub