Ctrl-World, a world model you can drive with a keyboard

A world model for policy evaluation, not for control

Ctrl-World is the official PyTorch implementation for the ICLR 2026 paper of the same name, a controllable generative world model for robot manipulation, from authors at Stanford University and Tsinghua University, with the project page at ctrl-world.github.io and the paper on arXiv.

The description in the README is the clearest statement of the contribution, and it is worth reading closely because it defines what the model is for.

Ctrl-World is an action-conditioned world model compatible with modern VLA policies, and it enables policy-in-the-loop rollouts entirely in imagination, which can be used to evaluate and improve the instruction following ability of VLA.

Three phrases carry the idea. Action-conditioned means the model is given a sequence of actions and generates what the camera would see as a result, so the model is a forward simulator of the scene rather than a video generator that ignores the robot. Compatible with modern VLA policies means it consumes the same action chunks a vision-language-action policy emits, so the two can be composed. And entirely in imagination means the policy under test never touches a real robot: it acts inside the world model, and the model generates the next observation.

That last point is the reason to care. Evaluating a robot policy normally means running it on a real robot, which is slow, wears out hardware, can break things, and is unsafe. Evaluating it in a learned simulator moves that loop into a GPU, where a policy can be run for thousands of episodes and compared against another policy on the same initial conditions.

The cost of that is fidelity, and the authors say so themselves. The repository carries a claim in italics: the model can evaluate a policy's instruction-following capability, but also can be imprecise in modeling physical interactions.

So the scope is narrow and deliberate. It is a model of what a policy will see and how the scene will respond, used to check whether a policy does what it was asked, not a model of friction, contact forces and grasp stability. A team that needs the latter is looking for a physics simulator, and this is not one.

Three ways to drive it, one of them a keyboard

The repository documents three interaction modes, and they are worth comparing because they show what the model was built to support.

The first is replaying recorded trajectories. You start from an initial observation sampled from the recorded trajectories and generate long trajectories by replaying the recorded actions. At each interaction step a one second action chunk is provided to the world model, and the interaction is repeated to produce the full rollout.

CUDA_VISIBLE_DEVICES=0 python scripts/rollout_replay_traj.py  --dataset_root_path dataset_example --dataset_meta_info_path dataset_meta_info --dataset_names droid_subset --svd_model_path ${path to svd folder} --clip_model_path ${path to clip folder} --ckpt_path ${path to ctrl-world ckpt}

The action chunk of one second is the unit of interaction, which tells you how the model is conditioned. It is not a single motor command, it is a short sequence, which is the granularity a modern VLA policy actually produces.

The second mode is keyboard control, and this is the one that makes the model fun to play with. You begin from an initial observation and use keyboard commands to control the robot interactively. The valid commands are l for left, r for right, f for forward, b for backward, u for up, d for down, o for open gripper and c for close gripper. You can input multiple commands at once, and the system executes them sequentially in an autoregressive manner.

CUDA_VISIBLE_DEVICES=0 python scripts/rollout_key_board.py  --dataset_root_path dataset_example --dataset_meta_info_path dataset_meta_info --dataset_names droid_subset --svd_model_path ${path to svd folder} --clip_model_path ${path to clip folder} --ckpt_path ${path to ctrl-world ckpt} --task_type keyboard --keyboard lllrrr

The example command string, `lllrrr`, is three lefts and three rights, executed one after another with each one generating a new observation that feeds the next. That autoregressive structure is the model doing the work: each action chunk is conditioned on the observation the model itself produced a moment earlier, so the trajectory can drift from the recorded data in ways the training set never showed.

The third mode is the one the paper is really about. You take a snapshot from a new DROID setup and perform policy-in-the-loop rollouts inside the world model, so that both the policy and Ctrl-World have to zero-shot transfer to a new setup. The example tasks are `pickplace`, `towel_fold`, `wipe_table`, `tissue`, `close_laptop` and `stack`.

CUDA_VISIBLE_DEVICES=0 XLA_PYTHON_CLIENT_MEM_FRACTION=0.4 python scripts/rollout_interact_pi.py  --dataset_root_path dataset_example --dataset_meta_info_path dataset_meta_info --dataset_names droid_subset --svd_model_path ${path to svd folder} --clip_model_path ${path to clip folder} --ckpt_path ${path to ctrl-world ckpt} --pi_ckpt ${path to ctrl-world ckpt} --task_type ${pickplace}

The `XLA_PYTHON_CLIENT_MEM_FRACTION=0.4` in that command is the giveaway that two models are resident on the same device, and that they are competing for the same memory pool.

The assets and what they cost

The checkpoint and dataset table is the most practically useful part of the README, and it is worth reading line by line because the sizes determine whether the project is runnable for you.

The table has four rows. The CLIP text and image encoder, `clip-vit-base-patch32`, at about 600M. A pretrained SVD video diffusion model, `stable-video-diffusion-img2vid`, at about 8G. The Ctrl-World model trained on the DROID dataset at about 8G. And the DROID dataset itself, the opensourced dataset, at about 370G with roughly 95k trajectories across 564 scenes.

The 370 gigabytes is the number that decides things. The repository does provide a very small subset of the DROID dataset in `dataset_example/droid_subset`, and the rollout commands point at `dataset_example` rather than the full set, so the intended first run is on the subset. But to replay more trajectories, you need to download and process the original DROID datasets, following the instructions in the training section.

The 564 scenes are the other number to hold on to, because they are the diversity of the training data. A world model trained on 564 scenes is a model of those scenes, and the zero-shot transfer claims are about new setups of the same kind of hardware, not about new kinds of environment.

The three model components suggest the architecture in outline. A CLIP encoder is there to turn a language instruction and an image into embeddings, a video diffusion model is the generative backbone that produces the next frames, and the Ctrl-World checkpoint is the trained part that ties them together and handles the action conditioning. The 8G weight of the diffusion model and the 8G of the trained checkpoint are comparable, which is consistent with the trained model being built on top of the pretrained one rather than replacing it.

The requirements file pins the exact versions of the three libraries that matter for reproducing a result.

diffusers==0.34.0
transformers==4.48.1
torch==2.7.1
mediapy
wandb
swanlab
accelerate
tqdm
numpy
decord
einops
scipy
pandas

Pinned diffusers and transformers versions are what you would expect in a research codebase where a checkpoint has to load with the same classes it was trained with. The rest of the list is the ordinary scientific Python stack, with `decord` for video decoding, `mediapy` for writing video, and both `wandb` and `swanlab` present for experiment tracking.

Installing, and the optional dependency that doubles the work

The installation is a conda environment, a requirements file, and one optional extra that is easy to miss.

conda create -n ctrl-world python==3.11
conda activate ctrl-world
pip install -r requirements.txt

Python 3.11 exactly, which is the environment the authors used, and the requirements install is a plain pip of the pinned file.

The optional part is the interaction with the pi_0.5 model. If you want to use Ctrl-World to interact with that model, you follow the pi official repository to install the pi model dependencies, and otherwise you can skip it. Those steps are a separate clone and a separate install.

git clone --recurse-submodules [email protected]:Physical-Intelligence/openpi.git
cd openpi
pip install uv
GIT_LFS_SKIP_SMUDGE=1 uv sync
GIT_LFS_SKIP_SMUDGE=1 uv pip install -e .

The `--recurse-submodules` flag on the clone and the two `GIT_LFS_SKIP_SMUDGE=1` environment variables are the details worth noting. The submodules are needed for the pi model to build, and skipping the Git LFS smudge avoids downloading the large model weights through the sync step, which is a large download you can do separately and deliberately.

The `pip install uv` step is a bit odd on the surface, and it is there because the pi repository uses uv rather than pip for its own environment management. So the installation is two dependency managers in sequence, and the two ecosystems are not merged.

The rollout scripts take their paths as arguments rather than reading configuration, which is why the command lines are long. The dataset root and metadata path, the dataset name, the path to the SVD folder, the path to the CLIP folder and the checkpoint path are all explicit, and the model paths are placeholders in the documentation that you fill in. The rollout configuration itself, the paper says, can be found in `config.py` in the function `__post_init__`, which is a dataclass pattern where the post-initialisation hook applies the derived defaults.

The runtime figure to plan around is in the tip at the end of the replay section: one interaction step takes around ten seconds on an A100 or about five seconds on an H100. Since an interaction step is a one second action chunk, a long rollout is minutes rather than seconds, and a policy evaluated over thousands of imagined episodes is a job for a cluster.

What the repository does not give you

Three limits are worth stating, and the third is the one that decides whether this project is useful for your work.

The first is the absence of a training entry point in the visible documentation. The repository has `scripts/`, `config.py`, `config_eval.py`, `models/`, `dataset/` and `synthetic_traj/`, and the README's content list mentions a training pipeline of Ctrl-World on the DROID dataset, with the training section referenced for the full dataset instructions. But the documented commands are the three rollout scripts, and the training instructions are behind the link in the training section. So the repository is presented as inference-first, with training documented but not in the front page.

The second is that there are no GitHub releases. The last push to the repository was on 2026-04-08, and the project is a paper artefact rather than a released library. The licence is MIT, and the top-level layout is a research repository: `config.py` at the root rather than a package directory, a `requirements.txt` rather than a pyproject, `dataset_example/` and `dataset_meta_info/` as data directories, and `synthetic_traj/` for generated trajectories.

The third is the fidelity limit, and it is the one to think about before planning around the model. The authors' own claim is that the model can evaluate a policy's instruction-following capability, but also can be imprecise in modeling physical interactions. That is the right way to read the whole project. The model is good enough to answer a question of the form, does this policy do what the instruction said, in a scene like this one, and not good enough to answer whether the grasp will hold.

So the honest use is comparative rather than absolute. If you are choosing between two instruction-following policies, running both in the same world model from the same initial observation is a controlled comparison, and the imprecision affects both arms equally. If you are trying to predict what will happen on a real robot, the imprecision is a direct error in your answer.

The alternatives are two. A physics simulator, with a real rigid body engine, is the tool for asking what will physically happen, at the cost of the realism that a learned model has and the speed that a physics engine does not. And a real robot, which is the ground truth, at the cost of everything that motivated the world model in the first place.

The contribution of Ctrl-World is the middle position, and it is a genuinely useful one: a policy evaluation loop that runs on a GPU instead of a robot, with the fidelity caveat stated by the people who built it.

Editorial conclusion

Ctrl-World suits a robotics researcher who wants to evaluate or improve a vision-language-action policy without collecting more robot data, and it suits a developer who wants to understand action-conditioned video generation from a working implementation. It does not suit a team that needs a simulator whose physics it can rely on, because the model is explicitly imprecise about physical interaction. Verify first that your task appears among the ones the authors tried, that the GPU you have matches the ten seconds per interaction step on an A100 they quote, and that you have the disk for the DROID dataset and the checkpoints, which the table puts at roughly 370 gigabytes plus the models.

Frequently asked questions

Official sources

1. Issues
2. License: MIT
3. Project website
4. README
5. Robert-gyj/Ctrl-World on GitHub