# UMI on Legs: putting a trained manipulation policy on a robot dog

> UMI on Legs is a research framework that lets a visuomotor manipulation policy trained on human demonstrations run on a quadruped with an arm, by putting a manipulation-centric whole-body controller underneath it. The code is published as separate submodules so each piece can be taken on its own.

**real-stanford/umi-on-legs** — UMI on Legs: Making Manipulation Policies Mobile with Manipulation-Centric Whole-body Controllers

- Repository: https://github.com/real-stanford/umi-on-legs
- Website: https://umi-on-legs.github.io/
- Stars: 558 · Forks: 37
- Language: Python
- License: MIT
- Published: 2026-10-22 · Updated: 2026-10-22 · Language: en
- Canonical page: https://hysenlabs.com/projects/real-stanford-umi-on-legs

## The idea: a policy that only knows how to move an arm

The project name carries the whole concept. UMI on Legs is a framework for combining real-world human demonstrations with simulation trained whole-body controllers, providing a scalable approach for manipulation skills on robot dogs with arms.

The problem it attacks is specific. A manipulation policy trained on human demonstrations, using UMI data, produces relative end-effector commands: move the gripper here, orient it like that, close. That policy knows nothing about balance, locomotion or where the robot's feet are. Run it on a standing quadruped and the robot falls over as soon as the arm moves, because moving the arm shifts the centre of mass.

The fix is to put a controller underneath the policy. The whole-body controller is trained in simulation to keep the robot stable while tracking whatever end-effector trajectory the manipulation policy asks for. The policy stays unchanged, and the controller absorbs the balance problem. That is what the README means by making your manipulation policies mobile, and the framing in the project is that you can plug-and-play your existing visuomotor policies onto a quadruped.

The manipulation-centric qualifier in the controller name is the design decision worth understanding. The controller is not a generic whole-body controller that treats the arm as an appendage to be kept out of the way. It is designed around the manipulation task, which means the objective it optimises includes tracking the commanded end-effector pose rather than merely maintaining a stable posture.

The authors are Huy Ha and Yihuai Gao with Zipeng Fu, Jie Tan and Shuran Song, from Stanford University, Columbia University and Google DeepMind, and the work was published at arxiv 2407.10353 with a project page at umi-on-legs.github.io. The primary language is Python and the licence is MIT.

## The repository is four submodules, on purpose

The top-level tree is unusually small: .gitmodules, LICENSE, README.md, arx5-sdk, iPhoneVIO, mani-centric-wbc/ and real-wbc/. Everything else is a submodule pointing elsewhere.

The README explains the reasoning, and it is a good model for how a robotics paper with a lot of hardware should be released. The code is published in a similar fashion to how it was developed, as separate submodules, with the hope that the community can easily take any component they find useful out and plug it into their own system.

So the four pieces are independently useful.

The whole-body controller simulation training is mani-centric-wbc/, and that is where the PPO training, the IsaacGym environment, the reward design and the checkpointing live. The whole-body controller deployment is real-wbc/, which holds the real-robot side of the same controller plus the hardware documentation. The iPhone odometry application is iPhoneVIO, an iOS app that publishes ARKit camera poses. And arx5-sdk is the SDK for the ARX5 robot arm, which is the arm the quadruped carries.

The UMI environment class is separate again. The UMI deployment codebase heavily builds upon the original UMI codebase in the universal_manipulation_interface repository, and the README gives credit to the UMI team. The Universal Manipulation Interface itself is therefore not in this repository at all. Data collection, the hardware guide and the preprocessing pipeline all live in that other project, and the README links to a UMI data collection tutorial and a hardware guide document for them.

That split has a practical consequence for anyone starting. If your question is how to collect the demonstrations, this repository will send you elsewhere, and the UMI project is where that work is done.

## Getting the code and rolling out the controller

The README points anyone who wants to start running commands while skimming the paper at mani-centric-wbc/docs/starter.md, which downloads data, checkpoints and rolls out the whole-body controller. The rest of the documentation is focused on setting up real world deployment, which is the harder half of the work.

The documentation is split along the same lines as the code. Under the Getting Started heading the sub-links are Setup, Checkpoint and Data, Rollout, Evaluation and Curves. Training and simulation are documented in mani-centric-wbc/docs/wbc.md, with sections for Train, Robustifying Sim2Real and Extending, the last covering More Robots and More Manipulation Trajectories. Real world deployment has its own docs directory under real-wbc/docs/, including reflections on hardware choices, a bill of materials, the assembly guide, the 3D printing guide and a Unitree robots network setup page.

One evaluation link is a Weights and Biases curve page for the run, which tells you the training pipeline reports to W&B.

The training stack is the one most robotics readers will recognise. The whole-body controller simulation training started with Nikita Rudin's implementation of PPO and the Gym environment wrapper around IsaacGym, legged gym, and the README credits that codebase as the starting point. The IsaacGym dependency is the significant practical requirement, since it needs a recent NVIDIA GPU and its own environment.

The codebase also includes a modified Perlin Noise Terrain taken from DeepWBC, and the README is clear that although it is not used in the final results of the paper, it is enabled by a single configuration override when you run training.

```bash
env.cfg.terrain.mode=perlin
```

That override is a good example of how the project handles optional features: the terrain generator is a runtime configuration value rather than a separate branch, so the same training script produces both setups.

## Sim2Real, and the parts that touch real hardware

The Robustifying Sim2Real section of the controller documentation is where the project addresses the gap between a trained controller and a robot with a real arm, a real payload and real balance dynamics.

The hardware side of that is documented in real-wbc/docs, and the README's hardware choices page is titled Reflections on Hardware Choices, which suggests the authors treat the hardware decisions as part of the research rather than as an afterthought. There is a bill of materials, a 3D printing guide for the parts that need to be made, and an assembly guide.

The arm is the ARX5, and its SDK is published separately as arx5-sdk, credited to Yihuai Gao. The iPhone odometry application in iPhoneVIO is credited to Zhenjia Xu for starter code for ARKit camera pose publishing, and it exists so the controller can know the true pose of the arm end effector in the world rather than trusting a forward-kinematic estimate, which matters as soon as the arm flexes or the robot body moves on uneven ground.

The whole-body controller deployment code credits Qi Wu for providing the initial deployment script.

The robot platform is a Unitree quadruped, and there is a dedicated network setup page for Unitree robots, which is the sort of detail that separates a working deployment guide from a paper. Deploying the WBC on real robots has its own page in real-wbc/docs/codebase_setup.md.

For the OptiTrack motion capture setup, the README credits Jingyun Yang and Zi-ang Cao for providing the motion capture code and helping set it up. An external motion capture system alongside phone-based odometry is how the project gets both a reliable ground truth for the body and a phone camera feed for the vision part of the manipulation policy.

The documentation structure also makes the dividing line explicit. The getting started material is about simulation and rollouts, while the deployment documentation is about the robot, the network, the arm and the assembly. The project assumes you have hardware before it gets you to a working real-world system.

## What this repository is not

There are three limits, and the first is the one the README itself flags.

The first is that the UMI code is not here. The repository contains the whole-body controller simulation training, the whole-body controller real-world deployment, the iPhone odometry iOS application, the UMI real-world environment class and the ARX5 SDK, and the README carries a note that the UMI code is not available in this repository yet. The Universal Manipulation Interface data collection, hardware guide and preprocessing live in their own repository. So this is a locomotion and control contribution that consumes UMI data, not a complete manipulation stack, and a user who has never collected UMI demonstrations has to go and learn that pipeline first.

The second is the hardware dependency. The full workflow needs a Unitree quadruped, an ARX5 arm, an OptiTrack motion capture setup, an iPhone running the ARKit odometry app, and the 3D printed and assembled parts described in the deployment docs. That is a real research platform, and the bill of materials page exists because the parts list is nontrivial. The simulation path is the accessible one, since training runs in IsaacGym with legged gym, and the rollout path needs a GPU but not a robot.

The third is that the project is a research release rather than a maintained package. It has no GitHub releases at all, the last push was on 2026-07-20, and the paper dates from 2024. The MIT licence is permissive and the code is readable, but there is no version to pin, no changelog to consult and no package you can install from an index. You are working with submodules of scripts and training code, which is the expected shape for this kind of artifact and also means every dependency is a manual step.

The honest alternative depends on what you are trying to do. For running a manipulation policy on a legged robot with an existing policy, the alternative is to write the whole-body controller yourself against your own robot, which is what the Extending section with its More Robots and More Manipulation Trajectories entries effectively invites, and the submodule structure is designed to let you do that. For collecting demonstrations without the UMI rig, the universal_manipulation_interface repository is the path and it is a different piece of software with its own hardware guide. For a commercial manipulation deployment, neither this nor UMI is a product: both are research artifacts released to be extended, and the parts you keep will be the parts you end up maintaining yourself.

## Conclusion

Adopt UMI on Legs if you already have a UMI-style visuomotor manipulation policy and want to run it on a robot dog with an arm, since the whole-body controller is the piece that makes the platform work. Do not adopt it as a ready-made manipulation stack, because the UMI data collection code lives in a separate repository. Before you start, clone with submodules and read mani-centric-wbc/docs/starter.md, and note the last push was on 2026-07-20 with no tagged releases.

## FAQ

### What is UMI on Legs and what problem does it solve?

It is a framework for combining real-world human demonstrations with simulation trained whole-body controllers, so manipulation skills can be applied to robot dogs with arms. A visuomotor manipulation policy that only outputs end-effector commands is left unchanged, and a manipulation-centric whole-body controller underneath keeps the quadruped stable while tracking those commands.

### How is the UMI on Legs repository organised?

As separate submodules: mani-centric-wbc for the whole-body controller simulation training, real-wbc for the whole-body controller real-world deployment, iPhoneVIO for the iPhone odometry iOS application, and arx5-sdk for the ARX5 robot arm SDK. The README states the code is published in the same fashion as it was developed so that individual components can be taken out and plugged into another system.

### Where is the Universal Manipulation Interface code?

Not in this repository. The UMI deployment codebase builds on the original UMI codebase in the universal_manipulation_interface repository, and data collection, the hardware guide and the preprocessing pipeline are documented there, while this repository covers the whole-body controller, the UMI environment class, the iPhone odometry app and the ARX5 SDK.

### What does UMI on Legs require to train and to deploy?

Training uses PPO with a Gym environment wrapper around IsaacGym, starting from the legged gym implementation by Nikita Rudin, with a modified Perlin Noise Terrain from DeepWBC available through env.cfg.terrain.mode=perlin. Real deployment additionally needs a Unitree quadruped, an ARX5 arm, an OptiTrack motion capture setup and the iPhone ARKit odometry app.

### Where do I start with UMI on Legs?

The README points to mani-centric-wbc/docs/starter.md, which downloads data and checkpoints and rolls out the whole-body controller, with sections for setup, downloads, rollout, evaluation and curves. Training and sim2real work is documented in mani-centric-wbc/docs/wbc.md and real world deployment in real-wbc/docs.

## Sources

- [Issues](https://github.com/real-stanford/umi-on-legs/issues)
- [License: MIT](https://github.com/real-stanford/umi-on-legs/blob/main/LICENSE)
- [Project website](https://umi-on-legs.github.io/)
- [README](https://github.com/real-stanford/umi-on-legs/blob/main/README.md)
- [real-stanford/umi-on-legs on GitHub](https://github.com/real-stanford/umi-on-legs)

---

Hysen Labs editorial analysis, written from the project's own repository and release notes. Cite the canonical page: https://hysenlabs.com/projects/real-stanford-umi-on-legs