# Universal Manipulation Interface for Robot Teaching

> Universal Manipulation Interface, or UMI, is a Python and robot-learning pipeline for collecting manipulation demonstrations in the wild and converting them into data for Diffusion Policy training. Its main constraint is the SLAM stage, which the README identifies as the most fragile part of the workflow.

**real-stanford/universal_manipulation_interface** — Universal Manipulation Interface: In-The-Wild Robot Teaching Without In-The-Wild Robots

- Repository: https://github.com/real-stanford/universal_manipulation_interface
- Website: https://umi-gripper.github.io/
- Stars: 1,612 · Forks: 295
- Language: Python
- License: MIT
- Published: 2026-10-16 · Updated: 2026-10-16 · Language: en
- Canonical page: https://hysenlabs.com/projects/real-stanford-universal-manipulation-interface

## What UMI is built to solve

Universal Manipulation Interface addresses a specific robotics problem: collecting useful manipulation demonstrations without requiring every demonstration to be performed inside a conventional robot laboratory or teleoperation environment. The project page describes the idea as robot teaching in the wild. The README connects this goal to a complete path that includes UMI SLAM, dataset generation, Diffusion Policy training, and real-world deployment. This makes UMI relevant to robotics engineers who need demonstrations from varied locations and want the resulting data to feed a policy-training workflow.

The intended user is not someone who only wants a simulator or a standalone grasp planner. The repository is designed around a physical setup that includes cameras, a robot arm, a gripper, and deployment software. It supports up to two arms and an unlimited number of cameras according to the deployment section. The example workflow uses a UR5 with a teach pendant and a WSG50 gripper, while `franka_instruction.md` covers Franka. That hardware breadth is useful, but it also means adoption depends on matching one of the documented integration paths.

UMI is therefore better understood as a collection-to-training system than as a single library. Its path runs from captured demonstrations to a replay-buffer dataset and then to a trained policy. The README links to a hardware guide, a data collection tutorial, and a fork of ORB_SLAM3. Those external documents are part of the operational surface of the project, because the Python code alone does not describe the complete capture setup.

## The UMI data path and processing stages

The repository layout separates the workflow into `run_slam_pipeline.py`, scripts under `scripts_slam_pipeline/`, the `diffusion_policy/` package, `train.py`, and deployment code under `umi/`. The documented first stage accepts a raw demonstration session and runs the UMI SLAM pipeline. The example output identifies cameras by serial number and assigns camera indices such as `right=0`, `left=1`, and `non_gripper=2,3`. It also reports how much raw data was used and how many demonstrations were dropped. These diagnostics are central to the pipeline because unusable camera geometry or failed tracking affects the dataset before policy training begins.

After SLAM, the command `07_generate_replay_buffer.py` produces a compressed Zarr dataset from the session directory. That dataset is the input to the training command, where the configuration is selected with `--config-name=train_diffusion_unet_timm_umi_workspace` and the dataset path is assigned through `task.dataset_path`. The published workflow therefore has a concrete data contract: a raw session is processed through SLAM, converted into `dataset.zarr.zip`, and consumed by the training configuration.

The README explicitly says that ORB_SLAM3 remains the most fragile part of the UMI pipeline and links to a fork optimized for the UMI workflow. That statement should shape an adoption decision. A team can have a working Docker environment and still lose data quality if SLAM fails. The example session reports 99% usable raw data and zero dropped demonstrations, but the repository does not present that value as a general success guarantee.

## Installing the Ubuntu environment

UMI is documented as tested only on Ubuntu 22.04. The installation section first points readers to Docker's official Ubuntu installation documentation and its Linux post-installation instructions. It then gives system-level dependencies. Install these before creating the Conda environment, because the later scripts depend on the graphics and patching packages named in the README.

```bash
sudo apt install -y libosmesa6-dev libgl1-mesa-glx libglfw3 patchelf
```

The README recommends Miniforge over standard Anaconda for faster installation. Create the repository environment from the included `conda_environment.yaml` file. The command creates an environment named `umi`.

```bash
mamba env create -f conda_environment.yaml
```

After creation, activate the environment before running the SLAM or training scripts. The README shows the prompt changing from the shell prompt to the `umi` environment, which is the visible indication that subsequent commands use that environment.

```bash
conda activate umi
```

This installation path is not platform-neutral. The repository provides Ubuntu instructions and says the project was only tested on Ubuntu 22.04. It does not document installation on macOS, Windows, or other Linux distributions in the README, so those environments should not be treated as supported by this repository.

## A first SLAM run with example data

The quickest first use is the example session supplied by the project. Download it from the Stanford-hosted data directory while the `umi` environment is active. The command is recursive and preserves the directory structure needed by the example invocation.

```bash
wget --recursive --no-parent --no-host-directories --cut-dirs=2 --relative --reject="index.html*" https://real.stanford.edu/umi/data/example_demo_session/
```

Run the SLAM pipeline against the downloaded `example_demo_session` directory. The README's example output includes camera discovery, camera assignment, usable-data percentage, and the number of dropped demonstrations. For this dataset, the documented output reports 99% usable raw data and zero dropped demonstrations. Treat that as an expected example result, not as a required result for a new capture session.

```bash
python run_slam_pipeline.py example_demo_session
```

If the success rate is low, the README directs users to revisit the data collection instruction and says to check whether it was followed carefully. This is the first practical diagnostic in the project. It also exposes the principal weakness of the workflow: SLAM quality is not independent of the collection procedure, camera setup, or scene conditions. UMI offers a processing pipeline, but it does not remove the need to inspect those upstream causes.

Once the output is acceptable, generate the training dataset with the replay-buffer script. The command writes a Zarr archive into the example session directory.

```bash
python scripts_slam_pipeline/07_generate_replay_buffer.py -o example_demo_session/dataset.zarr.zip example_demo_session
```

## Training Diffusion Policy on a UMI dataset

UMI includes a single-GPU training path through `train.py`. The README reports that this path was tested on an RTX3090 with 24GB of memory. The command selects the UMI workspace configuration and supplies the generated dataset path.

```bash
python train.py --config-name=train_diffusion_unet_timm_umi_workspace task.dataset_path=example_demo_session/dataset.zarr.zip
```

For multiple GPUs, the project uses `accelerate`. The number of processes is supplied as `<ngpus>`, which is a placeholder in the README rather than a literal value. Substitute the actual number of GPUs available in the deployment environment.

```bash
accelerate --num_processes <ngpus> train.py --config-name=train_diffusion_unet_timm_umi_workspace task.dataset_path=example_demo_session/dataset.zarr.zip
```

The repository also publishes a processed in-the-wild cup arrangement dataset and demonstrates multi-GPU training with it. The download command is:

```bash
wget https://real.stanford.edu/umi/data/zarr_datasets/cup_in_the_wild.zarr.zip
```

The cup arrangement example is a task-specific dataset, not a generic guarantee that every UMI user can reproduce every policy. It is associated with espresso cups with saucers distributed across 30 locations around Stanford, according to the README. The project also publishes a pre-trained checkpoint, but deployment still requires the matching hardware and configuration work described in the repository.

## Real-world robot, gripper, and camera integration

Real-world deployment requires physical configuration before policy evaluation. The README documents a UR5 with a teach pendant, a WSG50 gripper, a GoPro, an HDMI capture path, and a 3Dconnexion SpaceMouse. For the UR5, the operator must set the robot IP in `example/eval_robots_config.yaml`, configure a payload mass of 1.81 kg, enter the documented center of gravity, and switch the arm to remote control. The evaluation script sets the TCP automatically, according to the README.

The WSG50 requires its own network address in `eval_robots_config.yaml`. Its web interface settings also matter: the README says to disable the text-based interface, enable CRC, upload `umi/real_world/cmd_measure.lua`, enable the startup script, and select `/user/cmd_measure.lua`. These settings define how the gripper communicates and how the measurement command starts.

The GoPro setup includes installing the listed GoPro Labs firmware, setting the date and time, and using the README's HDMI output configuration. The SpaceMouse path depends on `libspnav` and `spnavd`. The README gives this installation and startup sequence:

```bash
sudo apt install libspnav-dev spacenavd
sudo systemctl start spacenavd
```

The deployment section says the system supports up to two arms and unlimited cameras, but the concrete example is a cup arrangement policy requiring a single arm and camera. The broader support statement should not be read as proof that every camera and robot pairing is supported without corresponding calibration and configuration.

## Limitations, alternatives, and maintenance

The clearest limitation is the SLAM dependency. The README calls ORB_SLAM3 the most fragile part of the UMI pipeline and points to a dedicated fork. Low SLAM success can force the operator back to the collection tutorial. This makes UMI a poor fit for a team that needs a dependable, hardware-independent data pipeline with no SLAM debugging capacity.

A practical alternative is a robot-specific teleoperation and imitation-learning stack in which demonstrations are captured directly through a robot interface, then converted into a task format for a policy trainer. The difference is the collection method. UMI emphasizes portable, in-the-wild demonstrations and reconstructs them through a SLAM pipeline before training. A direct teleoperation system avoids that reconstruction stage but requires the robot and control interface to be available at collection time. Another alternative is to build a custom dataset pipeline around the robot's native logs. That may reduce integration work when the task is tied to one arm and one environment, while giving up UMI's documented cross-location collection workflow.

The repository has no GitHub releases, so the README does not provide a release version or an upgrade path through tagged packages. The project is not archived, and the last push was on 2026-05-26. That fact does not by itself establish long-term compatibility or an ongoing support commitment. Upgrade cost is tied to rebuilding the Conda environment, checking Docker and hardware dependencies, and reviewing changes to the SLAM fork and deployment scripts. The repository uses the MIT license. That permits use and modification under the license terms, but the README does not address warranty, liability, hardware compatibility, or obligations specific to a deployment.

## Conclusion

Adopt UMI when the team has Ubuntu 22.04, the specified robot and camera hardware, and enough SLAM expertise to diagnose failed demonstrations. Do not choose it as a general-purpose robotics platform without accepting the hardware setup, calibration requirements, and documented SLAM limitation. Before committing to a deployment, verify camera assignment, dataset generation, and the exact robot and gripper configuration files used by the intended task.

## FAQ

### What operating system does Universal Manipulation Interface support?

The README says UMI was tested only on Ubuntu 22.04. It does not document support for macOS, Windows, or other Linux distributions.

### How do I create a training dataset with Universal Manipulation Interface?

First run the SLAM pipeline on the raw session, then use `python scripts_slam_pipeline/07_generate_replay_buffer.py -o example_demo_session/dataset.zarr.zip example_demo_session`. The resulting `dataset.zarr.zip` is passed to `train.py` through `task.dataset_path`.

### What is the main limitation of the Universal Manipulation Interface workflow?

The README identifies ORB_SLAM3 as the most fragile part of the pipeline. A low SLAM success rate can cause demonstrations to be unusable or dropped, so users may need to revisit the data collection procedure.

## Sources

- [Issues](https://github.com/real-stanford/universal_manipulation_interface/issues)
- [License: MIT](https://github.com/real-stanford/universal_manipulation_interface/blob/main/LICENSE)
- [Project website](https://umi-gripper.github.io/)
- [README](https://github.com/real-stanford/universal_manipulation_interface/blob/main/README.md)
- [real-stanford/universal_manipulation_interface on GitHub](https://github.com/real-stanford/universal_manipulation_interface)

---

Hysen Labs editorial analysis, written from the project's own repository and release notes. Cite the canonical page: https://hysenlabs.com/projects/real-stanford-universal-manipulation-interface