# EgoVerse packages egocentric robot data from many vendors into one corpus

> EgoVerse is the data processing, training and evaluation code for a dataset of egocentric human demonstrations and robot episodes collected from vendors around the world, normalised into zarr episodes with a mandatory camera intrinsics record and a collapsed set of human and robot embodiments. It suits a robotics researcher training a policy across hardware, and a recent schema change invalidates cached episodes.

**GaTech-RL2/EgoVerse** — EgoVerse: Egocentric Data for Robot Learning from Around the World

- Repository: https://github.com/GaTech-RL2/EgoVerse
- Stars: 573 · Forks: 54
- Language: Python
- License: MIT
- Published: 2026-10-22 · Updated: 2026-10-22 · Language: en
- Canonical page: https://hysenlabs.com/projects/gatech-rl2-egoverse

## One corpus, many vendors, and a schema that moved under you

EgoVerse holds the data processing, training and evaluation code for a dataset of egocentric data collected for robot learning from around the world, and the changelog at the top of the README is the most informative document in the repository. The entry dated 07/08/2026 is titled Mandatory Camera Intrinsics and Human Embodiment Collapse, and it is a hard break rather than a soft change. Camera intrinsics are now mandatory in every episode's zarr.json, stored as a dict mapping a camera key to a 3 by 4 K matrix, with a single camera meaning one entry, and ZarrWriter.create_and_write raises if it is missing or is not a non-empty dict. Embodiments have been collapsed, with all human demonstration data now under a single human_ prefix, covering human_right_arm, human_left_arm and human_bimanual with ids 1 to 3, and the robot Eva under eva_ prefixes with ids 4 to 6. The vendor labels such as aria_, mecka_, scale_ and lightwheel_ are removed at the embodiment level, with the data source surviving only in the SQL lab field.

## Why the collapse broke local caches

The consequence is spelled out with unusual candour, and it is the part to read before you touch anything. The embodiment string is stored inside each episode's zarr.json, and the reader looks it up with no alias fallback. A locally cached episode written before the collapse still says aria_bimanual, scale_bimanual or mecka_bimanual, and loading it now hard-crashes with a KeyError. The instruction is to delete any local cache and re-pull the reprocessed episodes, because they carry the human_ prefix and the mandatory intrinsics. Data producers are asked to re-process and re-upload so that zarr.json includes intrinsics, with a pointer to CONTRIBUTING_DATA.md, and the note that Mecka and Scale will be asked to add intrinsics to their exports. The earlier changelog entries follow the same pattern of churn, with the Scale dataset fully reprocessed on 05/03/2026 so that all active Scale episodes use newly generated episode hashes, Zarr paths and preview MP4 paths, and old Scale hashes should be treated as stale.

## The orientation fix that makes human and robot data align

The other half of the same changelog entry is a geometry fix, and it is the part that makes joint training possible rather than merely tidy. Aria end-effector pose and wrist-pose orientations are now correct and consistent across both hands, with the left and right hand sharing the same canonical T_ROT_CAM convention, aligned with the robot Eva tool frame, so human and robot end-effector orientations line up for joint training and visualization. Anyone with processed Aria data needs to re-process it to pick up the corrected orientations. This is the kind of change that is easy to miss and expensive to discover, because a mismatch here produces a model that trains without error and behaves badly, since the human demonstrations and the robot episodes disagree about what direction a hand is pointing. The canonical convention is what makes the two comparable at all, and the changelog says so directly.

## The code layout is a training script, a config tree and two converters

The Structure section names the pieces. egomimic/trainHydra.py is the main training script, powered by PyTorch Lightning and Hydra with DDP supported. egomimic/hydra_configs holds the train configs for each algorithm. egomimic/algo holds the algorithm code, described as EgoMimic, which is HPT based, and Pi, and the section is truncated in the README so the full list of algorithms is not visible here. The two conversion scripts are what produce the corpus from raw recordings. egomimic/scripts/aloha_process processes raw ALOHA hdf5 into zarr and lerobot format, and egomimic/scripts/aria_process processes Aria vrs into zarr and lerobot format. The conversion scripts sit in the same tree as the training code, which is worth noting, because it means the data format and the model that consumes it are developed together rather than handed across a boundary. The repository root also holds the Dockerfile, the requirements files, pull_models.sh, run_eva_docker.sh, run_inspector.sh, a training.md, a pi05.md and an AGENTS.md, plus a convention.png and a mano_keypoints.png at the top level.

## Installing with uv, and the dependency conflict in plain sight

The installation section recommends uv and shows the whole sequence:

```bash
git clone git@github.com:GaTech-RL2/EgoVerse.git
cd EgoVerse
uv venv emimic --python 3.11
source emimic/bin/activate
uv pip install -e .
uv run pre-commit install
```

The pyproject.toml, locked in uv.lock, is the dependency spec, and three things live outside it. requirements.txt is used only by the Eva robot Dockerfile, requirements-ray.txt is for the Ray data-processing image, and scale_sensor_fusion_io is needed only by the Scale converter under external/scale/scripts. The Pi0.5 policy needs the openpi fork from the submodule, installed without its own dependencies because its lerobot pin needs av 14.2 or newer, which conflicts with the av 12.0.0 the project pins, and then its transformers patch is copied into the environment:

```bash
git submodule update --init external/openpi
uv pip install --no-deps -e external/openpi
cp -r external/openpi/src/openpi/models_pytorch/transformers_replace/* emimic/lib/python3.11/site-packages/transformers/
```

The copy has to be re-run after any uv sync or transformers reinstall, which silently undoes the patch. A conda route is also given, cloning with --recursive, creating an environment with python 3.11 and installing with pip install -e .

## Configuration through environment variables, and where the data lives

Every cluster-specific default in the configs can be overridden without editing YAML, and the table lists the variables and what they override. EGOVERSE_DATASET_DIR overrides paths.dataset_dir, the local zarr episode folder, defaulting to a lab path. EGOVERSE_LOG_DIR is the root of every run directory, defaulting to $EGOVERSE_ROOT/logs. EGOVERSE_CACHE_DIR covers paths.cache_dir, where norm stats are cached by content under a norm_stats/ directory, defaulting to the log directory's cache. EGOVERSE_ROOT is paths.root_dir and is the directory you launch from. EGOVERSE_PI05_WEIGHTS points at the pi0.5 base weights directory used by the pytorch_weight_path config. And WANDB_ENTITY and WANDB_PROJECT override the logger's wandb entity and project, with defaults of rl2-group and zarr_test. The data itself comes from cloud storage, with an AWS setup script that lets your current environment download from cloudflare, and a web viewer at the partners.mecka.ai site for browsing the whole dataset, plus a notebook for programmatic visualisation.

## A dependency set that tells you the scope

The pyproject dependency list is long and it describes the research stack precisely. The learning side has torch 2.7.1, torchvision 0.22.1, lightning for the training loop, positional-encodings with the pytorch extra, einops, timm, transformers 4.53.2, geomloss, tslearn and arm_pytorch_utilities, and the PyTorch 3D stack through mujoco 3.4.0, dm-control 1.0.8, mink 0.0.13 and pytorch-kinematics, which is the simulation and whole-body control side. The data side has zarr 3.1.5 and datasets 4.0.0 for the episode format, av 12.0.0 for video, decord and simplejpeg for decoding, projectaria-tools 2.0.0 with the all extra for the Aria sensors, h5py for the raw ALOHA hdf5 files, and s5cmd for object storage transfers, with boto3, cloudpathlib, sqlalchemy and psycopg for the catalogue side. The requirements.txt for the Eva robot image pins torch 2.6.0 and transformers 4.57.3 instead, which is a small divergence worth noticing, and the Dockerfile is built on ros:humble-ros-base with git-lfs, can-utils, udev and the ARM robot development libraries, so the container is set up to talk to real robot hardware rather than only to data.

## Conclusion

Adopt EgoVerse if you are training a generalist robot policy and want human demonstrations and robot episodes from several vendors in one corpus with shared protocol and configuration, and you can work with zarr episodes on object storage. Do not adopt it if you expect a stable dataset contract, because the changelog shows the embodiment scheme and the intrinsics requirement changed in mid 2026 with no alias fallback, so cached episodes break. Verify first that you re-download episodes after the collapse, that each episode's zarr.json carries the intrinsics dict, and that your environment matches the pinned torch, av and transformers versions before you start a training run.

## FAQ

### What is EgoVerse and what does the repository contain?

It contains the data processing, training and evaluation code for EgoVerse, a dataset of egocentric human demonstrations and robot episodes for robot learning collected from vendors around the world. The main training script is egomimic/trainHydra.py, powered by PyTorch Lightning and Hydra with DDP supported, with configs in egomimic/hydra_configs and algorithm code in egomimic/algo.

### What changed in the 07/08/2026 EgoVerse update?

Camera intrinsics became mandatory in every episode's zarr.json as a dict of camera key to 3 by 4 K matrix, and ZarrWriter.create_and_write raises if it is missing. Embodiments were collapsed so human data is human_ with ids 1 to 3 and the robot Eva is eva_ with ids 4 to 6, and the vendor labels are removed at the embodiment level with the source surviving only in the SQL lab field.

### Why does loading an old EgoVerse episode fail?

The embodiment string is stored in each episode's zarr.json and the reader looks it up with no alias fallback, so a cached episode written before the collapse still says aria_bimanual and loading it hard-crashes with a KeyError. The fix is to delete any local cache and re-pull the reprocessed episodes, which carry the human_ prefix and the mandatory intrinsics.

### How do I install EgoVerse and the Pi0.5 policy?

With uv, clone the repository, create a Python 3.11 environment with uv venv emimic --python 3.11, activate it, and run uv pip install -e . followed by uv run pre-commit install. The Pi0.5 policy needs the openpi submodule installed with uv pip install --no-deps -e external/openpi, because its lerobot pin needs av 14.2 or newer which conflicts with the pinned av 12.0.0, and its transformers patch has to be copied into site-packages after every sync.

## Sources

- [GaTech-RL2/EgoVerse on GitHub](https://github.com/GaTech-RL2/EgoVerse)
- [Issues](https://github.com/GaTech-RL2/EgoVerse/issues)
- [License: MIT](https://github.com/GaTech-RL2/EgoVerse/blob/main/LICENSE)
- [README](https://github.com/GaTech-RL2/EgoVerse/blob/main/README.md)

---

Hysen Labs editorial analysis, written from the project's own repository and release notes. Cite the canonical page: https://hysenlabs.com/projects/gatech-rl2-egoverse