SimplerEnv real to sim evaluation for Google Robot and WidowX policies

Real to sim evaluation for Google Robot and WidowX

Evaluating a manipulation policy on hardware is slow. Every trial needs a robot, a reset, an object placed by hand, and a human decision about success, which is why the project describes real-world evaluation as operationally expensive and inefficient. SimplerEnv, presented at CoRL 2024, argues for physical simulators as a complement to that bench, and rebuilds the Google Robot and WidowX+Bridge setups so checkpoint selection and failure analysis can happen without the hardware.

Two evaluation modes sit at the centre. Visual matching matches real and sim visual appearances by overlaying real-world images onto simulation backgrounds and adjusting foreground object and robot textures in the sim. Variant aggregation builds variants of the same scene, changing backgrounds, lightings, distractors and table textures, then averages the results. One holds appearance still while a policy runs; the other moves the scene around the policy and reports a mean.

The reader this serves is a policy team with trained weights who needs a cheap proxy metric, not someone writing a new robot controller. The package exposes environments through a single factory call, and the shipped scripts cover RT-1, RT-1-X and Octo, so the intended workload is evaluation of existing policies rather than training from scratch.

SAPIEN and ManiSkill2 with a faster ManiSkill3 branch

Environment code sits on the SAPIEN physics simulator and the CPU based ManiSkill2 benchmark. That base choice sets the hardware floor for the whole project: SAPIEN requires a GPU to run, and TPU is not supported.

For batch work the Bridge dataset environments were also integrated into ManiSkill3, which offers GPU parallelization and, in the README's own number, runs 10-15x faster than the ManiSkill2 version. The catch is where that code lives. Instructions for the GPU parallelized environments point at a separate tree, the maniskill3 branch of the same repository, while the default branch is main. A plain clone of the default branch gives you the slower CPU path described everywhere else in the file, and reaching the ManiSkill3 environments plus their evaluation instructions means switching branches by hand.

Four install commands and one line that will not parse

The documented path is a conda environment on Python 3.10 or 3.11, a recursive clone, a numpy pin, then two editable installs. The heavier work also needs CUDA version >=11.8 and < 13 and an NVIDIA GPU, ideally RTX.

conda create -n simpler_env python=3.10 (3.10 or 3.11)
conda activate simpler_env
git clone https://github.com/simpler-env/SimplerEnv --recurse-submodules
pip install numpy==1.24.4

The `--recurse-submodules` flag matters because the ManiSkill2 real-to-sim environments arrive as a submodule rather than as plain files. In the README the clone directory is written as a placeholder, so replace it with wherever you cloned into, and install the benchmark package before the top level package:

cd {this_repo}/ManiSkill2_real2sim
pip install -e .
cd {this_repo}
pip install -e .

Both installs are editable, so the package you import as `simpler_env` is built from the clone on your own machine rather than downloaded as a wheel.

Details that decide whether the install actually runs

Three things in that path reward a closer look. The parenthesised note inside the conda command, `conda create -n simpler_env python=3.10 (3.10 or 3.11)`, is printed as a single line, and pasting it unchanged hands conda three extra arguments after the package specification. Read it as an either-or for the interpreter, not as part of the command.

Next is the numpy pin, `pip install numpy==1.24.4`, which the README ties to errors in IK inside pinocchio when numpy is 2.0 or newer. It is applied before the editable packages rather than recorded in `requirements_full_install.txt`, and `setup.py` lists no install_requires at all, so that editable install on its own resolves nothing and the pin survives until you install something else.

Packaging is thin, which suits research code. `setup.py` declares `name="simpler_env"` with `python_requires=">=3.10"` while the README narrows you to 3.10 or 3.11, and the formatting config in `pyproject.toml` pins `black` and `isort` to `line-length` 120 with `target-version` `py310`. Alongside sit `.flake8` and `.pre-commit-config.yaml`, so changes are linted on commit.

name="simpler_env"
version="0.0.1"
python_requires=">=3.10"

Three context lines from `setup.py` say the whole story about how this is packaged.

The minimal script samples actions instead of running a policy

The quickstart snippet is written for interactive python and deliberately incomplete. It builds a prepackaged visual matching environment, reads the language instruction, and stands in a random action where inference belongs:

env = simpler_env.make('google_robot_pick_coke_can')
obs, reset_info = env.reset()
instruction = env.get_language_instruction()
image = get_image_from_maniskill2_obs_dict(env, obs)
action = env.action_space.sample()
obs, reward, done, truncated, info = env.step(action)
episode_stats = info.get('episode_stats', {})
print("Episode stats", episode_stats)

The action layout is fixed by the environment: `action[:3]` is a delta xyz, `action[3:6]` a delta rotation in axis-angle representation, and `action[6:7]` the gripper, where open and close depend on the robot URDF. Run exactly as shipped and the loop rolls out a random policy, reporting whatever lands in `episode_stats` on the final step.

Long horizon tasks add a second mechanism. A new instruction arrives when the robot proceeds to the next subtask, and the loop prints it when it differs from the previous one. You can drive that transition yourself with `env.advance_to_next_subtask()`, or the environment autoadvances once `env._elapsed_steps` passes a threshold. Nothing in the snippet or the surrounding instructions defines a step budget or a success threshold, so the number you get back carries no scale until you add your own logging.

RT-1 and Octo pull in a heavier installation path

Scoring an actual policy is a separate job from creating an environment. RT-1 and Octo inference, and adding new robots or environments, are routed to a full installation section the minimal path does not cover, and `requirements_full_install.txt` at the top level belongs to that route.

Two entry points sit on that side. `example.ipynb` runs the same inference in Colab through the notebook badge, and `scripts/` holds the reproduction scripts for the Google Robot and WidowX results, with sweeps over object and robot poses and extra logging, carrying both the visual matching and variant aggregation setups. For interactive inspection with no policy attached, manual control lives in the submodule at `demo_manual_control_custom_envs.py`.

Upgrade cost is where the project is thinnest. `setup.py` declares the package as `version="0.0.1"`, the repository has no GitHub releases to pin against, and the last push was on 2026-09-30 under the MIT licence, so there is no tagged artifact to lock in and the README does not document rollback. Depending on it means tracking a moving main branch.

Where the simulated bench stops matching hardware

The strongest objection to leaning on this project is what visual matching deliberately gives up. It fixes appearance by pasting real images onto simulation backgrounds and editing foreground object and robot textures, so an environment that scores well has been tuned to look like the real setup. As documented, that procedure changes appearance only, and variant aggregation varies one simulator rather than substituting a different one.

The hardware floor is stated plainly. SAPIEN requires a GPU, TPU is listed as unsupported for that reason, and the README names 1080Ti and A100 as non-RTX cases where environments involving ray tracing will be slow. There is no documented CPU-only substitute for those environments, even though ManiSkill2 is described as a CPU based benchmark.

Two narrower limits matter for adoption. The provided policies are RT-1, RT-1-X and Octo, and anything else depends on the path for adding new policies. And the repository's primary language is recorded as Jupyter Notebook, which says where the working material lives: `example.ipynb` and the Colab badge, not a packaged library with its own release cadence. Add `python_requires=">=3.10"` and you inherit that floor in anything you build on top.

Simulated scores against the physical robot bench

The real alternative is the bench the simulation imitates: run the policy on the Google Robot or the WidowX+Bridge hardware. The difference in approach is the entire argument for SimplerEnv. Physical evaluation reports the behaviour of the real sensor stack, the real controller and the real object, and the README calls that route operationally expensive and inefficient, which is exactly why it frames simulators as a complement rather than a replacement.

What simulation buys is repetition with control. Visual matching holds the appearance fixed, variant aggregation sweeps backgrounds, lightings, distractors and table textures and averages the result, and the inference scripts in `scripts/` sweep over object and robot poses so a checkpoint is read across a pose distribution instead of a handful of hand-picked trials. A score from that setup is reproducible on another machine, which a physical trial is not.

What it costs is fidelity, and the boundary is worth naming: the number describes behaviour inside a matched simulator. Converting it into a physical success rate means running the comparison yourself, and nothing in the quickstart or the examples does that conversion for you.

Editorial conclusion

SimplerEnv fits a team that already holds RT-1, RT-1-X or Octo weights and needs a repeatable number before touching hardware. It is the wrong tool for anyone asking how a policy behaves under real sensor noise, since visual matching edits foreground object and robot textures rather than the scene physics. Check your GPU is RTX before starting, because the README names 1080Ti and A100 as slow ray tracing cases, and read the quickstart closely, since it samples random actions and fixes no step budget or success threshold.

Frequently asked questions

Official sources

1. Issues
2. License: MIT
3. Project website
4. README
5. simpler-env/SimplerEnv on GitHub