RoboLab: a simulation benchmark for generalist robot policies

The unit of evaluation is a task with a success predicate

RoboLab is described as a task-based evaluation benchmark for robot manipulation policies built on NVIDIA Isaac Lab, providing more than 100 manipulation tasks with automated success detection, a server-client policy architecture and multi-environment parallel evaluation, designed for reproducible, large-scale benchmarking of generalist robot policies in simulation. The design decision worth naming is that the unit of evaluation is a task rather than a scene. A task carries a language instruction and a success or failure condition expressed as composable predicates, so a policy is scored by whether it satisfied a stated goal rather than by whether a human watched it and agreed. That is the piece that makes the benchmark usable at scale, because human judgement does not scale and the alternative to a predicate is a rubric. The key features list leads with RoboLab-120, an initial set of 120 brand new benchmark tasks spanning pick-and-place, stacking, rearrangement, tool use and more, each with language instructions and automated success or failure detection via composable predicates. The second feature is the one that makes it a generalist benchmark rather than a suite for one machine, since tasks are not tied to a specific robot embodiment and you can plug in any robot compatible with Isaac Lab. The rest of the repository follows that structure, with separate asset libraries for objects, scenes and curated backgrounds, a tasks directory, a robots directory and a policies directory for clients.

Your policy runs as a server and RoboLab talks to it

The server-client architecture is the part that will shape how you integrate, and it is a deliberate choice rather than an implementation detail. Your model runs as a standalone server, and RoboLab connects to it via a lightweight inference client. The quick path to try it is Pi0.5 via OpenPI, and the README gives the invocation.

cd robolab
uv run python policies/pi0_family/run.py --policy pi05 --task BananaInBowlTask --num-envs 10

The reason to run a policy as a server rather than in-process is that generalist policies are large, often served from remote hardware, and sometimes shared between teams. A server boundary means the same policy can be pointed at by several evaluation runs, and the environment is not coupled to the model. The dependency list backs this up, with openai and requests for remote inference, pyzmq and msgpack with msgpack-numpy for the message layer, websockets for the streaming path, and json_numpy for the array serialisation that crosses that boundary. A protocol that has to carry observations out and actions back in has to be efficient, and the choice of a binary format with numpy support rather than JSON for the payload is the sort of thing that only matters once you are running a thousand parallel environments. The policies directory holds the clients, and the README points to a pi0_family README for the Pi0.5 example specifically.

Installing means choosing one of two Isaac stacks

The requirements are uv, a system ffmpeg used for video recording, and the IsaacSim and IsaacLab stack, which is selected at install time through a mutually exclusive extra. There are two: isaac50 for IsaacSim 5.0 with IsaacLab 2.2.0, which is the default, and isaac51 for IsaacSim 5.1 with IsaacLab 2.3.2.post1.

sudo apt install ffmpeg
git clone <repo_url>
cd robolab
uv venv --python 3.11
source .venv/bin/activate
uv sync --extra isaac50          # IsaacSim 5.0 / IsaacLab 2.2.0 (default)
# uv sync --extra isaac51        # IsaacSim 5.1 / IsaacLab 2.3.2.post1

The two stacks cannot coexist in one environment, and the README explains how to keep both available by installing each into its own virtual environment with the UV_PROJECT_ENVIRONMENT variable.

UV_PROJECT_ENVIRONMENT=.venv    uv sync --extra isaac50
UV_PROJECT_ENVIRONMENT=.venv-51 uv sync --extra isaac51

The pyproject file enforces this at the packaging level, with the two stacks declared as conflicting under the tool.uv configuration, and the comment in the dependency list explains that IsaacLab and IsaacSim are deliberately not pinned as core dependencies because they are version-specific stacks selected by the extras. The version pairing is exact rather than loose, with isaaclab at 2.2.0 and isaacsim with all and extscache extras at 5.0.0 in one extra, and 2.3.2.post1 and 5.1.0 in the other.

Verifying the install with a suite that runs a full episode

Installation is checked with a single command.

uv run pytest tests/

What that suite does is worth noting, because it tells you what the maintainers consider a working installation. It runs the install-verification suite end to end, checking that isaaclab is importable, that all task definitions are valid, that the environment factory is populated, and that one full episode runs. The suite auto-accepts the NVIDIA Omniverse EULA, so the run is fully headless with no prompts. There is a related note for the rest of the tooling, that when running other entry points such as policies/pi0_family/run.py for the first time you set export OMNI_KIT_ACCEPT_EULA=Y once, and that the acceptance is cached afterwards. There is also a note about not activating the virtual environment, in which case you prefix every python command with uv run. The rest of the test and examples infrastructure follows the same pattern, with a scripts directory for entry points, a docker directory for containerised runs, and a tests directory whose names suggest both unit level checks of task definitions and heavier end-to-end runs.

Running without a policy, and the limits of recorded episodes

Before you attach a model, you can check the environment itself. Three example scripts cover the cases you would otherwise need a policy for.

# Run an empty episode with random actions
python examples/run_empty.py --headless

# Playback recorded demonstration data
python examples/run_recorded.py --headless

# Toggle the gripper open/closed while holding the arm fixed (sanity-check
# the gripper action path; saves sensor + viewport video to
# output/run_gripper_toggle/<task>/)
python examples/run_gripper_toggle.py --task BananaInBowlTask --headless

The gripper toggle example is the clever one, because it sanity-checks the action path without a policy at all, holding the arm fixed while opening and closing the gripper, and it writes sensor and viewport video so you can confirm the scene looks right. The replay script is more consequential, and the README is unusually candid about its limits. run_recorded.py restores the recorded initial state and replays the recorded actions open-loop, and by default replays with the exact environment configuration saved next to the recording in env_cfg.json. The important sentence is that the recorded outcome is not invariant across simulator versions, because contact dynamics evolve between IsaacSim and IsaacLab releases, and faithful reproduction requires recording and replaying with a single environment. There is a dedicated replay document covering replaying your own recordings, the --env-config option and a --validate-states option. The examples directory lists the rest, with run_abs_ik_demo.py, run_rel_ik_demo.py and run_kinova_jointpos.py pointing at different embodiments and control modes.

Generating scenes and tasks from natural language

The most unusual feature in the key features list is the AI-enabled workflow, which generates new scenes and tasks in minutes using natural language with the /robolab-scenegen and /robolab-taskgen Claude Code skills. Those names are literal, since the skills directory at the top level of the repository holds them, and they are slash commands available inside a Claude Code session. The point is throughput. Writing a manipulation task by hand means writing a scene, an object set, a language instruction and a set of success predicates, and doing that for a hundred tasks is a project. If a natural language description can produce a candidate task that a human then reviews and corrects, the marginal cost of a hundred tasks drops considerably, which is what a benchmark needs in order to have breadth. The risks are the obvious ones, that generated predicates may not measure what the instruction says, and that a benchmark whose tasks are machine-generated is measuring something subtly different from a curated one, so the generated tasks and the hand-written ones should not be mixed in one leaderboard without saying so. The repository also has a skills directory alongside the task, object, scene and robot directories, and the documentation set covers the ecosystem, the dashboard, debugging, replay and the task libraries.

Reading the results, and where the benchmark's limits are

The dashboard is described as a self-contained web dashboard for browsing scenes and tasks, replaying episode videos and comparing results across experiments, and the README points to the output written to your local folder. A benchmark is only as good as the ability to inspect the runs behind a number, and a dashboard with episode video is a better diagnostic than a leaderboard table alone, because a policy that scores badly on a task often fails in one identifiable way that the video makes obvious. The leaderboard lives on the project site. The limits are the ones the README is honest about. Everything happens in simulation, so the score measures behaviour in a simulator whose contact model changes between releases, and the recorded-outcome caveat means a number from one IsaacSim version is not strictly comparable with a number from another. Parallel evaluation also changes what you are measuring, since a task that depends on precise placement can behave differently when it is one of many environments stepping in parallel, and per-environment termination and vectorized conditionals are the mechanisms that make that work. The remaining limit is scope, since the task set is 120 tasks plus ecosystem contributions, with LADDER adding 44 long-horizon tasks in a 145-task benchmark spanning three simulation suites, and RoboVoLo extending the set further. A benchmark that is still being extended is a moving target, which is the right thing for a research field in motion and the wrong thing if you need a fixed yardstick for a release decision.

Editorial conclusion

Adopt RoboLab if you are evaluating a robot manipulation policy in simulation and need a repeatable set of tasks with automated success detection rather than a hand-built environment. Skip it if your interest is real hardware, since the whole benchmark is simulated, or if you cannot commit the GPU and disk this stack needs. Verify first which IsaacSim and IsaacLab combination you install, since the two extras cannot coexist in one environment, that you set the EULA acceptance variable before running a policy entry point outside the test suite, and that the leaderboard you are comparing against used the same simulator version, because recorded outcomes are not invariant across IsaacSim releases.

Frequently asked questions

Official sources

1. License: Apache-2.0
2. NVlabs/RoboLab on GitHub
3. Project website
4. README
5. Releases