Inspect Robots: an eval harness for VLA policies and LLM-driven arms

The gap Inspect Robots fills between benchmark scripts and policy repos

Most robotics evaluation code lives inside the repository of the policy it evaluates. A VLA project ships a rollout script wired to one arm and one task set; a benchmark ships its own harness wired to one simulator. Comparing two policies then means porting one of them, and the port is usually where the numbers stop being comparable. Inspect Robots takes the opposite position: the benchmark is defined once, and the policy and the embodiment are both swappable behind it. The README frames the project as "that for robotics" in reference to Inspect AI, the UK AI Safety Institute's evaluation framework for language models, and the borrowing is structural rather than cosmetic. The audience is engineers who need to run more than one policy against more than one rig and keep the results auditable. The repository topics name the same scope: benchmark, embodied-ai, evaluation, physical-ai, vision-language-action. A team that only ever evaluates a single VLA on a single arm gains little from the abstraction; a team comparing an LLM agent against a VLA on the same hardware is the case the design assumes.

Policy, embodiment and grader as separate plugin slots

The architecture visible in the README is a pipeline with three swappable parts. The policy is whatever produces actions: a VLA served over HTTP, an LLM agent driving the rig through tool calls, or an in-process policy such as the mock `scripted`. The embodiment is the rig, supplied by an installable plugin that also declares its behaviour toggles and numeric settings. The grader scores the run. What comes out is an eval log containing the grader scores, the LLM transcript where relevant, and the full configuration, plus a Rerun stream of cameras, proprioception and actions saved as a `.rrd` file beside the log. That last part is the mechanism worth understanding: the same data the policy consumed is recorded, so a disputed result can be replayed rather than argued about. The agent path adds a constraint of its own. According to the README, a frontier LLM drives the rig through tool calls at one approver-checked motion chunk per call, which means the control rate is bounded by inference latency and every motion passes an approval step. That is a deliberate safety choice and a throughput cost at the same time.

Installing the harness and writing the config file

Installation is a virtual environment plus a pip install, with an extra for the viewer. The README gives `uv venv && uv pip install "inspect-robots[rerun]"` for the full install and `uv venv && uv pip install inspect-robots` for the numpy-only core. It also warns against `uv run inspect-robots` inside an existing uv project, because that re-syncs to the lockfile and silently uninstalls what `uv pip install` just added. Configuration is a wizard. You install the plugin for your rig, for example `uv pip install inspect-robots-yam` for the MolmoAct2 policy and yam arms, then run `inspect-robots setup`. The wizard picks defaults, finds cameras, and asks about behaviour toggles and numeric settings declared by the embodiment plugin, such as yam's `auto_start`, before writing `~/.config/inspect-robots/config.ini`. On a rig without a plugin you type component names at the prompts or write the file by hand. Running an eval is then a single command, `inspect-robots "place the fork on the plate"`, with CLI flags overriding any default: `--no-rerun-save`, `--no-rerun`, `--no-store-frames`, `--max-steps 300`. The LLM path needs `uv pip install inspect-robots-agent`, an `ANTHROPIC_API_KEY` in a `.env` file the CLI loads automatically, and `--policy agent` with policy arguments such as `-P model=anthropic/claude-fable-5 -P effort=low`. Results are read back with `inspect-robots inspect LOG.json --transcript` or `inspect-robots view LOG.json`, and `inspect-robots view logs/` renders a browsable index once several runs exist.

The MolmoAct2 policy is a client, and nothing moves until the server does

The most concrete limitation in the README concerns the shipped example rather than the framework. The `molmoact2` policy is described as only a client: nothing moves until the MolmoAct2 server is listening, and the server does not start itself or survive a reboot. Setup is manual and lives in the yam plugin's README, with the command `python examples/yam/host_server_yam.py --host 0.0.0.0 --port 8202` run from the MolmoAct2 repository on the GPU machine, kept alive in something like tmux, and a readiness check of `curl http://127.0.0.1:8202/act` where a 200 means the server is ready. In-process policies such as `agent` or the mock `scripted` need no server. The practical consequence is that a benchmark run depends on a process outside the eval framework, and the eval framework has no supervisor for it. There is also a scope boundary worth stating plainly: the wizard finds cameras and writes settings, but the README does not describe calibration, safety limits, or a rollback path for a bad configuration, and the framework is marked alpha with an explicit warning that the API may change between releases. Treat the harness as the layer that records and scores, not the layer that makes a rig safe to operate. The voice plugin carries its own boundary: `--voice` keeps the microphone open for the run and delivers spoken remarks to the policy at its next inference, transcribed locally with no keys and no network, but silence sends nothing and voice is feedback-only, since ending an episode and recording verdicts stay on the keyboard.

Where Inspect Robots is the wrong tool

Three cases argue against it. First, if your evaluation is a single policy on a single arm and you already have a rollout script that works, the plugin boundary is overhead: you would install a rig plugin, run a wizard, and learn a config format to reproduce a script you already trust. Second, if your policy has no HTTP endpoint and no plugin, the framework gives you nothing to plug into; the README's own examples are a served VLA, an agent policy, and a mock, and an in-process policy is the only other shape it names. Third, if you need API stability for a product built on top of the eval layer, the alpha warning and the version cadence work against you. The releases listed run from v0.57.0 on 2026-08-19 to v0.58.0 on 2026-09-02, which is a fast minor-version cadence, and the README's advice is to pin a version before depending on it. That is a reasonable stance for a young framework and an awkward one for anyone who wants the eval schema frozen. The recent activity also means the last push was on 2026-09-02, so the project is moving; it does not mean the interfaces have settled.

Compared with a bespoke rollout script, and with sim-only harnesses

The realistic alternative is the one most labs already have: a repository-local rollout script that loads a checkpoint, steps the environment, and prints a success rate. The difference is not sophistication but where the boundary sits. A bespoke script couples the policy to the rig, so swapping either means editing the script, and the record of what happened is whatever the script chose to print. Inspect Robots puts the boundary at the plugin and writes a structured log with the full config, grader scores, the transcript, and a replayable Rerun stream. The second alternative is a simulator-only benchmark harness, which gives reproducible scenes and cheap parallel rollouts but cannot tell you how a policy behaves on your arm, with your cameras, under your latency. Inspect Robots explicitly spans both sides, since the embodiment can be a real arm, a humanoid, or a simulator, and the policy can be a VLA or an LLM agent. The trade is that a real-rig run inherits every operational dependency of the rig, including the manually started policy server, while a sim-only harness does not. If your question is about policy behaviour in a controlled scene, a sim-only harness answers it with less machinery; if your question is whether a policy survives contact with your hardware, the real-rig path is the point.

Licence, release cadence and the cost of keeping a rig plugin current

Inspect Robots is MIT licensed, which permits commercial use, modification and redistribution provided the copyright notice and permission notice are retained. That matters here because the plugin boundary invites forking: if a rig plugin stops tracking the core, an MIT core means you can maintain the plugin yourself rather than wait. The README's pin-a-version advice is the other half of that trade. The releases shown are v0.57.0 on 2026-08-19, v0.57.1 on 2026-08-20, and v0.58.0 on 2026-09-02, with the last push on 2026-09-02, so upgrades arrive often enough that an unpinned dependency will move under you. The upgrade cost is not only the core package. Each rig needs its own plugin, and the plugin declares the behaviour toggles and numeric settings the wizard prompts for, so a core change that touches the plugin interface propagates to every rig you support. The README also notes that the yam plugin's setup instructions live in a separate repository, which means version compatibility spans at least two packages. Budget for that: pin `inspect-robots` and each rig plugin, and test an upgrade on one rig before rolling it across a fleet. Nothing here is legal advice; if you redistribute a modified plugin, read the MIT text in the LICENSE file yourself.

Editorial conclusion

Adopt Inspect Robots if you already have a rig plugin, a policy that can speak the client/server or in-process interface, and a need for replayable eval logs rather than one-off scripts; the MIT licence and the pin-a-version warning in the README leave you free to fork the harness if a plugin lags. Do not adopt it if you need a stable API today or if your policy has no plugin and no HTTP endpoint, because the framework is alpha and the README states the API may change between releases. Before committing, verify that your rig has a published plugin, that `inspect-robots setup` writes a config your hardware accepts, and that your policy server answers `curl http://127.0.0.1:8202/act` with a 200.

Official sources

1. License: MIT
2. Project website
3. README
4. Releases
5. robocurve/inspect-robots on GitHub