vla-eval measures VLA models against many simulation benchmarks the same way
One framework to evaluate any VLA model on any robot simulation benchmark.
At a glance
- What is it?
- The AllenAI vla-evaluation-harness is a framework for scoring vision-language-action models on simulation benchmarks, with benchmark and model definitions kept as configuration files and a status flag that distinguishes reproduced results from merely integrated ones. It ships on PyPI as vla-eval and is versioned under Apache-2.0.
- Who is it for?
- vla-eval earns a place in a research pipeline that has to compare models rather than run one, because putting the benchmark definition, the model server and the scoring in one configuration driven framework is what makes a comparison meaningful across projects.
- Can I use it commercially?
- Yes. Apache-2.0 is a permissive licence: you can use, modify and sell software built on it, as long as you keep its copyright and licence notices.
- Is it still maintained?
- Yes. The repository received new commits within the last day.
- What is it written in?
- Mainly Python, according to GitHub's language statistics.
Answers come from the project's GitHub data, last synced on October 23, 2026, and from our analysis. They are not legal advice.
Editorial analysis
One scoring framework across many simulators and many model families
The vla-evaluation-harness states its purpose in a single line: one framework to evaluate any VLA model on any robot simulation benchmark. The ambition is visible in the coverage tables at the top of the README, which pair a long list of benchmarks with a long list of models. The benchmarks include LIBERO, SimplerEnv, CALVIN, LIBERO-Pro, LIBERO-Plus, LIBERO-Mem, ManiSkill2, RoboCasa and RoboCasa365, VLABench, MIKASA-Robo, RoboTwin, RLBench, RoboCerebra, BEHAVIOR-1K for 2025, Kinetix, RoboMME, MolmoSpaces-Bench, DuoBench, RoboDojo and FurnitureBench. The model side covers official configurations for OpenVLA, GR00T N1.6, OFT, X-VLA, CogACT, RTC, VLANeXt and MolmoBot, plus entries for models from other projects such as dexbotic, starVLA and LeRobot. The point of a shared harness is that the numbers become comparable, since two papers reporting success rates on the same task set through different code paths are not obviously comparable.
The status flag separates reproduced numbers from integrated ones
The detail that makes the coverage tables usable is the legend under them, which marks each entry with one of three symbols. A tick means reproduced, and the legend links that state to a reproductions document in the docs directory. A diamond means integrated but awaiting a first reproduction. A plain dot means planned. Read strictly, that means a good part of the table is aspirational, and the honest reading of the tables is that only the ticked entries have numbers someone else has confirmed. The most recent news entry makes the same point in numbers: four checkpoints are reported as reproducing their published LIBERO scores, listed at 100 percent, 99 percent, 97 percent and 96 percent, through the LeRobot bridge pinned at version v0.6.0. The gap between ticked and diamond entries is the single most useful signal in the README for anyone deciding whether to trust a leaderboard row, and it is why the project also keeps a leaderboard directory and a reproductions document set rather than a single table of results.
Benchmarks and models are configuration, not code
Everything the harness runs is described by files in the configs directory, with a subdirectory per benchmark and another for model servers. The benchmark entries in the table point at paths such as configs/benchmarks/libero, configs/benchmarks/calvin, configs/benchmarks/robocasa and configs/benchmarks/robomme, and the model entries point at paths such as configs/model_servers/openvla, configs/model_servers/groot and configs/model_servers/lerobot. The package dependencies explain how those files are consumed: jsonargparse with signature support provides class driven argument parsing and yaml, omegaconf handles configuration composition, and msgpack carries messages between processes. A configuration driven design is the right call for a project whose whole purpose is to run the same code against twenty different simulators, because adding a benchmark should mean writing a description of the task set, not a new adapter class. It also means the behaviour of a benchmark is defined by files you can read and diff, which is what makes a reproduction possible at all.
Model servers run out of process and talk over websockets
The model side is separated from the evaluation side, and the dependency list shows how they connect. The websockets library is pinned at 13.0 or newer specifically because that version adds the asyncio server implementation that the serve module imports, and msgpack handles the message encoding. A model server configuration therefore describes a process that loads a checkpoint, accepts action requests over a websocket, and returns actions, while the evaluation loop lives elsewhere and drives the simulator. That separation is what lets a model that needs a GPU in one container be evaluated against a simulator that needs a GPU in another, and it is also why the project leans on containers and on Charliecloud as a runtime. The cost is that every run now has a network hop in it, so a flaky websocket is a new failure mode for a harness whose job is to produce numbers people can compare.
Installable from PyPI, with a Makefile that exposes the whole workflow
The distribution story is conventional and easy to check. The package is built with hatchling and hatch-vcs, so the version comes from the version control system, and it is published to PyPI under the name vla-eval, which the README says is all a new example project needs. The console entry point is declared in the project scripts section, and the package targets Python 3.8 or newer, with a lazyregistry dependency present specifically to keep 3.8 support. The development workflow is exposed as Makefile targets, and the smoke target is the one worth knowing about, because it runs the test command of the tool itself against every configured environment.
lint:
uv run ruff check --fix
uv run ruff format
check:
uv run ruff check
uv run ruff format --check
uv run ty check
test:
uv run --group leaderboard pytest
smoke:
uv run vla-eval test --all
smoke-list:
uv run vla-eval test --listThe check target combines linting, a formatting check and a type check, and the test target uses the leaderboard dependency group, which the manifest comments explain is kept separate so that CI can sync the leaderboard tooling without pulling in the runtime dependencies of the evaluation package itself.
Recent releases added runtimes, recording and reproducible images
The release history shows what the project has been working on and it is a good guide to its current shape. Version 0.6.0 in September 2026 added a Python API for training time evaluation, the Charliecloud runtime and a docker build entry point. Version 0.5.0 added RoboDojo, the split between RoboCasa and RoboCasa365, and a CPU render backend, which matters for anyone without a GPU. Version 0.4.0 turned recording on by default, added pinned reproducible Docker rebuilds, added DuoBench and added the LeRobot bridge. Version 0.8.0 was published on 2026-09-30, with 0.7.0 on 2026-09-19 and 0.6.0 on 2026-09-17. Three minor releases in two weeks is a fast cadence, and the September 2026 Push-T example is a good illustration of the intended use: a self-contained project that trains on its own environment and evaluates it through the harness during training, with no clone of the repository required.
Editorial conclusion
vla-eval earns a place in a research pipeline that has to compare models rather than run one, because putting the benchmark definition, the model server and the scoring in one configuration driven framework is what makes a comparison meaningful across projects. It is the wrong tool for a single model on a single simulator, where the overhead of the harness buys nothing, and the wrong tool if you need a stable interface, because the version line is moving quickly with monthly minor releases. Verify three things before you build on it: which benchmarks and models carry the reproduced status rather than the merely integrated status, since the distinction is the difference between a number you can cite and one you have to check yourself, whether the environment you run in matches the Charliecloud and container workflows the project uses, and whether the model you care about has a server configuration that suits your checkpoint format. The package is on PyPI as vla-eval, the licence is Apache-2.0, and the last push was on 2026-10-15.
Frequently asked questions
What does the vla-evaluation-harness actually evaluate?
It is a framework for evaluating vision-language-action models on robot simulation benchmarks, with benchmark definitions for LIBERO, SimplerEnv, CALVIN, RoboCasa, RoboMME and many others, and model server configurations for OpenVLA, GR00T, OFT, X-VLA and LeRobot served models.
How do I know which results in the harness are trustworthy?
The coverage table marks each entry with a symbol: a tick for reproduced, which links to a document in the reproductions directory, a diamond for integrated but awaiting a first reproduction, and a dot for planned. Only the ticked entries have confirmed numbers.
How do I install and smoke test vla-eval?
The package is on PyPI as vla-eval, so pip install vla-eval is enough for a project that only needs the Python API. The Makefile in the repository exposes a smoke target that runs the tool's own test command across environments, and a smoke-list target that lists them.
Does vla-eval need a GPU to run the simulators?
Not necessarily. The 0.5.0 release notes mention the addition of a CPU render backend, and the framework separates model serving from evaluation so the two can run in different environments. The project also supports container images published to ghcr.io and a Charliecloud runtime.
Official sources
Add this badge to your README
If you maintain this project, the badge below links readers to this analysis and shows its maintenance status from the daily GitHub snapshot. Paste the markdown into your README; add ?metric=license or ?metric=stars to the image URL for a different field.
[](https://hysenlabs.com/projects/allenai-vla-evaluation-harness)Was this useful?