ClawProBench: a live-runtime harness for grading OpenClaw agents
ClawProBench is a live-first benchmark harness for evaluating LLM agents in the OpenClaw runtime with deterministic grading and repeated-trial reliability.
At a glance
- What is it?
- ClawProBench runs LLM agents inside the OpenClaw runtime and grades their traces with deterministic checkers. It is a Rust-backed harness with a Python entry point, 102 active scenarios, and a leaderboard that its own maintainers admit vendors can optimize against.
- Who is it for?
- Adopt ClawProBench if you need to compare models or agent configurations inside the OpenClaw runtime itself and you want graded traces rather than a static question set. Do not adopt it if your agents do not run in OpenClaw, or if you need a neutral public ranking: the maintainers state in the v1.1.4 notes that a fully open benchmark cannot fully avoid vendors optimizing specifically for it.
- Can I use it commercially?
- Yes. Apache-2.0 is a permissive licence: you can use, modify and sell software built on it, as long as you keep its copyright and licence notices.
- Is it still maintained?
- Yes. The repository last received commits 22 days ago.
- What is it written in?
- Mainly Rust, according to GitHub's language statistics.
Answers come from the project's GitHub data, last synced on September 15, 2026, and from our analysis. They are not legal advice.
DEEP OPEN-SOURCE ANALYSIS
The gap ClawProBench is aimed at
Most agent benchmarks hand a model a task in a sandbox that the benchmark authors control. ClawProBench takes the opposite position: it evaluates models inside the OpenClaw runtime, which the README calls a "live-first" approach. The claim is that capability measured in a purpose-built sandbox does not transfer cleanly to the runtime an agent actually operates in, because tool availability, permission boundaries, and execution semantics differ. Whether that gap is large is an empirical question, and the harness is the instrument the project built to answer it.
The audience is narrow and specific. You are building or operating agents on OpenClaw and you want to know which model, or which configuration, completes real tasks more reliably. You are not looking for a general-purpose LLM scoreboard. The project also publishes a leaderboard at suyoumo.github.io/bench, which is aimed at a different reader: someone comparing frontier models who will never run the harness. Those two audiences want different things from the same repository, and the README serves both.
Profiles, scenarios, and what the numbers mean
The unit of work is a scenario. The README reports 102 active scenarios and 164 total catalog scenarios, with 62 of those incubating. The default ranking path is the core profile, which the badges describe as 26 scenarios. Four broader profiles exist: intelligence, coverage, native, and full. This is a deliberate split between a cheap, stable comparison set and a larger set that changes more often.
The distinction between active, catalog, and incubating is the part worth understanding before you quote a number. Catalog scenarios exist in the repository but are not counted as benchmark-active. Incubating scenarios are the 62 that have not graduated. If you report "164 scenarios" as the benchmark size, you are overstating it: the project's own framing treats 102 as the active count. The README points to two commands for checking this on your own checkout rather than trusting the badge.
How grading actually works
Grading is deterministic, and the release notes give the clearest evidence of what that means in practice. The v2.0.3 entry mentions hardening multiple custom checkers with "deterministic order-insensitive matchers." That single phrase tells you a lot about the design. Checkers compare structured outputs, and the comparison has to tolerate variation in the order in which an agent produces equivalent results, while still being reproducible. An LLM-as-judge design would not need order-insensitive matchers, because a judge would not care about ordering in the first place.
Custom checkers are loadable, per the v1.1.3 note about "custom-check loading" fixes. So the grading surface is extensible, and scenarios can carry their own check logic. The trade-off is that determinism is only as good as the checker. A checker that is order-insensitive but wrong about what counts as equivalent will produce consistent, reproducible, incorrect scores. The README does not describe a validation suite for checkers, and the material does not say how a contributed checker is reviewed.
The harness is written in Rust, with a Python entry point at run.py. That combination usually means the execution and grading core is compiled while orchestration and reporting stay scriptable. The README does not spell out the boundary, so treat that as an inference from the repository layout rather than a documented fact.
Running it: the commands the README gives
The README supplies two concrete invocations. To see the inventory as JSON, run python3 run.py inventory --json. To see the full status breakdown across active and incubating scenarios, run python3 run.py inventory --benchmark-status all --json. Those are the only commands shown in the material provided, and both are inspection commands rather than evaluation commands. The README references a Quick Start section by anchor in its badges, but the body text of that section is not in the material I have, so I cannot tell you the exact flag that selects the core profile or the flag that sets trial count. Do not assume a --profile core syntax exists just because the profiles are named.
On configuration, the v2.0.4 release note says the Hugging Face dataset ships "prompts, fixtures, and scenario configurations for all benchmark-active tasks." So scenario configuration is data, not code, and it is published. That is a meaningful commitment: you can read what a scenario does before you run it, and you can diff configurations between versions. The v2.0.3 note also mentions a "resume model-alias preservation fix," which implies runs can be resumed and that model identity is tracked across a resume. If you interrupt long runs, that fix is the one to check for in your version.
The contamination problem the maintainers name themselves
The v1.1.4 release note contains an unusually direct admission: "Because the benchmark is fully open source, it cannot fully avoid vendors optimizing specifically for the public benchmark." The stated response was a closed-dataset leaderboard, released in v2.0.0 with 33 model results. This is the central limitation of the project and it is not a hypothetical. An open scenario set with published prompts and fixtures is a training target. Once a model vendor has the scenarios, a high score on the open leaderboard no longer distinguishes general capability from targeted preparation.
The closed-dataset board is the mitigation, and it introduces its own problem. A closed dataset cannot be audited by the people reading the scores. You have to trust the maintainers' grading, and the maintainers are also the ones running models on the leaderboard, sometimes with platform access acknowledged in the README's Get Involved section. That is a normal arrangement for a research leaderboard and it is disclosed, but it means the public numbers are not independent.
A second limitation is scope. Everything here is OpenClaw. If your agents run on a different runtime, the scenarios may still be informative as task descriptions, but the live execution path does not apply, and the project vendors IronClaw and NanoClaw harness sources specifically to support cross-harness evaluation. That vendoring exists because the default path is single-runtime.
Where report aggregation fits
LLMLeadBoard is a separate component in the same repository family, and it solves a different problem. It aggregates benchmark scores by model and inference mode from public model reports. The README reports 63 models, 55 source reports, and 444 benchmarks in the current coverage.
The difference in approach is the point of comparison. ClawProBench generates its own scores by executing agents; LLMLeadBoard collects scores that vendors and researchers already published. The first gives you control over the evaluation and no control over contamination. The second gives you breadth across benchmark families and no control over methodology, because each source report ran its own evaluation. LLMLeadBoard's stacked-cell feature, which shows different scores when reports disagree about the same model and benchmark, is an honest way to present that: it displays the disagreement instead of averaging it away. If you need a single number per model, that design will frustrate you.
Maintenance, licensing, and what to verify
The repository is active, with the last push dated 2026-08-25, and the release cadence visible in the notes is frequent: v1.1.1 through v1.1.6, then v2.0.0 through v2.0.4. Each release adds leaderboard entries or fixes harness behavior. That cadence is a real cost. Scenario counts, profile membership, and checker behavior can shift between minor versions, so a score you recorded against v1.1.5 is not directly comparable to one recorded against v2.0.3 unless the core profile's 26 scenarios are unchanged. Pin a version and record it alongside any result you publish.
The licence is Apache-2.0, which permits commercial use and modification and includes a patent grant. It also requires that you preserve notices and state changes. I am not giving legal advice; read the LICENSE file in the repository for the terms that bind you, particularly if you redistribute the scenario data from Hugging Face rather than just the harness code. The dataset and the code may carry different terms, and the README does not state the dataset licence in the material provided.
Before you invest in a run: check the inventory on your checkout, confirm which profile you are scoring against, and verify that the checkers for your chosen scenarios are the hardened ones from v2.0.3 rather than older versions. The resume fix matters too if your runs are long enough to be interrupted.
Editorial conclusion
Adopt ClawProBench if you need to compare models or agent configurations inside the OpenClaw runtime itself and you want graded traces rather than a static question set. Do not adopt it if your agents do not run in OpenClaw, or if you need a neutral public ranking: the maintainers state in the v1.1.4 notes that a fully open benchmark cannot fully avoid vendors optimizing specifically for it. Before committing, run python3 run.py inventory --json to confirm the scenario count on your checkout, then check whether the checkers your chosen profile depends on are marked deterministic, and read the Apache-2.0 LICENSE file for the terms that apply to your redistribution.
Community notes