Claw-Eval: A Human-Verified Agent Benchmark With a Three-Trial Pass Rule

The problem Claw-Eval targets: agent scores that do not survive a rerun

Single-run agent evaluations are easy to game by accident. A model that flails through a task and happens to land on the right final state looks identical, in a pass/fail table, to a model that took a direct route. Claw-Eval's stated answer is procedural rather than architectural: the README describes a Primary Metric called Pass^3, under which a model must pass a task across three independent trials (N=3) before it earns a success credit, and a task is marked passed only if the success criteria hold in all three runs. The project also frames this as a trust problem, subtitling itself "Towards Trustworthy Evaluation of Autonomous Agents" and stating that all 300 tasks carry human-verified rubrics, 2,159 of them in total.

The intended user is not someone benchmarking a toy prompt. It is a team that has an agent loop, a model endpoint, and a claim to make about reliability, and needs a task set where the grading criteria were written by people rather than generated. The nine categories and three splits (general with 161 tasks, multimodal with 101, multi_turn with 38) suggest the target is breadth across office-style work rather than a single vertical.

How grading actually works: trajectories, rubrics and three dimensions

The README says agents are graded through full-trajectory auditing on three dimensions: Completion (did the agent finish the task), Safety (did it avoid harmful or unauthorized actions), and Robustness (does it pass consistently across multiple trials). That is a different data flow from benchmarks that only inspect a final diff or a final answer string. The trajectory is the unit of evidence, which is why safety can be scored at all: an agent that reaches the correct end state by deleting a directory it should not have touched is visible in the trace and not in the output.

The dataset schema is small and worth reading before you plan an integration. Each record carries task_id, query, fixture (a list of required fixture files, shipped in data/fixtures.tar.gz), language (en or zh), and category. The fixture field is the load-bearing one. A task is not self-contained in the dataset row; it needs the files named there to be present in the sandbox before the agent starts. The README also notes that, due to file size limits, the complete fixtures including all videos live on Hugging Face rather than in the repository, so a clone alone may not give you a runnable multimodal split.

One design choice deserves a direct comment. The README states that grading uses gemini-3-flash for general and multimodal tasks, and claude opus4.6 for both the grader and the user-agent in multi_turn tasks. Using a model as the judge is common, but it makes the score a function of the judge as well as the agent. The multi_turn split is the most exposed here, because the same model family plays two roles: it simulates the user persona and it grades the conversation. That is a real coupling, and the README does not describe a mitigation for it.

Getting a run started: uv, two keys, and the batch command

The Quick Start is short, which is a good sign for a harness. It recommends uv for dependency management and gives these steps:

pip install uv uv venv --python 3.11 source .venv/bin/activate

export OPENROUTER_API_KEY=sk-or-... export SERP_DEV_KEY=... bash scripts/test_sandbox.sh

The second key is conditional. The README says to add SERP_DEV_KEY for tasks that need real web search, and points at novada.com as a convenient source. The test_sandbox.sh script is the environment check; the README does not document what it prints on success, so treat the first run as exploratory rather than as a pass/fail gate.

The actual evaluation is a single command with a model config file:

claw-eval batch --config model_configs/claude_opus_46.yaml --sandbox --trials 3 --parallel 16

The --trials 3 flag is where Pass^3 is enforced, and the README notes that different task families use different configs: config_general.yaml, config_multimodal.yaml, and config_user_agent.yaml. The --sandbox flag and the roadmap item "Enhanced sandbox iso" imply isolation is part of the run, though the README does not specify the isolation mechanism, so verify that yourself before pointing this at an agent with real credentials. There is also an EvalScope integration listed under Updates, which the README says supports running Claw-Eval against OpenAI-compatible endpoints with CLI or Python usage, saved predictions, scoring, and standardized reports.

The reproducibility promise is still a promise

Under Evaluation Logic, the README states: "We are committed to end-to-end reproducibility. Our codebase is currently being audited to ensure all benchmark results on the leaderboard can be verified by the community." That sentence is doing a lot of work, and it is the single most important line in the document for anyone making an adoption decision. It says the audit is in progress, not finished. Until it lands, a leaderboard number from claw-eval.github.io is something you can read but not necessarily reproduce from a fresh clone.

The same section describes a manual escape hatch: when execution errors come from network or API fluctuations, the team manually re-triggers the evaluation to guarantee exactly 3 successful trajectories. Read carefully, that is a hand-operated retry loop, not an automated one, and the README does not describe a cap on retries. Retrying until three clean trajectories exist is defensible when the failures are genuinely infrastructural, but it does mean the published numbers were produced with human intervention in the loop. If you plan to quote Claw-Eval results in a comparison, that is a detail to state rather than bury.

Where Claw-Eval is the wrong tool

Three cases. First, if your agent's value is in code changes to an existing repository, Claw-Eval's splits (general, multimodal, multi_turn) do not describe that workflow, and the fixture model assumes task-local files rather than a checked-out project. Second, if you need an offline or air-gapped benchmark, this is not it: the README's own grader choices are hosted models reached through OPENROUTER_API_KEY, and web-dependent tasks want SERP_DEV_KEY from a commercial provider. There is no documented local-judge path.

Third, and less obviously, the three-trial rule has a cost profile that scales badly for slow agents. Pass^3 means three full trajectories per task per model, and the README's example uses --parallel 16, which implies the run is expected to be wide. On a 300-task set that is up to 900 trajectories, and the README's note about re-triggering failed runs means the true trajectory count can exceed that. If you are iterating on an agent design daily, the Pass^3 protocol is a release gate, not a development loop. Use a single-trial subset while you build, and reserve the full three-trial run for the number you intend to publish.

How it differs from SWE-bench and tau-bench

The closest well-known comparison points are SWE-bench and tau-bench, and the difference is in what counts as evidence. SWE-bench grades a patch against a test suite: the artifact is the diff, and the verdict is deterministic once the tests run. Claw-Eval grades a trajectory against human-written rubrics, which lets it score safety and process, at the cost of introducing a model judge into the loop. tau-bench grades multi-turn tool use against a database end state and a simulated user; Claw-Eval's multi_turn split is the analogous piece, but it is 38 tasks out of 300, so it is a component rather than the whole benchmark.

The practical consequence: SWE-bench results are cheap to reproduce because the oracle is code, and Claw-Eval results are more expensive to reproduce because the oracle is a hosted model plus a rubric. That trade buys coverage of tasks where there is no test suite to run, such as webpage generation or document extraction, which is exactly the gap the multimodal split is aimed at.

Licence, upkeep, and what a fork inherits

The repository is MIT-licensed, which permits commercial use, modification and redistribution provided the copyright notice and permission notice are preserved. That is a permissive baseline and it is the same licence the dataset carries on Hugging Face as far as the README indicates, but the README does not spell out dataset licensing separately, so confirm the terms on the Hugging Face and ModelScope dataset pages before redistributing fixtures. Nothing here is legal advice.

The maintenance cost you inherit is not the Python code, which is small enough to read. It is the external surface: an OpenRouter key, an optional search API key, and two named judge models whose availability and pricing you do not control. The README's own updates section shows the benchmark versioning itself (v0.0.0 in March 2026, v1.0.0, v1.1.0 with 300 tasks), which means a score is only meaningful with the version attached. Pin the commit and record the config file you used, because model_configs/claude_opus_46.yaml is a file in the repository and it can change under you.

Editorial conclusion

Adopt Claw-Eval if you need an agent score that survives a rerun and you are willing to pay for a hosted judge: the Pass^3 rule means a task only counts when all three trials succeed, which is a stricter bar than most agent leaderboards use. Do not adopt it if you need a fully offline, self-contained benchmark, because the README states the graders are gemini-3-flash and claude opus4.6, and web-dependent tasks want a SERP_DEV_KEY. Before committing, verify three things in the repository: that data/fixtures.tar.gz is complete for your chosen split, that your OpenRouter account can reach the named grader models, and that the audit mentioned under Reproducibility has actually landed in the code you clone.

Official sources

1. claw-eval/claw-eval on GitHub
2. Issues
3. License: MIT
4. Project website
5. README