WildClawBench: grading AI agents inside a live OpenClaw instance

The gap WildClawBench is aimed at

Most agent benchmarks measure a capability in isolation: a function call, a JSON parse, a single instruction followed correctly. WildClawBench is built on the opposite premise. The README states that the benchmark drops agents into a live OpenClaw environment, the same open-source personal AI assistant it says real users rely on, and runs 60 original tasks against them. The tasks are described as end-to-end work: clipping goal highlights from a football match, negotiating meeting times over multi-round emails, hunting contradictions in search results, writing inference scripts for undocumented codebases, catching privacy leaks. The intended audience is anyone who has to answer a procurement question rather than a research question. If you are picking a model or a harness for an assistant that is expected to chain tool calls and recover from failures, a benchmark that only checks whether the model can emit a valid tool call tells you almost nothing about the workload you are buying for. The README frames the gap bluntly: agents must chain 10 to 60+ tool calls, adapt when services fail, and decide what to do rather than only how. That is the claim being tested.

Four harnesses, one suite, and why that split matters

The most consequential design decision in WildClawBench is not the task count. It is that OpenClaw, Claude Code, Codex CLI, and Hermes Agent all execute the same 60 tasks under the same grading. The README states the purpose directly: this separates model capability from harness scaffolding, so you can see how much an agent's score depends on its surrounding tools versus the underlying LLM. For an engineering team, that is the difference between two very different purchases. A model that scores well inside one scaffold and poorly inside another is telling you that your tool definitions, retry policy, and context management are doing the work, not the weights. Most public agent leaderboards conflate the two and report a single number. WildClawBench's four-harness layout is the part of the project most worth copying even if you never run its tasks. The cost is real, though: four harnesses means four Docker images and four sets of integration surface to keep current, and the README's news section shows the harness roster was itself an addition in the 2026-05 release rather than part of the original design.

Task shape: multimodal, long-horizon, coding and safety in one suite

The 60 tasks are not a single skill measured 60 ways. The README groups them into five categories with concrete examples. Agency covers multi-step tool orchestration, error recovery, and autonomous planning. Multimodal covers video understanding, image generation, and cross-modal synthesis, with one example describing tracking events across a 45-minute match video to clip precise highlights, and another describing classifying 12 clothing photos, assembling 4 styled outfits, and generating full-body model images for each. Long-horizon covers workflows the README says span 10 to 20 minutes of wall-clock execution, such as negotiating meeting times over multiple email rounds or crawling, classifying and summarizing 50+ academic papers. Coding covers reading an undocumented codebase, installing dependencies, and writing working inference from source alone. Safety covers prompt injection defense, credential leak detection, and harmful content refusal, with the README noting that harmful instructions are buried deep inside normal-looking documents and API keys are scattered across a large git history. That spread is a deliberate trade-off. A suite this heterogeneous is harder to score on a single axis and harder to compare against a benchmark that isolates one capability, but it is also much harder to game by tuning for the task format. The safety tasks in particular are not separable from the rest: an agent that refuses too aggressively will lose points elsewhere in the same run.

Isolation and grading: what the README actually commits to

WildClawBench claims reproducibility through container isolation. According to the README, each task runs in its own Docker container, with the same image, the same data, and the same grading code, and scores are reproducible across machines. The mechanism that prevents leakage is stated plainly: ground truth and grading scripts are injected only after the agent finishes, and are never visible during execution. That is the detail to check first if you are evaluating the benchmark itself. A benchmark that ships its answer key inside the container is measuring whether the agent can find the answer key. Post-run injection removes that path, and it also means the grading step is a separate phase from the agent phase, which has consequences for how you build your own harness around it. The README does not describe the grading rubric in the repository text, and it does not reproduce the scoring code in the tables it presents. The 62.2% figure is attributed to a technical report snapshot and the 67.2% figure to the latest audited OpenClaw runs, but the README does not break either number down by task category. If you need per-category scores to decide whether a model is weak specifically at multimodal work, the repository text alone will not give you that; the arXiv report at 2605.10912 is the place the README points.

Running it: three datasets and a Harbor path

The evaluation code lives in the repository, but the benchmark data does not. The README splits the project into three Hugging Face datasets and tells you which to pick. WildClawBench holds task data and Docker images for all four harnesses, and is the one to use if you want to reproduce the paper's evaluation with this repository's pipeline. WildClawBench-Harbor repackages all 60 tasks in the Harbor format, so that any Harbor-supported agent can be evaluated with a single harbor run command and no benchmark-specific setup. WildClawBench-Trajectories holds complete agent trajectories for the full 60-task suite across a growing roster of frontier models, plus raw evaluation outputs, and the README suggests two uses: inspecting how models actually behave, or mining long-horizon traces for analysis and training. The trajectories dataset is browsable in the HF Agent Trace Viewer according to the news section. That third dataset is the one people underestimate. If your goal is to improve an agent rather than rank it, complete traces of a strong model failing a 45-minute video task are more useful than the score, because they show where the tool chain broke. The README does not spell out the exact harbor run invocation beyond naming the command, so treat the Harbor dataset card as the source for the full argument list.

Where WildClawBench is the wrong tool

The honest limitation is cost and turnaround. The README describes tasks that chain 10 to 60+ tool calls and workflows spanning 10 to 20 minutes of wall-clock execution, run inside Docker containers with real browser, bash, file system, email and calendar tools. That is not a benchmark you run on every commit. If your development loop needs a signal in seconds, a suite built around multi-round email negotiation and 45-minute video processing will be the wrong instrument, and you will end up running a subset so small that the score stops meaning anything. There is a second, subtler limitation. The environment is a live OpenClaw instance, which is the point, but it also means the benchmark is coupled to one assistant's tool surface. A team building a coding agent that never touches email or calendar will find a meaningful share of the 60 tasks measuring capabilities they will never ship. The README does not offer a documented task-subset mechanism for trimming the suite to a domain, so any subsetting you do is your own bookkeeping and your scores stop being comparable to the published leaderboard. Finally, the repository text does not state a per-task runtime budget or a timeout policy, and it does not describe what happens when a container fails mid-run. Those are the operational questions to resolve before you put this in a CI pipeline.

Alternatives and the actual difference in approach

The natural comparison is with single-capability agent benchmarks, and the difference is architectural rather than a matter of task count. A function-calling or tool-use benchmark typically presents a fixed API surface, a scripted environment, and a grader that checks the final call or the final answer. WildClawBench instead hands the agent a live OpenClaw instance with browser, bash, file system, email and calendar tools, and grades after the run with ground truth injected post-hoc. The consequence is that failure modes differ. In a scripted benchmark, an agent fails by producing the wrong output. In WildClawBench, an agent can fail because a service was slow, because it chose the wrong tool at step 12 of 40, or because it never recovered from an error it caused itself. That is closer to production and correspondingly noisier. The second comparison is internal to the project: WildClawBench's own four-harness layout is a comparison against the common practice of publishing one number per model. If you have already standardized on Harbor, the Harbor-format dataset is the lower-friction entry point, since the README says it needs no benchmark-specific setup. If you have not, the main dataset plus this repository's pipeline is the path the README describes for reproducing the paper.

Maintenance, licence and what to verify first

WildClawBench is MIT-licensed, which permits commercial and internal use with the usual attribution and warranty-disclaimer conditions; that is a factual note about the licence identifier, not legal advice, and you should have your own counsel review anything you redistribute. The repository is Python, not archived, and the last push recorded is 2026-08-17, with the news section showing a steady release rhythm: four harnesses added in 2026-05, leaderboard expansion in 2026-07, and the Harbor and Trajectories datasets in 2026-08. There are no retrieved releases, so versioning appears to move through the main branch and the Hugging Face datasets rather than tagged artefacts. That matters for upgrade cost. Because the task data and Docker images live on Hugging Face while the evaluation code lives in the repository, the two can drift, and pinning a dataset revision is the only way to keep a score comparable over time. The README also notes that trajectories are continuously updated as new models are evaluated, which is good for analysis and bad for reproducibility unless you pin. Before adopting, verify three things: that the WildClawBench dataset revision you pull matches the harness you intend to run, that the grading scripts in the repository match the rubric described in the arXiv report, and that your infrastructure can host the four harness images, since the README states each task runs in its own Docker container and does not describe a lighter-weight execution mode.

Editorial conclusion

Adopt WildClawBench if you are choosing between agent harnesses or between models for a tool-heavy assistant workload and you need a score that is not produced by a mock environment. Do not adopt it if you need a fast inner-loop signal during development, or if you cannot run Docker with real browser, bash, email and calendar tools. Before you commit, verify that the task data and Docker images you pull from the WildClawBench Hugging Face dataset match the harness you actually intend to evaluate, and check the arXiv report for the grading rubric behind the 62.2% and 67.2% figures, since the README reports those numbers without reproducing the scoring code in the same table.

Official sources

1. InternLM/WildClawBench on GitHub
2. Issues
3. License: MIT
4. Project website
5. README