ClawBench: A Browser Agent Benchmark Built Around Everyday Tasks
Open-source benchmark for browser AI agents on daily tasks.
At a glance
- What is it?
- ClawBench is an Apache-2.0 Python benchmark from TIGER-AI-Lab that scores browser agents on daily web tasks. Its value is the task distribution and the trace datasets, not the leaderboard rank.
- Who is it for?
- Adopt ClawBench if you are building or evaluating a browser agent and need tasks drawn from ordinary web use rather than synthetic navigation puzzles, and if you can host the evaluation loop yourself. Skip it if you need an offline, deterministic harness: the README points to live online tasks, so results depend on third-party sites you do not control.
- Can I use it commercially?
- Yes. Apache-2.0 is a permissive licence: you can use, modify and sell software built on it, as long as you keep its copyright and licence notices.
- Is it still maintained?
- Yes. The repository last received commits 2 days ago.
- What is it written in?
- Mainly Python, according to GitHub's language statistics.
Answers come from the project's GitHub data, last synced on September 15, 2026, and from our analysis. They are not legal advice.
DEEP OPEN-SOURCE ANALYSIS
The gap ClawBench targets: everyday browser tasks, not navigation puzzles
Most browser agent evaluations ask whether a model can click through a synthetic obstacle course. ClawBench asks whether it can finish things people actually do in a browser. The repository topics list everyday-tasks, online-tasks and real-world-benchmark alongside browser-agent and computer-use, and the project description states it is an open-source benchmark for browser AI agents on daily tasks. That framing is the whole product. If your agent is meant to book, fill, search, compare or submit something on a live site, the task distribution matters more than the harness code. The project is written in Python, licensed Apache-2.0, and ships as a PyPI package named clawbench-eval, with a leaderboard and dataset hosted on Hugging Face and a project page at claw-bench.com. The README also notes an EMNLP 2026 Findings track and an arXiv identifier, which tells you the intended audience: researchers who need a citable evaluation, and engineering teams who want a number that reflects consumer web behaviour rather than a lab sandbox.
How the evaluation loop is put together
The material shows a split between the package and the data. clawbench-eval is the installable Python side; the tasks live in a Hugging Face dataset (NAIL-Group/ClawBench), and recorded agent behaviour lives in separate trace datasets, TIGER-Lab/ClawBenchV2Trace and NAIL-Group/ClawBenchV1Trace. That separation is the architecture worth understanding. The evaluator does not bundle the tasks; it pulls them, runs your agent against them, and the resulting trajectories are published as their own artefact. Two trace generations exist, V1 and V2, which suggests the task set or the recording format changed between them. The README excerpt does not spell out the task schema, the scoring function, or how a run is resumed, so treat those as things to read in the repository rather than assume. What is clear from the layout is that ClawBench is designed as an online benchmark: the topics include online-tasks, and the tasks are described as daily browser work, which in practice means live pages reached over the network.
Getting it running: the commands the README actually gives
The concrete install path is short. The PyPI badge points at the clawbench-eval project, so the package name is clawbench-eval, not clawbench. A Codespaces quickstart link is present in the README badge block, which is the fastest way to see the expected environment without assembling dependencies yourself. Beyond the package name and the Codespaces entry point, the excerpt does not expose CLI flags, config keys or environment variables. That is a real gap for anyone trying to plan an integration: you cannot tell from this material whether runs are configured by a YAML file, by command-line arguments, or by a Python API call. The Hugging Face dataset page and the repository README are where that has to be resolved. If you are evaluating the project for adoption, budget time for reading the task loader before you promise anyone a completion date, because the install is the easy part and the run configuration is the part the excerpt leaves open.
Where ClawBench is the wrong tool
Live online tasks are the design choice and also the main failure mode. A benchmark that depends on third-party websites inherits their uptime, their layout changes, their bot detection and their regional differences. A task that passed last month can fail today because a button moved, and that failure is indistinguishable from an agent failure unless the harness records enough context to tell the two apart. The trace datasets exist partly for that reason, but they are a record, not a fix. There is a second limitation: the README excerpt gives no scoring details, so you cannot tell how partial credit works, whether a task can be retried, or how variance across runs is reported. For a benchmark whose selling point is realism, the reporting methodology is exactly what a careful reader needs and what this excerpt does not provide. If your requirement is a deterministic, offline, reproducible harness with pinned fixtures, ClawBench is not that, and no amount of Apache-2.0 licensing changes it.
How it differs from WebArena-style self-hosted environments
The obvious comparison is with benchmarks such as WebArena, which stand up self-hosted replicas of real sites and evaluate against them. The difference in approach is where the environment lives. A self-hosted replica is reproducible and offline; you control the DOM, so a failure is almost always the agent's fault. ClawBench points the other way, at online-tasks and everyday-tasks, which means the environment is the real web and reproducibility is traded for realism. That trade is defensible when your goal is to predict how an agent behaves for a user, and it is a liability when your goal is to compare two model checkpoints under controlled conditions. The trace datasets published alongside ClawBench are the mitigation: they let a reader inspect what happened in a run even when the site has since changed. Neither approach is strictly better. Pick based on whether your bottleneck is measurement noise or ecological validity.
Maintenance cost and the Apache-2.0 terms
The release cadence visible here is fast: v0.9.1 on 2026-08-08, v0.9.2 on 2026-08-19, v0.10.0 on 2026-08-30, with a last push dated 2026-09-10. Three releases in roughly three weeks means the interface you integrate against can move, and pinning clawbench-eval to a specific version is the sensible default rather than tracking the latest. The licence is Apache-2.0, which permits commercial and private use and requires that you keep the licence and notice files and state significant changes; it also includes a patent grant. That is a permissive arrangement, but it is not legal advice and it does not cover the datasets, which are hosted separately on Hugging Face and may carry their own terms. Check the dataset cards before you redistribute task content or traces. The upgrade cost is mostly the usual one for a fast-moving evaluation package: re-run your baseline after every minor bump, because a task-set change between V1 and V2 traces already shows that the ground truth is not frozen.
Who should adopt ClawBench, and what to check first
The fit is narrow but real. If you are training or shipping a browser agent and you want an external number that reflects ordinary web use, ClawBench is aimed at you, and the combination of a PyPI package, a Hugging Face dataset, published traces and a leaderboard means you can both run it and compare against others. If you are writing a paper, the EMNLP 2026 Findings listing and the arXiv identifier give you something citable. If you need a controlled, offline harness for regression testing, or if your agent only ever touches sites you host, the online task model works against you. Before you commit, read the dataset card for NAIL-Group/ClawBench to learn the task schema, confirm that the clawbench-eval version on PyPI corresponds to the v0.10.0 tag, and inspect the V2 trace dataset to see whether its tasks overlap the categories you care about. The scoring rules are the open question this README excerpt does not answer, and they are the first thing to read in the repository.
Editorial conclusion
Adopt ClawBench if you are building or evaluating a browser agent and need tasks drawn from ordinary web use rather than synthetic navigation puzzles, and if you can host the evaluation loop yourself. Skip it if you need an offline, deterministic harness: the README points to live online tasks, so results depend on third-party sites you do not control. Before committing, verify the task schema in the Hugging Face dataset, confirm that clawbench-eval on PyPI matches the v0.10.0 tag, and check whether the trace datasets cover the task categories you care about.
Community notes