PinchBench: Grading LLM Models as OpenClaw Coding Agents

What PinchBench Measures That Prompt Benchmarks Do Not

PinchBench evaluates a model in the role of the reasoning component inside an OpenClaw agent. The README frames the target directly: it measures how well LLM models perform as the brain of an OpenClaw agent. The tasks are drawn from ordinary agent work rather than from isolated capability tests. The repository lists eight categories: productivity (calendar, daily summaries), research (stock prices, conferences, markets), writing (blog posts, emails, humanization), coding (weather scripts, file structures), analysis (spreadsheets, PDFs, documents), email (triage, search), memory (context retrieval), and skills (ClawHub, skill discovery). The stated intent is that a model should be judged on whether it called the right tool with the right parameters, chained actions across steps, coped with ambiguous instructions, and produced a concrete outcome such as a created file, a sent email or a scheduled meeting. That framing defines the audience: teams choosing a model to sit behind an agent, not teams benchmarking raw text generation. The task count is published in the README as 53, and the badge block in the file is generated, so the number moves with the repository rather than being fixed prose.

How a Run Is Structured: Tasks, Graders and the Judge Switch

The unit of work is a task, and the README points contributors at tasks/TASK_TEMPLATE.md for the format. Grading is not uniform. The README states that each task is graded automatically, by an LLM judge, or both, which means the suite mixes deterministic checks with model-scored ones. That distinction matters when you read a score: an automated check can assert that a file exists or that a calendar entry was written, while a judge call is a second model reading the transcript. The judge itself has two modes. By default, with no --judge flag, the judge runs as an OpenClaw agent session. When --judge is passed, the runner calls the model API directly, which the README describes as bypassing OpenClaw personality injection. That is the interesting design choice here: the default judge inherits whatever persona and system context the OpenClaw instance carries, and the flag exists to remove it. If you are comparing scores across runs, you need to know which mode produced them, because the two are not the same grader. Session transcripts are saved to results/{run_id}_transcripts/, so the raw material behind a score is retained locally.

Getting a First Run Without Touching the Leaderboard

The requirements are explicit: Python 3.10 or newer, the uv package manager, and a running OpenClaw instance. The clone and run sequence in the README is three commands. Clone the repository, change into it, then invoke the runner with a model ID that includes its provider prefix, for example ./scripts/run.sh --model openrouter/anthropic/claude-sonnet-4. To narrow the run, pass a comma-separated list of task IDs through --suite, as in ./scripts/run.sh --model openrouter/openai/gpt-4o --suite task_calendar,task_stock. The suite flag also accepts all and automated-only, and automated-only is the one to reach for if you want to avoid judge calls entirely. Other flags worth knowing before you start: --runs N averages multiple attempts per task, --timeout-multiplier N scales timeouts for slower models, --output-dir sets where results land (default results/), and --thinking takes off, minimal, low, medium, high, xhigh or adaptive. Add --no-upload to keep everything local. OpenRouter is described as the default provider used for routing, and the model ID prefix determines which API key is required.

Uploading, Official Keys and the Leaderboard Boundary

Uploading is opt-in through registration. Running ./scripts/run.sh --register requests an API token once, and after that a normal run uploads results automatically with the token. To mark a submission as official on the leaderboard you either export PINCHBENCH_OFFICIAL_KEY or pass --official-key KEY on the command line. A previous results JSON can be re-sent with --upload FILE. The boundary to keep in mind is stated at the top of the README: this repository contains the benchmark skill and tasks, and it is not the source of official leaderboard results. Adding models to the official results happens in a separate repository, pinchbench/scripts, by editing default-models.yml. So the artifact you clone and the artifact that defines the official comparison set are two different things, maintained by the same organisation. If your goal is to appear on the leaderboard under the official label, you need the official key, and the model list is governed elsewhere. That is a governance detail, not a technical one, but it determines what a leaderboard position actually represents.

Where PinchBench Breaks Down

The hard dependency is OpenClaw. There is no mode described in the README that runs the task suite without a live OpenClaw instance, because the agent session is the thing under test. If you do not run OpenClaw, this benchmark is not for you, and no amount of provider configuration changes that. The second constraint is the judge. Because the default judge is an OpenClaw agent session, a model's score can be influenced by the personality injection of your instance, which is precisely why the --judge flag exists to bypass it. Two teams running the same model against differently configured OpenClaw instances are not guaranteed comparable numbers, and the README does not describe a standard instance profile. Third, the grading split between automated checks and judge calls means not every task has an objective answer; the README does not publish a per-task breakdown of which tasks use which grader in the material available here. Fourth, reproducibility depends on external services. Research tasks involve web search and live data such as stock prices and conferences, so identical runs on different days are not identical inputs. Finally, results are only as current as the suite: with 53 tasks and a v2.0.0 release line, task churn will move scores, and the README does not describe a versioning policy that pins a leaderboard entry to a task-set revision.

How It Differs From SWE-bench-Style Harnesses

The obvious comparison is SWE-bench, which grades a model on resolving real GitHub issues against repository test suites. The difference is the object under test. SWE-bench measures patch correctness inside a code repository; PinchBench measures whether an agent completed an office-shaped job: scheduling a meeting, triaging an inbox, extracting a stock price, writing a blog post. Its success criteria are correspondingly broader and softer. A SWE-bench task has a failing test that either passes or does not after the patch. A PinchBench task may end in a judge call reading a transcript. That is a deliberate trade: broader coverage of agent behaviour at the cost of a harder-to-audit score. The other difference is the harness assumption. SWE-bench can run in a container with the repository and its tests. PinchBench assumes a running OpenClaw instance and, for research tasks, live web data. If your question is which model writes correct code against a test suite, this is the wrong tool. If your question is which model behaves competently as an agent with tools, calendars, files and mail, the task categories map onto that question more directly.

Licence, Maintenance and the Cost of Keeping Up

The licence is MIT, stated in the README badge and the repository metadata. MIT is permissive: it allows use, modification and redistribution with the licence and copyright notice retained. That is a statement about the licence text, not legal advice, and it says nothing about the terms attached to the models you benchmark or the providers you route through. On maintenance, the release history shows a v2.0.0 line with a series of release candidates (rc8 and rc9 both dated 27 April 2026, with v2.0.0 on 6 May 2026) and a last push of 2 July 2026, so the project is active on a roughly monthly cadence with a stabilisation phase before the major tag. The upgrade cost sits in the tasks, not the runner. A task-set change alters what a score means, so any internal comparison you keep across versions needs the task revision recorded alongside the numbers. The runner itself is a shell entry point over a Python project managed with uv, so the dependency surface is small. The ongoing cost is operational: a reachable OpenClaw instance, provider API keys for both the tested model and the judge, and the time each run takes across 53 tasks, multiplied by --runs if you average.

Who Should Run This, and What to Check First

PinchBench fits teams that already operate OpenClaw and need to pick or re-validate a model for agent work. It gives you a fixed task list, a runner with a documented flag set, local transcripts, and the option to keep results off the leaderboard with --no-upload. It does not fit anyone without an OpenClaw instance, anyone wanting a hermetic offline benchmark, or anyone treating the public leaderboard as the canonical ranking, since the README says official results are produced from a different repository. The first thing to verify is connectivity: the runner needs both your OpenClaw instance and the provider API key implied by the model ID prefix. The second is grader mode, since a default judge session and a --judge direct API call are different graders and mixing them invalidates comparisons. The third is scope: start with --suite automated-only to see how much of the 53-task set you can grade without a judge model at all, and only then decide whether the judged tasks are worth the extra API spend.

Editorial conclusion

Adopt PinchBench if you already run an OpenClaw instance and want a repeatable way to compare models on tool-calling and multi-step tasks rather than on isolated prompts. Do not adopt it if you cannot stand up OpenClaw, if you need a benchmark that runs without network access to a provider, or if you intend to treat the public leaderboard as an authority: the repository states it is not the source of official results. Before committing, verify three things: that your OpenClaw instance is reachable from the runner, that your provider key matches the prefix of every model ID you pass, and whether your tasks of interest are in the automated-only suite or depend on the LLM judge.

Official sources

1. License: MIT
2. pinchbench/skill on GitHub
3. Project website
4. README
5. Releases