ExploitBench: scoring AI agents on the Chromium V8 exploitation ladder
ExploitBench measures how far AI agents climb, from reaching vulnerable code, to triggering the bug, to building exploit primitives, to arbitrary code execution.
At a glance
- What is it?
- ExploitBench is an MIT-licensed Python harness that runs AI agents inside per-CVE V8 containers and grades how far up the exploitation ladder they climb, from reaching vulnerable code to arbitrary code execution. The interesting part is not the score, it is the 16-flag capability bitmap the grader is built around.
- Who is it for?
- Adopt ExploitBench if you need a graded, per-CVE signal of how far an agent gets in V8 exploitation and you can absorb a 300-turn, multi-seed budget. Do not adopt it if you want a general vulnerability-discovery benchmark, if you plan to train on it (the README explicitly asks that you not do reinforcement learning on it), or if you need to modify bug environments without learning the local rebuild path under benchmarks/bench-v8/.
- Can I use it commercially?
- Yes. MIT is a permissive licence: you can use, modify and sell software built on it, as long as you keep its copyright and licence notices.
- Is it still maintained?
- Yes. The repository last received commits 74 days ago.
- What is it written in?
- Mainly Python, according to GitHub's language statistics.
Answers come from the project's GitHub data, last synced on September 15, 2026, and from our analysis. They are not legal advice.
DEEP OPEN-SOURCE ANALYSIS
The gap ExploitBench was built to close
Most agent evaluations collapse a long technical process into one bit: did the agent solve the task or not. Exploitation does not work that way. Reaching vulnerable code, triggering the bug, turning the trigger into a primitive such as addrof or fakeobj, and finally reaching arbitrary code execution are separate achievements with separate failure modes. An agent that reliably triggers a bug but never builds a primitive is meaningfully different from one that never gets past reconnaissance, and a yes/no score hides that difference entirely. The README states the framing directly: real exploitation is a ladder. ExploitBench is aimed at security researchers and model providers who want the partial credit, not just the summit. The primary audience named in the README is academic researchers who need to replicate experiments, and model providers who want a model tested. The repository asks the second group to supply model credits, which tells you the intended scale of a full run.
How the ladder is actually measured: MCP interfaces and a 16-flag bitmap
The scoring mechanism is not a diff against a reference exploit. Each environment declares an interface, and the shipped config uses rl.mcp.v8_exploit.v1, described in the README as a V8-specific MCP contract carrying a 16-flag capability bitmap covering things like addrof and fakeobj. The container exposes an MCP server to the agent, and the agent is expected to call setup() to learn about the target and grade(...) to evaluate progress. Bug-specific framing lives in the container's setup(), not in the harness prompt, which keeps the harness generic across bugs. That design choice is the strongest thing here: because the capability set is a bitmap rather than a single pass/fail, the same interface can score partial climbs consistently across 41 V8 bugs. It also means the benchmark measures what the interface knows how to detect. If a capability is not one of the 16 flags, an agent that achieves it gets no credit for it. You can inspect the available contracts with exploitbench list-interfaces, which is the right first command if you are considering a non-V8 target.
Model dispatch is prefix-based, which is convenient and slightly blunt
Routing is decided at runtime by the model-id prefix. anthropic/ uses the native SDK and ANTHROPIC_API_KEY with cache_control; openai/, gemini/ and openrouter/ go through LiteLLM with OPENAI_API_KEY, GEMINI_API_KEY and OPENROUTER_API_KEY respectively. Setting OPENAI_API_BASE routes every openai/* id through that gateway, so vLLM, a LiteLLM proxy, Ollama and OpenRouter all work without a code change. The README's example config mixes an openai/llama-3.3-70b entry routed through a gateway with an anthropic/claude-sonnet-4-5 entry that still uses the Anthropic key. Model params are per-entry: the sample sets reasoning_effort: xhigh for a gpt-5 id and notes it pops temperature. The trade-off is that provider-specific behaviour is keyed off a string prefix rather than a declared client type, so a gateway that serves a model under an unexpected prefix will not route the way you expect.
Getting a first run without pulling 70 GB
Installation is make install, which creates .venv/ and installs in editable mode, followed by source .venv/bin/activate so the exploitbench command resolves. Keys go in .env, and exploitbench doctor verifies the environment, Docker and dependencies. The cheap path avoids both image pulls and real spend: make smoke uses a sample environment with --mock-llm, and exploitbench benchmark --test is quoted at roughly $0.04 on a Haiku model. A real single-cell run looks like exploitbench benchmark --config benchmarks/v8.yaml --models anthropic/claude-haiku-4-5 --envs v8-cve-2024-1939 --seeds 1 --turn-budget 100 --cost-cap-usd 1.50, described as about five minutes wallclock. Results are read back with exploitbench summary to list benchmark_ids and exploitbench aggregate --benchmark-id v8 -f markdown. Pre-built images are published to GHCR and pulled on first use, so the roughly 70 GB per-bug images do not have to be built locally unless you are changing a bug environment.
Budgets are the real cost model, and the defaults are not small
The config exposes four separate budgets: turn_budget (300 in the canonical matrix), token_budget at 2500000 counted as output plus creation plus cache_read multiplied by 0.1, context_budget at 180000 for a single turn's input plus output, and max_tokens at 16384. The full matrix in benchmarks/v8.yaml is N models times 41 V8 bugs times M seeds. The README is explicit that you should not run it cold, and lays out a verification ladder in docs/RUNBOOK.md: caching preflight, then a 20-turn smoke, then the full 300-turn run, then an audit, then scaling to all bugs and seeds. That is a sensible order, and it also tells you the honest cost shape of this benchmark. A single cell is minutes and a few dollars. The matrix is a serious spend, and the cost cap flag exists precisely because a misconfigured run can burn budget quietly.
Where ExploitBench is the wrong tool
Two limits stand out. First, the comparison baseline is narrow: benchmarks/v8-small.yaml is described as the 14-bug subset that matches the Claude Opus 4.6 baseline, and the README directs you to it for apples-to-apples comparison against imported-opus historical rows. Run the 41-bug matrix instead and your numbers are not directly comparable to those rows. Second, the whole harness is organised around V8. The MCP interface is named v8_exploit, the capability bitmap is V8-specific, and the pre-built images are V8 images. Nothing in the README claims the design generalises to other targets, and the local rebuild path under benchmarks/bench-v8/ is where you would have to work if you wanted a different bug environment. If your question is about vulnerability discovery in general rather than exploitation depth in V8, this benchmark answers a different question than the one you are asking. The README also asks that you not perform reinforcement learning on it, on the grounds that it can pollute results, and points to Bugcrowd for separate security RL environments. Treat that as a hard constraint on how you use the results, not a suggestion.
Alternatives, and what actually differs
The direct alternative is to build your own harness around the same container images and MCP interface, scoring runs yourself. That is a real option here in a way it usually is not, because the images are public on GHCR and the interface contract is inspectable via exploitbench list-interfaces. What you would lose is the aggregation layer: benchmark_id tracking, exploitbench summary and exploitbench aggregate, the YAML matrix format with --set overrides, and the per-CVE drilldowns published on exploitbench.ai. The other alternative, for teams whose question is about finding bugs rather than exploiting them, is a fuzzing or discovery benchmark, which scores a different activity entirely. The distinction matters because ExploitBench assumes a known CVE and a prepared environment. It measures the climb from a starting point that discovery benchmarks do not give you for free.
Maintenance, licensing and what the material does not tell you
The repository is MIT-licensed and not archived, with a last push of 2026-07-04, so the licence permits commercial use and modification without a copyleft obligation. That is about all the material supports on licensing; nothing here describes how the GHCR images are licensed, and image licensing is a separate question from repository licensing that you would need to check yourself. The material also does not give release history (no releases were retrieved), a support commitment, or any statement about how often the 16-flag interface changes. Because the interface name carries a version suffix, rl.mcp.v8_exploit.v1, the naming implies that a v2 could change the contract, and a change to the bitmap would silently alter what scores mean across runs. If you are building a comparison over time, pin the interface version and record it alongside the benchmark_id. Support is described as best-effort by email for academic replication, which is a reasonable arrangement for a research benchmark and a weak one for anything you plan to depend on operationally.
Editorial conclusion
Adopt ExploitBench if you need a graded, per-CVE signal of how far an agent gets in V8 exploitation and you can absorb a 300-turn, multi-seed budget. Do not adopt it if you want a general vulnerability-discovery benchmark, if you plan to train on it (the README explicitly asks that you not do reinforcement learning on it), or if you need to modify bug environments without learning the local rebuild path under benchmarks/bench-v8/. Verify three things before you spend money: that exploitbench doctor passes on your host, that the GHCR image for your target CVE pulls successfully, and that benchmarks/v8-small.yaml actually matches the historical baseline rows you intend to compare against.
Community notes