ExploitGym: A 869-Instance Exploit Development Benchmark With a Controller, Firewall and LLM Proxy

What ExploitGym measures that a CVE list does not

A list of vulnerable programs tells you nothing about whether an agent can turn one into a working exploit. ExploitGym's stated purpose is to close that gap: it is a benchmark built from real-world vulnerabilities across userspace programs, Google's V8 engine and the Linux kernel, and its target metric is an agent's ability to develop exploits against those targets. The audience is research groups and evaluation teams who need a repeatable harness rather than a hand-assembled lab. The repository is Apache-2.0, the current release is v1.0 with 869 instances, and the canonical task list for that release is data/task_ids/v1.txt. The citation in the README points to an arXiv preprint, so the scoring methodology lives in the paper, not in the repository text. If you are choosing between this and writing your own container-per-CVE harness, the deciding factor is whether you want the surrounding infrastructure (firewall, model proxy, token minting) maintained by someone else.

The controller, firewall and proxy triangle

The architecture visible in the README is three cooperating services plus per-task containers. The controller issues task tokens and derives expected flags, which is why it holds a salt and a flag seed. A Squid firewall image, pulled as ubuntu/squid:latest, provides outbound network isolation for agent containers, documented separately in docs/firewall.md. An LLM proxy sits between the agent and the model providers, and its admin key is exported alongside the controller secrets. The README is explicit that no secret is baked into the repository: the controller generates its token salt, flag seed and API key on startup, or takes them from CYBERGYM_SERVER_SALT, CYBERGYM_SERVER_FLAG_SEED and CYBERGYM_SERVER_API_KEY. Both ends must use the same values. That design has a direct consequence for anyone running this on a cluster: the controller is not stateless across restarts unless you pin those three variables, and the runner needs CYBERGYM_ADMIN_KEY to mint task tokens at all. The flag derivation scheme means the expected answer for each task is computed rather than stored, so you cannot inspect a static answer file to debug a mismatch.

Getting from a clone to a first agent run

The quick start is six steps and none of them are optional. Python dependencies come from uv sync --extra proxy; the proxy extra is separate because the LLM proxy is not part of the base install. Runtime artifacts (gdb, socat, nc, node and the agent CLIs) plus task data extraction come from bash scripts/setup/setup_data.sh, and bash scripts/setup/validate.sh is the install check. Target images are pulled per task family with uv run scripts/setup/pull_images.py data/task_ids/sample.txt, and the Squid image with docker pull ubuntu/squid:latest. Provider credentials are exported as OPENAI_API_KEY and ANTHROPIC_API_KEY. Then uv run scripts/setup/pre_run.py data/task_ids/sample.txt runs readiness checks and starts the controller, firewall and LLM proxy, auto-detecting any that are already running. The final block exports CYBERGYM_ADMIN_KEY plus the three controller secrets that pre_run.py prints, after which uv run examples/run_agent.py --help is the entry point. Note the ordering trap: pre_run.py prints the secrets you need for the next step, so you cannot fully prepare the environment before the first run. docs/setup.md covers system dependencies, GDB, static node and the agent CLIs; docs/eval.md covers starting the services by hand if you would rather not use pre_run.py.

Where the harness fights you

The heaviest constraint is that this is not a library. There is no importable scorer you can call from an existing evaluation pipeline; the documented path is a controller process, a firewall container, a proxy and per-task Docker images, all addressed through environment variables. If your infrastructure does not permit long-lived local services or Docker image pulls per task family, the benchmark does not degrade gracefully. The second constraint is secret coupling. Because the controller generates its salt, flag seed and API key at startup, any restart without the exported variables produces a new set, and the README's warning that both ends must use the same values means a stale export in your shell silently breaks token minting or flag derivation rather than failing loudly at startup. Third, the README gives no timing, resource or isolation requirements for the kernel and V8 tasks specifically, so capacity planning for those families is not answerable from the repository text alone. Finally, the task data under data/tasks/ derives from external upstreams and retains their licenses per DATA_LICENSE.md; the Apache-2.0 grant covers the source code, not that data, so redistribution of task content is a separate question from redistributing the harness.

How this differs from CyberGym and from hand-rolled CVE harnesses

The README's own badges point at cybergym.io and the paper title names CyberGym, which makes the relationship worth stating plainly: ExploitGym is the benchmark release associated with that line of work, and the repository is the harness rather than a hosted service. The practical difference from a hand-rolled harness is the firewall and proxy layer. A typical in-house setup lets the agent container reach the model API directly and trusts the agent not to exfiltrate or fetch a public exploit. ExploitGym inserts a Squid firewall for outbound isolation and routes model traffic through its own proxy with an admin key, so the network path is a controlled part of the experiment instead of an assumption. That is a real methodological difference: it constrains what the agent can retrieve mid-episode, which changes what a passing score means. It also means you inherit a Squid dependency and a proxy that must be running before any task executes. If your evaluation does not care about outbound isolation, you are paying setup complexity for a property you are not measuring.

Maintenance, versions and what the licence does not cover

The repository is not archived, and the README states the released benchmark is actively maintained with v1.0 as the current release and CHANGELOG.md holding the version history. There are no retrieved releases, so versioning appears to run through the changelog and the task list files rather than tagged artifacts. That matters for reproducibility: if you score against data/task_ids/v1.txt, record the commit you used, because the task list is a file in the repository and can change without a release tag. On upgrades, the setup scripts are the interface you depend on, so a change to setup_data.sh, validate.sh or pre_run.py is a breaking change for your pipeline even if no version number moves. The licence split is the other maintenance item. Source code is Apache-2.0; the bundled task data under data/tasks/ comes from external upstreams and keeps their licenses per DATA_LICENSE.md. Apache-2.0 on the harness therefore tells you nothing about whether you may redistribute the vulnerabilities themselves. Check DATA_LICENSE.md before mirroring data/tasks/ anywhere, and treat that as an engineering constraint on your artifact pipeline rather than a legal question this article can settle.

Who this is for, and the first three things to check

ExploitGym fits teams who already run container-based agent evaluations and want exploit development measured against real vulnerabilities with a controlled network path. It does not fit anyone who needs a quick local score, a pure-Python dependency, or a benchmark that runs without Docker. The setup sequence assumes you can pull images, run a Squid container and keep three services alive for the duration of a run. Verify three things before committing: run bash scripts/setup/validate.sh and confirm it passes on your host, since it is the only documented install check; confirm that data/task_ids/v1.txt is the list you intend to score against and pin the commit, because the 869-instance count belongs to release v1.0 and the file can move; and decide up front how you will persist CYBERGYM_SERVER_SALT, CYBERGYM_SERVER_FLAG_SEED and CYBERGYM_SERVER_API_KEY across controller restarts, because regenerating them mid-evaluation invalidates the tokens and flags your runner is deriving. The kernel and V8 task families are the ones to size first, since the README documents no resource envelope for them.

Editorial conclusion

Adopt ExploitGym if you are evaluating agent exploit development against real vulnerabilities and can absorb a multi-service setup: Python via uv, built runtime artifacts, per-task Docker images, a Squid firewall and an LLM proxy with a shared admin key. Do not adopt it if you want a single pip install and a pytest run, or if you cannot give agent containers outbound network isolation, since the firewall step is part of the documented flow. Before running anything, verify three things: that data/task_ids/v1.txt matches the 869 instances of release v1.0 you intend to score, that scripts/setup/validate.sh passes on your host, and that the controller salt, flag seed and API key are exported identically at both ends, because mismatched values break token minting and expected-flag derivation.

Official sources

1. Issues
2. License: Apache-2.0
3. README
4. sunblaze-ucb/exploitgym on GitHub