BoxPwnr: A Solver-Pluggable Harness for Grading Agents on CTF and Security Labs

The gap BoxPwnr fills between agent code and challenge platforms

Most agent benchmarks ship as a fixed pair: one harness, one task set, one scoring rule. BoxPwnr inverts that. The README describes it as a plug and play system for testing the performance of different agentic architectures, and the two axes it exposes on the command line are the platform and the solver. The platform axis names HackTheBox, TryHackMe, PortSwigger Labs, Cybench, picoCTF, CyberGym, XBOW, HackBench, ExploitBench, LevelUpCTF, Argus, CTFd instances and a local mode. The solver axis names claude_code, codex, cursor-cli, grok, kiro_cli, external, single_loop_xmltag, single_loop, single_loop_compactation and hacksynth.

The audience is narrower than the description suggests. This is for people who already have a reason to run agents against offensive-security targets: a research group comparing scaffolding choices, a team evaluating whether a CLI coding agent transfers to exploitation work, or an individual who wants to see how a model behaves across hundreds of machines rather than one. It is not a teaching tool and it is not a scanner. Nothing in the repository layout suggests a path for someone who wants to point it at production infrastructure, and the supported targets are all lab environments or CTF instances by name.

The execution loop, the executor, and where traces come from

For the default single_loop family of solvers, the mechanism is a plain read-evaluate-print cycle wrapped around a shell. The README lists the steps: the model receives a system prompt from src/boxpwnr/prompts/generic_prompt.yaml that defines its task and constraints, proposes a command based on previous output, the command runs in the executor, the output goes back to the model, and the cycle repeats until the flag or the platform's success criteria is met.

The executor is a Docker container running Kali Linux by default, selected with --executor docker. The README notes the container is built automatically on first run and that this takes roughly ten minutes, and that a VPN connection is established automatically when the platform requires one. That VPN detail matters more than it looks: platforms like HackTheBox and TryHackMe hand out targets over a tunnel, so the harness has to own the network layer, not just the process layer. It also means the container needs the privileges to bring a tunnel up, which is the kind of thing that behaves differently on a laptop than on a CI runner.

The CLI solvers work differently. claude_code, codex, grok, cursor-cli and kiro_cli run their own agent loop and stream results back to BoxPwnr, according to the README. So the harness is not always the thing deciding the next command. In those modes it is closer to a supervisor and recorder, which is a meaningful distinction when you are interpreting a score: a failure in single_loop is a failure of the prompt and the model, while a failure under claude_code may be a failure of an agent you did not configure.

Output is persisted as traces. The README says conversation and commands are saved for analysis and replay, that a summary can be generated once a flag is found, and that usage statistics covering tokens, cost and turns are tracked. There is a companion site, boxpwnr.info, where the project publishes traces and an interactive viewer for replaying a solve step by step.

Getting it running: submodules, uv, Docker, and .env

The install path in the README is short and specific. Clone with submodules, because the repository depends on them:

git clone --recurse-submodules https://github.com/0ca/BoxPwnr cd BoxPwnr

Then install uv if you do not have it, and sync:

curl -LsSf https://astral.sh/uv/install.sh | sh uv sync

The README states that uv sync creates a .venv. Docker must be installed and running separately; BoxPwnr does not bundle it.

The run command is:

uv run boxpwnr --platform htb --target meow [options]

On first run you are prompted for any required API keys and the answers are written to .env for later runs. That file is the credential surface for the whole tool. The README notes that CLI solvers use their own subscription auth instead of, or in addition to, API keys, so a run under claude_code or codex may not touch .env at all.

The two flags worth internalising are --platform and --solver, both of which take the enumerated values listed above. Everything else in the option set is secondary to those two, because together they define what is being measured. If you change either one between runs, you are no longer comparing like with like, and the trace metadata is the only place that will tell you so later.

What the published numbers do and do not tell you

The README carries a statistics block that is generated between BEGIN_BENCHMARK_STATS and END_BENCHMARK_STATS markers. It reports totals of 4,749 challenges, 2,295 solved, 10,009 traces and 16 platforms, then breaks results down per platform. The spread is wide. HTB Starting Point is listed at 25/25, Cybench at 40/40, picoCTF at 502/503, and XBOW at 102/104, while ExploitBench sits at 2/42 and LevelUpCTF at 50/254.

Read that table as a description of what has been attempted, not as a leaderboard. The rows are not comparable to each other. A challenge on picoCTF and a lab on HackTheBox differ in difficulty, in how the flag is verified, and in how much of the environment the agent controls. The completion percentages are also cumulative across solver versions and model versions, and the README does not break them down by which solver produced which trace. A 51.0 percent on HTB Labs is an aggregate over runs that may have used different scaffolding and different models.

The generated markers are the useful part. Because the block is machine-written, it reflects the repository's current state rather than a frozen claim, which means a number you quote today may not be the number on the page next month. If you need a fixed figure for a paper or an internal report, pin the commit and record the marker contents yourself.

The constraint that shapes everything: the first build and the tunnel

Two operational facts limit how BoxPwnr is used in practice, and both come from the README rather than from speculation.

First, the Docker image is built on first run and takes about ten minutes. That is a one-time cost on a workstation and a recurring cost on ephemeral infrastructure. If you plan to fan runs out across short-lived machines, each one pays it. The README does not describe a prebuilt image registry or a documented way to skip the build, so caching the image yourself is the obvious mitigation and is not something the documentation currently offers.

Second, the VPN is established automatically when the platform requires it. Automatic is convenient until it fails, at which point a run can stall in a way that looks like model incompetence rather than a network problem. The README does not document a troubleshooting path for tunnel failures or a flag to disable the automatic setup. If you are running against a platform that needs the tunnel, treat connectivity as a precondition to verify before you attribute a bad score to the model.

A third, softer limitation: the harness assumes the agent can drive a shell non-interactively. The README states that agents are instructed to provide fully automated commands with no manual interaction, and that commands should include proper timeouts and handle service delays. That is a prompt-level instruction, not an enforced sandbox rule. Nothing described here prevents an agent from issuing a command that blocks, and the quality of the timeout discipline is a property of the prompt and the model, not of the executor.

How BoxPwnr differs from a single-agent CLI harness

The nearest alternative in kind is running a CLI coding agent directly against a target and reading its transcript afterwards. That approach has real advantages: no Python dependency tree, no Docker build, no .env file, and the agent's own logging is the record. It is also what most people do first.

The difference is what gets held constant. When you invoke an agent by hand, the target selection, the credential handling, the network setup and the stopping condition are all your responsibility and all vary between sessions. BoxPwnr moves those into flags and configuration, and it produces a trace in a consistent shape regardless of which solver ran. That is what makes a comparison between single_loop and claude_code possible at all: without a shared recording layer, you are comparing two transcripts written by two different tools.

The cost of that consistency is the abstraction itself. A solver that does not fit the loop model has to be adapted, and the README's external solver option is the escape hatch for agents that BoxPwnr does not ship. Whether that adaptation preserves the agent's behaviour is something you would have to check on your own runs; the documentation does not describe the external interface in the excerpt available here.

Maintenance, versioning and the AGPL-3.0 boundary

BoxPwnr is licensed AGPL-3.0. The practical consequence of that choice, stated plainly and without pretending to give legal advice: if you modify BoxPwnr and let users interact with it over a network, the licence's source-availability condition is generally understood to reach your modified version. That is a different posture from a permissive licence, and it is the kind of thing to put in front of whoever handles licensing at your organisation before you build a hosted service on top of it. Running it locally for internal evaluation is a much simpler case.

On maintenance, the release cadence visible in the material is two releases in roughly five months: v0.3.0 in late February and v0.4.0 in mid July. The project is not archived and the last push is dated after the most recent release, which suggests ongoing work rather than a frozen snapshot. Two releases in that window is a moderate pace, and for a tool whose value depends on tracking external platforms, the pace matters. Platforms change their APIs and their lab formats, and each such change is a maintenance obligation for BoxPwnr rather than for you.

Upgrade cost is concentrated in the two axes. A new release can add or rename platforms and solvers, which means a script that hardcodes --platform values can break on upgrade. The README's platform list already includes entries like Argus and LevelUpCTF alongside the well-known names, and the list has clearly grown over time. Pin a version if you need reproducibility, and expect to re-read the option list when you move.

Who should adopt BoxPwnr, and what to check before you do

Adopt it if your question is comparative. If you want to know whether a different scaffolding, a different prompt, or a different model changes how far an agent gets on security targets, BoxPwnr gives you the two knobs and a consistent trace format to answer that. The published trace viewer at boxpwnr.info is a genuine asset here: being able to replay a solve step by step is more informative than a completion percentage, and it is the part of the project that is hardest to replicate yourself.

Do not adopt it if you need a benchmark with a fixed, citable task set and a stable score. The platform list, the solver list and the target counts all move between releases, and the headline statistics are generated content that changes with the repository. A number quoted from the README without a pinned commit is not a reproducible measurement.

Before you invest time, verify three things on your own machine. That the Docker container builds and that the roughly ten minute first-run cost is acceptable for your setup. That the credential flow for your chosen platform completes, including the VPN step where the platform requires one. And that the solver you intend to compare is still in the --solver list at the specific tag you have checked out, since that list is the part of the interface most likely to have shifted since the documentation you read was written.

Editorial conclusion

Adopt BoxPwnr if you need to compare agentic strategies on the same security targets and want the conversation logs, not just a score. Do not adopt it if you need a stable, versioned benchmark scoreboard: the platform list, solver list and target counts change with each release, and the README's own statistics table is generated content. Before committing, verify three things in your own checkout: that the Docker image builds on your host, that your chosen platform's credential flow works end to end, and that the solver you intend to compare is still present in the --solver list at the tag you pin.

Official sources

1. 0ca/BoxPwnr on GitHub
2. Issues
3. License: AGPL-3.0
4. README
5. Releases