augment-swebench-agent: a small agent harness for the SWE-bench Verified benchmark

What SWE-bench actually measures, and why it is harder than LeetCode

The framing is in the first paragraph and it matters for judging the tool. SWE-bench Verified tests how well AI systems handle software engineering tasks pulled from actual GitHub issues in popular open source projects, and where most coding benchmarks focus on isolated Leetcode style programming problems, SWE-bench involves codebase navigation, iterating against a suite of regression tests, and overall much more complexity. So the benchmark is not about writing a function from a signature, it is about finding the right place in an unfamiliar repository, changing it, and knowing whether the change was right, which is the part the regression suite answers. The repository description calls itself the number one open source SWE-bench Verified implementation, and the reported figure is 65.4 percent success on a first ever submission, achieved with Claude Sonnet 3.7 as the core driver and OpenAI's o1 as the ensembler. The stated reason for deferring their own models is that they wanted a strong open-source baseline built on off-the-shelf models.

A small agent with three tools and a borrowed architecture

The feature list makes the deliberate smallness the point. It is a small and simple coding agent implementation plus a SWE-bench Docker harness that is easy to run and build on top of. The agent has three tools: bash command execution, file viewing and editing, and sequential thinking for complex problem solving. The architecture was forked from Anthropic's own blog post about SWE-bench, and since Anthropic's models were considered state of the art on code at the time, Claude Sonnet 3.7 became the agent's core driver. The rest of the feature list covers the prompt template and system prompt from the submission, the integration with Anthropic's Claude for the core agent and OpenAI models for ensembling, command approval management for safe execution, the majority vote ensembler, and support for running the agent inside a Docker container or running the SWE-bench evaluation harness. The repository tree is small enough to read in an afternoon, with cli.py, run_agent_on_swebench_problem.py and majority_vote_ensembler.py at the root, a prompts directory, a tools directory and a utils directory.

Setup is a clone, a script and two API keys

The prerequisites are Docker, which was tested with version 26.1.3 build 26.1.3 on Ubuntu 22.04, an Anthropic API key for the Claude models and an OpenAI API key for the OpenAI models. Then the repository is cloned, the dependencies installed through the setup script, and the keys exported:

   git clone https://github.com/augmentcode/augment-swebench-agent.git
   cd augment-swebench-agent

   ./setup.sh
   source .venv/bin/activate

   # For Anthropic Claude models
   export ANTHROPIC_API_KEY=your_anthropic_api_key_here

   # For OpenAI models
   export OPENAI_API_KEY=your_openai_api_key_here

Both keys are needed even for an interactive session, because the ensembler path uses the OpenAI model, and the project names both in the prerequisites rather than making the second optional. The Python side requires 3.11 or newer, and the manifest pins the two model SDKs exactly, anthropic at 0.47.0 and openai at 1.59.x, which is the sort of pinning that keeps a research harness reproducible months later. The toolchain is otherwise ordinary: datasets for the benchmark data, docker and docker-pycreds for the container side, prompt-toolkit and rich for the terminal interface, and pyright for type checking with a configuration file for CI.

The interactive mode is for experimenting, and it has a container escape hatch

There are two ways to use the repository. Interactive mode runs cli.py to spin up an interactive agent for experimentation or as a personal coding assistant, and the agent runs in the current directory by default. The command-line options are a workspace path, a problem statement that makes the agent non-interactive, a flag for requiring permission before executing commands, and two options for Docker, one giving the shared volume path and one the container id, each of which must be set if the other is used:

python cli.py --workspace /path/to/project --problem-statement "Fix the login issue"

python cli.py --use-container-workspace --docker-container-id <container_id> --workspace /path/to/docker/volume

The permission flag is the one worth noting. Command approval management is listed as a feature, and the needs-permission option controls whether the agent asks before it runs commands, which defaults to false. An agent that can run bash is a significant capability to hand over by default, so if you are pointing this at a real repository rather than a scratch directory, turning permission on is the difference between a supervised tool and an unsupervised one. A non-interactive run with a problem statement is also supported, for example python cli.py --problem-statement "Implement a feature to sort items by date", which is how the same agent is driven by the benchmark harness rather than by a person.

The evaluation harness, and why eight processes is the ceiling

The SWE-bench mode runs the agent on benchmark problems through run_agent_on_swebench_problem.py, and the small test run is the recommended first step:

python run_agent_on_swebench_problem.py --num-examples 5 --num-candidate-solutions 2

The exact script name in the documentation is run_agent_on_swebench_problem.py, and the same script generates two candidate solutions for each of five problems, runs the evaluation step for each candidate, and prints instructions for running the ensembler on the results. The options are the number of examples, which defaults to running on all of them, the number of candidate solutions per example, defaulting to eight, the number of processes for each example, also defaulting to eight, and the shard count and shard id. The parallelism advice is the most useful thing in the README. There are 500 examples in SWE-bench Verified, and beyond eight processes Docker hits issues, so the recommendation is to run eight processes and use the shard count and shard id flags to split the dataset across machines, which circumvents the Docker scaling limit. The measured cost is stated too: a couple of hours for the full evaluation of one candidate solution per problem, with ten shards across separate Kubernetes pods running eight processes each.

The ensembler asks a model to pick the most common solution

The Majority Vote Ensembler is the piece that turns several candidate patches into one submission, and the mechanism is worth understanding because it is not a test-based selection. The tool takes a JSON file containing multiple candidate solutions for a problem, presents them to OpenAI's o1 model, and asks it to analyse and select the most common solution. The word most is doing the work: the ensembler is looking for the answer that several candidates agree on, not the one that a test suite likes best, which means the evaluation step and the selection step are independent of each other. The repository ships example data for it, an example_ensembler_data.jsonl and an example_ensembler_results.json, so you can see the input and output shapes without spending anything on model calls. There is also a merge_shards.py script at the root, which is what joins the output of the sharded runs back together, and swebench_patch.diff as a reference patch. The licence field on the repository is not one GitHub can classify, so the terms are in the LICENSE file and are worth reading given the code came out of a commercial submission.

Editorial conclusion

augment-swebench-agent suits a developer who wants to read a complete agent implementation, run it against real issues from popular open source projects, and see the ensembling step rather than a framework abstraction. It does not suit a production coding assistant, since the interactive mode is explicitly for experimentation, and it does not suit a team without Anthropic and OpenAI API keys and a Docker setup. Before running a full evaluation, do the small test run the project suggests, and read the rate limit warning, because running eighty agents in parallel against Anthropic is something the authors can do and you may not be able to.

Frequently asked questions

Official sources

1. augmentcode/augment-swebench-agent on GitHub
2. Issues
3. Project website
4. README