AAABench: a harness that hands an agent Unreal Engine 5 and asks for an open-world game

The problem AAABench is aimed at

Most agent benchmarks hand over a task with a known shape: fix this function, pass these tests, close this issue. A model that has seen a thousand similar tasks can pattern-match its way to a plausible answer without understanding anything underneath. AAABench takes the opposite approach. It gives an agent a real Unreal Engine 5 editor, driven live over MCP, a production brief, a shelf of reference documentation, and no human help, then asks it to build an open-world game. The README's framing is blunt: the agent decides the geography, the districts, the roads, the buildings, the people, the traffic, the weather, the game's own screens, and what to fix when it does not work. Nobody points at anything.

The audience is narrow and specific. This is for people evaluating long-horizon agent capability, not for people who want a game built for them. The README lists what the task is meant to expose: real-world understanding of how a city works, causal reasoning rather than imitation, execution across sessions that start cold, prose quality in the story bible and radio scripts, self-verification through viewport capture, and systems thinking across traffic, crowds, weather and economy. The claim worth taking seriously is that a world cannot be faked by pattern-matching, because a place built without knowledge of how places work looks wrong instantly, to anyone, with no expertise required. That is the design bet.

What the harness actually contains

The repository is deliberately thin at the top level: `.claude/`, `HARNESS-RULES.md`, `PROMPT.md`, `bin/`, `docs/`, `project/` and `tools/`. Each piece has a job. `PROMPT.md` is the demand itself, and the README says outright that this is the benchmark. `HARNESS-RULES.md` draws the line between operating the harness and doing the agent's job. `bin/` holds the scripts that run sessions and check whether they are working. `tools/` holds what the agent uses: eyes, widgets and image generation. `docs/` is the handbook the agent may consult, covering production workflow, the level pipeline, systems budgets, detail and density, parallelism, an inventory of real-world objects mined from OpenStreetMap, asset and mocap and map-data and rendering sources, and the engine's version traps. `project/` is the Unreal project skeleton.

There is also a `.claude/skills/` directory with 21 skill packs: game AI, level design, game feel, shaders, Niagara, Blueprints, Enhanced Input, behaviour trees, physics tuning, camera, dialogue, audio, save systems, performance, and reference-image search. That directory name is a signal in itself. The harness is built around a particular agent runtime's skill-loading convention, and the README does not present it as runtime-agnostic. If you run it on another model, the README asks you to open an issue with what broke, which is an honest admission that portability is untested.

The one rule, and why it constrains the operator

AAABench has a single governing constraint, stated in the README as the one rule: provide conditions, resources and the demand, but never diagnosis, never the fix, never an answer. The reasoning given is that whether the model notices its own mistakes is the capability being measured, so every hint is a result you can no longer claim. `HARNESS-RULES.md` is the document that turns that principle into an operating boundary.

This is the part of the design most likely to be violated in practice, and it is worth being direct about why. When you watch an agent spend an hour placing buildings on a floodplain, the instinct to intervene is strong, and intervening is cheap. The harness gives you no mechanism to record that you intervened, and no way to mark a run as contaminated. That is a real gap: the rule is enforced socially rather than technically. A harness that logged operator keystrokes or session interruptions would make the constraint auditable. As shipped, the honesty of the result depends on the honesty of the person running it. For a benchmark whose entire value rests on not helping, that is the weakest seam in the repository.

Installing AAABench and running a first session

The README gives a four-command quick start. The first clones the repository and moves into it, the second copies the Unreal project skeleton to a working directory named `AgentCity`, the third installs plugins, renderer features and Python libraries, and the fourth boots the editor, hands over the demand and keeps the session going.

git clone https://github.com/ukanwat/aaabench && cd aaabench
cp -R project AgentCity
./bin/setup-capabilities.sh
./bin/run-agent.sh

Expect `setup-capabilities.sh` to be the step that fails first on a machine that has not run Unreal before, since it touches plugins, renderer features and Python libraries, all of which are version-sensitive against your engine install. `run-agent.sh` handles one session: it boots the editor, waits for MCP, hands over the demand, resumes the session if it stops early, and relaunches the editor if it dies. That list of failure handling is itself informative. The script is written on the assumption that the editor will crash.

For unattended work, `bin/run-many.sh` runs sequential sessions and is single-instance locked, so you cannot accidentally start two runs against the same project. The agent's sensing side lives in `tools/ue_qa.py`, which the README describes as viewport capture and inspection. Without that sensor the agent is generating blind, and the self-verification portion of the benchmark measures nothing.

Where AAABench is the wrong tool

The repository contains no results. The README says this more than once, and the badge row carries a `contains: no results` marker. The images in `docs/media/` come from one run and exist to show what the demand asks for; the README states that everything an agent produces belongs to the run that produced it, not to the repository. So AAABench cannot tell you whether a given model is good at this. It can only let you find out for yourself.

That rules it out for several common needs. If you want a leaderboard to cite, there is nothing here to cite. If you want a fast signal, the design works against you: sessions run long, the editor crashes and gets relaunched, and `run-many.sh` is sequential rather than parallel, so throughput is bounded by single-instance locking. If you need reproducible scoring, note that the README describes certification requirements inside `PROMPT.md` but does not describe an automated scorer, and no scoring script appears in the top-level layout. Judging a run appears to mean looking at the world and the prose, which is the point of the design but is not a number. And if your interest is Unreal Engine assistance rather than Unreal Engine evaluation, this is the wrong category of tool entirely: nothing here helps you build a game faster.

How this differs from Unreal copilot tools

Searches around this project cluster on assistant products for Unreal Engine: AI copilot plugins, Codex for Unreal Engine, Blueprints AI, and assorted named tools in that space. The distinction matters because the names sound adjacent and the purposes are opposite. Those tools sit inside your workflow and try to reduce the work you do. AAABench sits outside the agent's workflow and tries to measure what the agent can do alone. One is a productivity layer; the other is an evaluation harness with a rule forbidding assistance.

The concrete difference is in the control surface and the success condition. A copilot typically exposes a chat panel or node suggestions and is judged by whether the human ships faster. AAABench drives the editor over MCP, hands the agent `PROMPT.md`, and judges the result by whether a stranger looking at the world finds it plausible and whether the game opens like a game. The README's stated bar is not that a level loads. It is a place that survives a stranger looking at it. There is no autocomplete in that loop, and no human in it either.

Licence, maintenance and the cost of running it

AAABench is MIT licensed, which is permissive and imposes no obligation on what you build with the harness. The licence covers the harness. It does not cover Unreal Engine, and it does not cover the assets, mocap, map data or rendering sources the handbook points at; those carry their own terms, and the README does not summarise them. If you plan to publish anything produced by a run, check those sources individually rather than assuming the MIT badge extends to them.

On maintenance, the last push to the default branch was on 2026-08-14. The repository is not archived. There are no retrieved releases, so there is no versioned artefact to pin and no changelog to read; you track `main`. Upgrading means pulling `main` and re-running `./bin/setup-capabilities.sh`, which is also the step most likely to break against a new engine version, since it installs plugins, renderer features and Python libraries. The handbook's coverage of the engine's version traps suggests the author has been bitten by this already. Budget for that re-run, and expect the cost of a run to be dominated by GPU time and wall-clock hours rather than by the harness itself.

Editorial conclusion

Adopt AAABench if you already run Unreal Engine 5 headlessly or on a workstation and want an evaluation whose output a stranger can look at and judge without expertise. Do not adopt it if you need a scored, reproducible number today: the repository is the harness, and the README states plainly that it contains no results. Before you commit GPU time, verify that bin/setup-capabilities.sh resolves the plugins, renderer features and Python libraries on your engine version, that MCP connects reliably, and that your agent can actually see its own viewport through tools/ue_qa.py. If the agent cannot look at what it made, the self-verification half of the benchmark is not being measured at all.

Frequently asked questions

Official sources

1. Issues
2. License: MIT
3. Project website
4. README
5. ukanwat/aaabench on GitHub