GameForgeBench: a terminal benchmark for coding agents that repair real game projects

The gap GameForgeBench targets: exit codes are not game behaviour

Most terminal benchmarks for coding agents end at the shell. A patch applies, a test suite runs, exit code zero means success. The README states the problem plainly: interactive software has a second truth, because the project must import, launch, accept controlled input, change runtime state, and produce the intended behaviour. A Godot scene that parses cleanly can still ship a camera that drags in the wrong direction, or a grenade whose burn pool never extinguishes.

The project positions itself as Terminal-Bench for interactive game-engine development agents. That framing matters for who should care. This is not a tool for game developers looking for an engine or a template. It is infrastructure for people who build, evaluate or train coding agents, and who need a task where the agent has to run the thing it just edited. The README is explicit that the benchmark target is a coding agent, not a player bot: evaluator-controlled probes supply runtime evidence, and the agent is judged on engineering work, debugging, behaviour and regression resistance.

How scoring works: deterministic probes, no LLM judge in the reward path

The loop the README prints is instruction, inspect, edit, run, observe, diagnose, repair, verify. Reward comes from task-specific runtime behaviour checked by what the README calls a deterministic native probe. File presence and project parse are supporting gates only; on their own they cannot produce a passing reward. The score is binary, 1 when all required behaviour checks pass and 0 otherwise, with an optional post-hoc diagnostic score that does not feed the reward.

That design is the most consequential thing about the project. It removes the judge model from the scoring path, which makes runs reproducible and cheap to re-score, but it also means every task needs a probe that can observe the behaviour in question. The repository layout reflects that cost: each task carries tests/test.sh plus runtime probes and fixtures, and a solution/ reference implementation used to prove solvability and never shown to the agent. A task without a hardened verifier cannot ship, which explains why the public list is shorter than the candidate corpus.

Task anatomy: task.toml, instruction.md, environment, tests, solution

Every contributed task follows one portable directory shape. task.toml holds metadata, schema version, timeouts and resource limits. instruction.md gives the starting condition, requested outcome, constraints and acceptance criteria without revealing the patch. environment/ pins the engine, operating system packages, project dependencies and a clean runtime entrypoint, with a Dockerfile. tests/ is benchmark-owned verification executed outside the candidate implementation. solution/ holds the reference implementation.

The README adds one constraint worth repeating: tasks must not depend on a contributor's local absolute paths. The truncated README cuts off mid-sentence at that point, so the full portability rules are not visible here. The released task list shows the breadth this shape supports: Godot tasks such as godot-replay-orbit-camera (fix drag direction, make rotation sensitivity DPI-independent) and godot-endless-chunks (deterministic chunk generation with spike hazards), Paper 1.20.4 server plugins such as minecraft-paper-settings-i18n with en_us/ru_ru switching backed by property files, plus cpp-shadow-map-pass and html5-cyberpunk-engine.

Installing the runner and validating a first task

The package is published as @gamephanes/core version 0.1.0, with a bin entry named gamephanes pointing at ./bin/gamephanes.js. The package declares "type": "module", so it expects a Node environment that can run ESM. Installing it globally gives you the CLI on your path:

npm install -g @gamephanes/core
gamephanes doctor

The doctor subcommand is the first thing to run. It checks the local environment before you spend time on a task. If you would rather not install globally, the package.json scripts call the binary directly with node:

node ./bin/gamephanes.js doctor
node ./bin/gamephanes.js validate ./benchmark/tasks/platformer-basic.json

validate takes a task JSON path and checks the task definition. The demo script then runs that same task end to end:

node ./bin/gamephanes.js run ./benchmark/tasks/platformer-basic.json

There is also an asset manifest path, used by the assets:validate and assets:list scripts against ./assets/manifest.json, and a showcase:run script that chains six task files (neon-relay, last-signal, gravity-lab, tiny-bastion, rift-arena, starfall-protocol) with &&. Note that the benchmark/ task files in package.json and the release/tasks/ directories in the README are two different paths; do not assume a task listed in the README table is present under benchmark/tasks/.

Where GameForgeBench is the wrong tool

The public corpus is the first limitation. The README describes 81 normalized executable candidates across Godot, Unity, Roblox, Minecraft, Unreal, Web and generic engine projects, then says tasks are released in batches as their Docker verifiers and oracle/no-op controls are hardened. The Released Tasks table lists seventeen tasks. Unity, Roblox Luau and Unreal appear only in the coming-soon sentence, so anyone who needs those engines today has nothing to run.

Binary reward is the second trade-off. A task either passes every required behaviour check or scores zero, so an agent that fixes four of five behaviours gets no partial credit in the reward, only the optional diagnostic score. That is good for comparability and bad for measuring incremental progress during training.

Third, this is a repair benchmark. The README contrasts it with generative work: every task is an existing-project repair inside a real engine. If your goal is to measure whether an agent can build a Godot project from a theme keyword, this benchmark does not answer that question. And because verification is Docker-based with pinned engines, the compute footprint per task is closer to a full engine run than to a unit test suite. The README does not document rollback for a failed or interrupted run.

How it compares with SWE-bench, JAMER and GameCraft-Bench

The README draws the contrasts itself. SWE-bench treats a resolved GitHub issue as the task and checks whether the repository's existing test suite passes after the patch. Reward is binary, the environment is a layered Docker image built from the original repo, and there is no engine runtime. GameForgeBench keeps the binary reward but moves the check from the test suite to a native runtime probe.

JAMER (arxiv 2606.19830) generates Godot projects from theme keywords and measures structural completeness and behavioural alignment through headless Godot runs, using Claude Code as the agent harness with model swapping. Its metrics are continuous rather than binary, and its tasks are generative rather than repair. If you want a continuous signal on generation, JAMER is the closer fit; if you want a pass/fail signal on repair inside a running engine, GameForgeBench is.

GameCraft-Bench (arxiv 2606.17861) covers 140 Godot tasks across 15 game families and judges output by replaying recorded gameplay through a multimodal LLM rubric judge. The README notes the top frontier agent reached 41.5% on that scale. The difference is the judge: GameCraft-Bench scores with a model, GameForgeBench scores with a probe. That makes GameForgeBench cheaper to re-score and harder to build for, since every behaviour needs an executable check.

Maintenance, licence and contribution cost

The repository is not archived. The last push was on 2026-09-12, five days before this writing, so the project is being worked on. There are no retrieved releases, so version 0.1.0 in package.json is the only version signal available, and the README's repeated coming-soon language suggests the task corpus is still filling in rather than frozen.

The licence is MIT, and the LICENSE file sits at the repository root alongside CITATION.cff and CONTRIBUTING.md. MIT is permissive, so reuse of the runner and task definitions is straightforward, but a benchmark has a second rights question that a licence file does not settle: the starter projects and fixtures bundled into each task may carry their own provenance. The README does not address asset licensing, so check the individual task directory before redistributing a corpus. This is not legal advice; it is a gap in the documentation.

The upgrade cost is real if you write your own tasks. A task is not a prompt plus a repo. It needs a Dockerfile with a pinned engine, a probe that observes the target behaviour, and a reference solution that proves the task is solvable. Contributing a Godot task means being able to write a headless check for something like DPI-independent camera sensitivity, which is a different skill from writing the fix itself.

Editorial conclusion

Adopt GameForgeBench if you are evaluating or training a coding agent that has to operate a real engine runtime, and you want a binary, probe-driven reward rather than an LLM judge. Do not adopt it if you need a large, stable task corpus today: the README describes 81 normalized executable candidates, but only the tasks listed in the Released Tasks table are public, and the rest are described as coming in batches. Before running anything, check the doctor output and the resource limits in task.toml, since the README does not document rollback or failure recovery for a benchmark run.

Frequently asked questions

Official sources

1. GamePhanesStudio/GamePhanes on GitHub
2. Issues
3. License: MIT
4. Project website
5. README