MineBench: a Minecraft-style voxel benchmark for comparing AI models

What MineBench measures that a text benchmark cannot

The README is explicit that MineBench is not technically a benchmark, because it has no objectively correct answers. It is described as a take on the LMSYS Chatbot Arena. That framing matters more than the name. A model is given a natural-language prompt such as a medieval castle with four towers, and must return raw 3D coordinates as JSON. There is no reference build to diff against, no pass or fail. What you get is a rendered voxel object and, over many votes, a ranking.

The audience follows from that. The README names two groups: people who want the general feel or vibe of a model, and AI labs that want to privately A/B test model checkpoints. If you need to know whether a checkpoint regressed on spatial construction before you ship it, the private comparison path is the intended use. If you need a number to put in a paper, this is the wrong instrument, and the project says so itself.

The README also makes a claim that is worth reading carefully. It states that spatial reasoning of this kind correlates strongly with a model's raw general intelligence, and that the leaderboard tracks the same hierarchy people observe in real-world usage. That is an anecdotal observation by the author, not a measured correlation, and the word the README uses is anecdotally. Treat the leaderboard as a signal about build quality, not as a validated proxy for general capability.

Coordinates, the voxel.exec tool, and where the votes come from

Two generation paths exist. In the plain path, the model emits JSON containing block coordinates directly. The README notes this is limited by token budget. In tool mode, the model calls voxel.exec with three primitives: block, box, and line. Building from primitives lets a model produce large structures that would not fit in a token-only JSON response, because a box call describes a volume rather than every coordinate inside it.

Ranking does not come from inspecting the build against a spec. It comes from blind head-to-head votes, aggregated with a global Bradley-Terry model that carries uncertainty intervals. The Arena feature presents pre-generated builds side by side; the Sandbox lets you compare existing builds, generate new ones, or import output from any model. The repository layout backs this up: app/ holds the Next.js routes, prisma/ holds the schema, lib/ holds shared logic, and docs/arena-ranking-system.md is the file that documents the ranking model.

There is a detail in the environment template that says something about the Arena's design. ARENA_MATCHUP_SIGNING_SECRET is described as an optional dedicated secret for encrypted arena matchup and vote tokens, and the comment states that if it is unset, ADMIN_TOKEN or NEXTAUTH_SECRET is required. Matchup and vote tokens are signed, which is the mechanism that keeps a voter from knowing which model produced which build before they vote.

The ranking is only as meaningful as the prompt pool and the voter pool. The README's own FAQ index includes entries asking how the Gallery shapes the benchmark and how nondeterminism is accounted for, which is a sign the author treats those as open methodology questions rather than settled ones.

Installing MineBench locally and generating your first build

The repository is a pnpm workspace running Next.js with Prisma over Postgres. The local database is defined in compose.yaml as postgres:16, published on host port 54327 with user, password and database all set to minebench. The .env.example already points DATABASE_URL and DIRECT_URL at that same port, so the defaults line up without editing.

The package.json exposes a single setup script that chains the whole first run. It ensures the env file, builds the texture atlas, resets the database, waits for it to accept connections, applies migrations, and starts the dev server.

pnpm install
pnpm dev:setup

According to the script definition, dev:setup runs env:ensure, atlas, db:reset, db:wait, prisma:migrate and dev in that order. The database reset is destructive by design: db:reset is defined as docker compose down -v followed by docker compose up -d, which drops the Postgres volume. Do not run it against a database you care about.

For day-to-day work after the first setup, the database and the app are started separately, and there is a variant that injects local environment values before launching Next.js.

pnpm db:up
pnpm local:dev

Generation needs at least one provider key. The .env.example lists direct keys for OpenAI, Anthropic, Google, Moonshot, DeepSeek, MiniMax, xAI, Meta and Z.AI, plus OPENROUTER_API_KEY as an alternative or fallback. The comment states that if a direct provider key is missing or fails, OpenRouter is used as fallback. There is also a script entry for a local prompt run, local:prompt, which wraps tsx scripts/prompt.ts with the local environment. The README does not document what that script prints, so treat it as a starting point to read rather than a documented interface.

The build size ceiling and the token-only failure mode

The clearest limitation is stated in the README rather than discovered: models producing raw JSON coordinates hit token limits, which is exactly why tool mode exists. That means a comparison between two models is not purely a comparison of spatial reasoning. It is also a comparison of how each model chooses to spend output tokens, and whether it uses voxel.exec at all. A model that never calls the tool and tries to enumerate blocks by hand will look worse on a large prompt for reasons that have nothing to do with geometry.

Grid size and block limits are configurable, and the FAQ index has a dedicated entry asking how grid size, block limits and different leaderboard settings work. That is a real methodological variable. If two runs use different limits, the resulting builds are not comparable, and the ranking model has no way to know that.

Contamination is another acknowledged risk. The FAQ index includes an entry asking whether models can train on MineBench or benchmax it, and the README separately discusses models that are clearly bench-maxed. A public prompt pool that appears in a Gallery is, by construction, visible to anyone training a model. The project acknowledges the question; the README does not describe a mitigation.

Finally, this is a hosted-API benchmark. Every generation path described in the environment template goes through a provider key or OpenRouter. There is no documented offline mode, so evaluation cost and rate limits are part of the workflow whether you want them or not.

How MineBench differs from text benchmarks and from Minecraft agents

The natural comparison is a standard text benchmark. Those score answers against a reference, which makes them reproducible and cheap to automate, but they say nothing about whether a model can hold a 3D structure in mind while emitting thousands of coordinates. MineBench inverts the trade: it measures something text benchmarks miss, and in exchange it gives up ground truth. The ranking is a preference aggregate, so it moves with the voter pool.

The second comparison is with Minecraft agents that drive the game through a protocol. The README's FAQ index has an entry asking directly whether MineBench uses Minecraft MCP, Blender MCP, or a coding agent. The answer implied by the architecture is no: the model never touches a game client. It emits coordinates or calls voxel.exec, and MineBench renders the result itself. That removes the whole class of failures around game control, pathing and inventory, and it also means MineBench cannot tell you anything about whether a model can operate inside a live world.

A third difference is the export path. Builds can be saved as GLB, STL, MagicaVoxel .vox, or WorldEdit .schem, which means output can be moved into Blender, a voxel editor, or an actual Minecraft server. That is a practical advantage over a benchmark that ends at a score, and it is the reason importing output from any model is a supported Sandbox workflow.

Licence, maintenance and upgrade cost

The repository is MIT licensed, which permits commercial use, modification and redistribution provided the copyright notice and permission notice are retained. That is a permissive licence and it does not attach copyleft obligations to your own code. It says nothing about the terms of the AI providers you connect it to, and those are separate agreements you accept by setting the API keys in .env.example. Nothing here is legal advice.

The last push to the master branch was on 2026-09-15, and the most recent releases are 4.5.0 on 2026-09-10, 4.4.0 on 2026-09-09 and 4.3.0 on 2026-09-05. Release cadence is high, and the release titles name specific models and features, which means the leaderboard population changes often. The repository is not archived.

That cadence is the upgrade cost. A Prisma schema lives in prisma/, migrations are applied with prisma migrate deploy, and the setup script resets the database, so schema changes are part of normal operation rather than an exception. If you run your own instance to A/B test checkpoints, pin to a release tag and read the migration before applying it. If you rely on the public leaderboard, remember that a ranking captured today is tied to the model set and prompt pool of that day.

Editorial conclusion

Adopt MineBench if you want a visual, vote-driven read on how a model handles 3D spatial prompts and you are willing to run Postgres plus at least one provider API key. Skip it if you need a scored benchmark with a ground truth, a single reproducible number per model, or an offline evaluation that never calls a hosted API. Before you commit, check whether the model you care about is on the official leaderboard, confirm from docs/arena-ranking-system.md how the Bradley-Terry intervals are computed, and verify at minebench.ai/faq whether the grid size and block limit you plan to use match the settings the public leaderboard runs.

Frequently asked questions

Official sources

1. Ammaar-Alam/minebench on GitHub
2. License: MIT
3. Project website
4. README
5. Releases