GameHorizon measures model gameplay ability across short, medium and long horizons

Three components: an annotator, a corpus and a benchmark

The overview describes three key components, and the project page breaks them down in a table. GameHorizon-Annotator is a scalable and automated annotation pipeline whose highlight is the L1 to L2 to L3 instruction pyramid, meaning it automatically produces a three-level set of short-horizon operations, medium-horizon goals and long-horizon strategies from gameplay. GameHorizon-Data is the corpus, described as a large-scale AAA gameplay corpus with 5,000 hours, AAA focus, 21 titles and temporally-aligned videos, actions and multi-horizon instructions. GameHorizon-Bench is the evaluation side, a reproducible offline and stepwise online track with 5,000 offline MCQs across three primary and ten variant tasks, 20 online tasks split into 62 subtasks, and 47 models benchmarked. The design intent is a unified yardstick across a broad range of model types, which is the gap the project says it addresses, since existing photorealistic-style gameplay benchmarks tend to test one horizon or one model family.

Horizon is the axis, and the pyramid is the mechanism

The idea worth understanding is that difficulty in gameplay is not one dimension. A model can recognise a hazard and dodge it, and that is a different capability from deciding over a two-minute window that the hazard is worth detouring around, and both are different from adopting a strategy that holds for the whole match. The annotator handles this by generating all three levels from the same gameplay rather than by collecting separate datasets per level, which is what makes the pyramid automatic and scalable. GameHorizon-Data then pairs temporally-aligned video, the action sequence and the multi-horizon instructions, so a training or evaluation pipeline can work from one aligned record rather than joining three sources itself. The design bet is that horizon becomes a measurable axis of model capability, so a leaderboard can say not just that a model plays games well but that it is strong at short horizons and weak at long ones, which is a far more actionable statement for anyone building an agent.

The offline track is thousands of MCQs, and the online track is stepwise

The two benchmark tracks test different things and the readme is explicit about how they differ. The offline track contains thousands of standardised multiple-choice questions across three primary tasks with diagnostic variants, and the leaderboard reports T1, T2 and T3 accuracies as percentages with Overall as the mean across the three. The online track tests order-dependent causal and order-flexible thematic tasks through the verifiable subtasks used for failure localisation, which is the more interesting engineering idea in the suite. If a task is composed of subtasks that can each be checked, then a failed episode tells you which step went wrong rather than only that the episode failed, and that is the difference between a score you can act on and a score you can only rank models by. The stepwise nature of the online track also means a partial result is meaningful, which matters for the long-horizon tasks where an agent that succeeds in the first minute and fails in the tenth has told you something specific.

The leaderboard groups models into four tiers by overall score

The leaderboard section reports accuracies as percentages with Overall as the mean across the three primary tasks, marking the best result in bold and the second best underlined, and grouping models into four tiers by overall score. The average line for the whole field is given at the top, 44 models in that average, with T1 at 57.3, T2 at 65.1, T3 at 71.6 and an overall of 64.7. The top of the table shows GPT-6-Astra at rank one with 69.4 on T1, 79.6 on T2, 91.5 on T3 and 80.2 overall, followed by Gemini 3.8 Flash with 77.3 overall and Gemini 3.7 Flash with 76.7, then Gemini 3.6 Flash with 75.3. Two things stand out in those numbers. The spread between the tiers on T1 is much wider than on T3, which suggests the shortest horizon is where models differ most, and the field average of 57.3 on T1 against 71.6 on T3 says the easiest task is still one that a typical model gets wrong nearly half the time. The benchmark table separately notes 47 models benchmarked, so the average line covers a subset.

As of the release, the code, benchmark and data are still on the way

This is the fact a reader has to plan around. The open-source plan is a checklist in the readme with three items marked done and four still open. Done are the project page, the technical report or paper, and the public leaderboard. Still to come are the GameHorizon-Annotator annotation toolkit and pipeline, GameHorizon-Bench offline with its MCQ suite and evaluation code, GameHorizon-Bench online with its online task environments, and GameHorizon-Data, the 5,000 hour AAA gameplay corpus. The news section states that the code, benchmark and data will be progressively released starting around 2026.10.25, and the badges in the header label the benchmark and the dataset as coming soon. The repository itself has only three top level entries, a licence, a readme and an assets directory, so there is nothing to clone and run yet. The paper and the project page are the usable artefacts today, and the leaderboard is live.

Where the data comes from shapes what you can ask of it

The corpus definition constrains the evaluation, and the numbers are specific enough to reason about. Five thousand hours of gameplay across 21 game titles, which is a small set of titles for a large amount of content, so the coverage per title is substantial while the breadth across genres is bounded by which titles were recorded. The titles are AAA games, which is deliberate, since the point is to measure the kind of gameplay that a commercial game asks for rather than the simplified mechanics of a training environment. The records are temporally aligned, so video, actions and instructions line up in time, and the instructions are multi-horizon, which is what lets the same recording feed the annotator at all three levels. A corpus of this shape suits training and evaluation of agents that watch and act, and it is the wrong shape for a benchmark that needs breadth across many genres or a large number of distinct game mechanics, and it is also the wrong shape for anything that needs clean, machine-generated action labels rather than human play.

Where GameHorizon is the wrong tool

The first wrong case is a team that needs to run the evaluation today, because the benchmark code, the offline evaluation code and the data are all listed as unreleased, and the release plan is a date rather than an artefact you can download. The second wrong case is a project working on a single game, since a 21 title AAA corpus is a general benchmark and a model tuned for one title with one action space would be measured against a set it was never aimed at. The third is a team that needs annotations for its own proprietary gameplay, since the annotator is designed to produce the L1 to L2 to L3 pyramid from the corpus, and the documentation does not describe a general annotation tool for arbitrary footage. The fourth is anyone building an online agent that must finish a task rather than answer a question, since the offline track is multiple choice and the online track is composed of verifiable subtasks, which is a proxy for task completion rather than the real thing. The fifth is a team that needs a licence for commercial use of the corpus itself, since the repository carries an Apache-2.0 licence while the video data comes from commercial titles and the terms for using captured gameplay are a separate question that the readme does not address.

Editorial conclusion

Researchers who need to compare models on gameplay rather than on static question answering, and who care about the difference between reacting quickly and planning over minutes, are the people GameHorizon is for, and the three-level instruction pyramid is the part that makes it more than a video question answering set. It is the wrong tool for anyone who needs to run the benchmark today, since the code, the benchmark and the data are all listed as still to be released, with the release plan starting around 2026.10.25, and the repository itself contains only a licence, a readme and assets. Check three things before you build on it: that the data release has actually landed by the time you read this, that the horizon definition matches your task, since the annotator produces short-horizon operations, medium-horizon goals and long-horizon strategies as separate levels, and that the online track's design is what you want, since it tests order-dependent causal tasks and order-flexible thematic tasks through verifiable subtasks used for failure localisation. The licence is Apache-2.0, the paper and project page were released on 2026.09.22, and the last push was on 2026-09-22.

Frequently asked questions

Official sources

1. Issues
2. License: Apache-2.0
3. README
4. TencentARC/GameHorizon on GitHub