arc-task-gen: a second ARC-AGI-1 eval set, written by five Python scripts

Why a second ARC-AGI-1 set is needed at all

The stated problem is contamination, expressed in one sentence: as a public benchmark, ARC-AGI-1 cannot fully isolate few-shot rule induction from potential prior familiarity with its tasks.

So the repository builds a complement rather than a replacement. Its description calls the output original ARC-AGI-1-style tasks distribution-matched to the public eval set, and the README says the generated tasks resemble the public dataset closely enough to be useful for evaluating models such as BDH-CQ on problems they are unlikely to have encountered before.

The point is comparability. Because the private set is built to have similar properties, a score on it can sit next to a public-benchmark score and mean something: if a model holds up on tasks generated after the fact, its rule induction is doing the work rather than its memory of the originals. That inference is the whole reason to spend effort on a generator.

The entire output contract is a two-key tasks.json

One file comes out of the pipeline, and its shape is shown in full:

{
  "train": [],
  "test": []
}

Two keys, train and test, and the README's claim is that this follows the standard ARC format and stays compatible with existing ARC evaluation harnesses.

What that claim rests on is the harnesses, not this page. The example is two empty arrays, so nothing here defines a single training pair: no key names for input and output grids, no statement of grid dimensions, no count of demonstration pairs a generated task should carry. If your evaluator is a standard ARC harness, the shape is enough to start. If you are writing your own loader, the README gives no field-level specification, so check instructions.md and the scripts.

Five Python scripts and a pixi environment, no packaging metadata

The repository root is flat and small: generate_tasks.py, generate_tasks_stratified.py, describe_eval_tasks.py, label_eval_tasks.py, visualize_tasks.py, instructions.md, and an assets/ directory. Alongside them sit pixi.toml and pixi.lock, which is how the environment is described.

There is no requirements.txt, no pyproject.toml and no setup.py. So there is no declared dependency set that pip or conda would read, and nothing to install this project from: the scripts are run from a checkout, inside whatever pixi.toml pins. That is a workable arrangement for a research artifact attached to a paper, and a poor one for a tool you want to drop into another project's dependencies.

For the three inspection scripts, the filenames describe the intent and nothing else does. describe_eval_tasks.py, label_eval_tasks.py and visualize_tasks.py imply a workflow of describing, labelling and visualizing tasks, but the README documents no argument, no output path and no flag for any of them.

Two generators exist, and the README explains neither

generate_tasks.py and generate_tasks_stratified.py sit side by side, which tells you the project has more than one generation strategy. The name of the second one points at stratification, and the description calls the result distribution-matched. Beyond that, the README stops.

No flag, no seed argument, no sample count, no statement about what is being stratified or against which reference distribution. There is no example invocation anywhere on the page.

That is the sharpest limitation for anyone planning to use this: the generator is where all the substance sits, and the project page describes none of its interface. The moment you need reproducible task sets, for a paper or for a regression test, you are reading instructions.md or the script source, not the documentation. Budget for that step instead of assuming a CLI will announce itself.

Most of the page describes the model, not the generator

Read the section order and the priorities are plain. The Overview spends its paragraphs on BDH-CQ, a reasoning model that combines in-context learning through evolving recurrent memory with iterative reasoning in a structured, continuous latent space, and on the Dragon Hatchling architecture it extends.

The numbers there belong to the model too. The README states that a 150M-parameter configuration reaches 29.5% pass@2 on the public ARC-AGI-1 evaluation set at a computed inference cost of $0.0007 per task, and puts that at 11× cheaper per task than GPT-5.6 Luna (Low) after an 80% price reduction on July 30. Pretraining experiments are reported from 1B to 600B parameters.

None of that tells you how a task gets generated. A reader who arrives wanting the tool has to get past the model write-up before reaching the section called What You'll Find in This Repository, which is only five short sentences long.

Two unfinished spots left in the README

There is a heading, ARC-AGI-1 Efficiency Frontier, with nothing beneath it. It is the section that would have carried the cost comparison the Overview quotes in passing, and it is empty in the published page.

Then there are the comments. One says to add the BDH-CQ arXiv link when available. A second says to replace the temporary paper link above with the arXiv URL when available, while the visible line above it already points at arxiv.org/abs/2608.09888, credited to B. Engdahl, A. Kosowski, J. Chorowski, Z. Stamirowska, P. Uznański, J. Jiang, R. Phadke, R. Kinas, and R. Zhong.

So the page is a work in progress with a mid-edit citation. Treat the figures as a snapshot of one moment rather than a settled claim, and check the paper itself before citing either the number or the link.

Reproduction is asserted by name, not by a test suite

The Independent Reproduction section names the people who ran the evaluation again: Łukasz Kaiser, described as a co-author of the Transformer architecture and TensorFlow, and Remigiusz Kinas, a contributor to Bielik, together with Richard Zhong, an NYU researcher. Their focus is given as model evaluation and benchmark robustness.

Nominating reviewers is worth something, and it is the only verification evidence this page offers. Look at the top-level entries and there is no CI configuration, no test directory and no workflow file. The five Python scripts are tools for generating and inspecting tasks, not a suite that asserts the generator is correct.

The practical consequence: the repository contains no check that the generated tasks are well-formed ARC pairs or that the distribution match holds. If task validity matters to your evaluation, verify a sample yourself before trusting a full run.

MIT licensed, with no releases to pin

Licensing is the simplest part: MIT, with the LICENSE file at the repository root. There is no separate third-party licence ledger and no homepage field, so there is nowhere else the project points you.

Versioning is thinner. The repository has no GitHub releases at all, and the last push was on 2026-08-11. That is normal for an artifact published alongside a paper, and it changes what reproducibility means in practice: you cannot say which version produced a task set, because no version was cut. You pin a commit, and a commit hash is a poor handle in a write-up.

There is also no install target and no library API. Everything here is a script you run yourself, which keeps the dependency surface at whatever pixi.toml pins and leaves the arc-task-gen name attached to a repository rather than to something in your environment.

Editorial conclusion

Use arc-task-gen when you need to check whether a reasoning model generalises rather than recalls, and when you already have a harness that reads the standard ARC shape, since that is all the output contract this repository spells out. Do not reach for it as an installable package: there are no GitHub releases, the last push was on 2026-08-11, and nothing in the README names a single command. Before you generate anything, read instructions.md, which the README names as the place to start generating tasks, and check the paper at arxiv.org/abs/2608.09888, because the README's own comment calls its citation link temporary.

Frequently asked questions

Official sources

1. Issues
2. License: MIT
3. pathwaycom/arc-task-gen on GitHub
4. README