arc-task-gen: generating private ARC-AGI-1-style tasks when the public eval set is too familiar

The problem arc-task-gen addresses: benchmark familiarity, not benchmark difficulty

ARC-AGI-1 is a public benchmark. Anything public can be memorised, and the README states the issue plainly: as a public benchmark, ARC-AGI-1 cannot fully isolate few-shot rule induction from potential prior familiarity with its tasks. That is a measurement problem rather than a difficulty problem. A model can score well because it reasons about the demonstrations, or because it has seen the grids before, and the public set alone cannot separate the two.

The repository's answer is a complementary measure. arc-task-gen creates a private evaluation set with similar properties, which the README frames as enabling meaningful comparisons between public-benchmark performance and performance on newly generated tasks. The audience is narrow and specific: people evaluating frontier models who already have an ARC harness and want a second set of tasks that behaves like the first one. If you are not running ARC evaluations, this repository has little to offer you.

What the repository actually contains, and what it does not

The top level holds four Python entry points: generate_tasks.py, generate_tasks_stratified.py, describe_eval_tasks.py, label_eval_tasks.py, plus visualize_tasks.py. There is an instructions.md, a pixi.toml and a pixi.lock, a LICENSE, an assets directory, and the README. The naming suggests a pipeline: describe and label the public eval tasks, generate new ones (optionally stratified), then visualise the result. The README does not document that pipeline step by step, so treat the file names as the map and instructions.md as the manual.

The README does not describe a Python API, a command line interface, or a hosted service. There is no documented way to check that a generated task is solvable or that it matches the distribution beyond the claim in the description. The README also does not document rollback or regeneration of a specific task. Those are gaps you should plan around rather than assume away.

Installing arc-task-gen and generating a first tasks.json

The README points at instructions.md for the actual steps and does not repeat them, so the commands below are the ones the repository layout makes visible rather than a documented quickstart. The project pins its environment with pixi, so the lock file is the intended entry point.

pixi install

After that, the generator script is the thing to run. The README does not print its flags, so check instructions.md before adding any.

pixi run python generate_tasks.py

The output is a tasks.json file. The README gives its shape explicitly: a JSON object with a train array and a test array, matching the standard ARC format.

{
  "train": [],
  "test": []
}

Because the README states the file is compatible with existing ARC evaluation harnesses, the first real use is to point your harness at the generated file instead of the public eval set and compare the two scores. If your harness expects a different on-disk layout, that mismatch will surface immediately, before you spend time on a full run.

Distribution matching is the whole claim, and it is also the weak point

The description says the generated tasks are distribution-matched to the public eval set. That is the property everything else depends on. If the generated tasks drift from the public distribution, a score gap between the two sets tells you about the drift, not about the model. The repository offers generate_tasks_stratified.py alongside the plain generator, which suggests stratification is the mechanism for keeping that match, but the README does not explain the strata or how they are derived.

The real limitation is verifiability. The README does not document a solver, a validity check, or a held-out comparison between generated and public task statistics. You can inspect the output with visualize_tasks.py, and you can describe and label the public tasks with describe_eval_tasks.py and label_eval_tasks.py, but the README does not say how those labels feed back into generation. If you need a generator whose distribution match you can audit from the documentation alone, this is not it.

How this differs from using ARC-DSL to build tasks

ARC-DSL is the better-known route to synthetic ARC tasks. It is a domain-specific language for expressing ARC transformations, and a generator built on it composes primitives from that language. The difference is where the constraints live. With a DSL, the grammar bounds what a task can be, and the task's structure follows from the program that produced it.

arc-task-gen takes a different route. Its stated goal is matching the distribution of the public eval set, and the repository is organised around describing and labelling the existing eval tasks before generating new ones. That is an empirical approach: measure the reference set, then sample against it. It gives you less formal control over what a single task means and more direct control over how the output set compares to the public one. Choose the DSL route if you want to reason about individual tasks as programs. Choose this route if your concern is the aggregate resemblance between two evaluation sets.

Maintenance, licence and what upgrading costs you

The repository is not archived, and the last push was on 2026-08-11, which is recent enough that the code is not stale. There are no retrieved releases, so there is no version number to pin against and no changelog to read before upgrading. In practice that means an upgrade is a git pull plus a pixi install, and the pixi.lock file is what keeps the environment reproducible when you do it.

The licence is MIT. That permits commercial and private use and modification, and it requires that the copyright notice and permission notice be included in copies or substantial portions. It comes with no warranty. This is a description of the licence text, not legal advice; if the generated tasks end up inside a product, have someone qualified read the terms.

One practical cost: because there are no releases, you cannot express a dependency on a specific version of arc-task-gen. If you need reproducibility across machines, pin the commit hash yourself.

Editorial conclusion

Use arc-task-gen if you already run an ARC evaluation harness and want a private set of tasks that follows the same format, because the README is explicit that the public benchmark cannot fully isolate few-shot rule induction from prior familiarity with its tasks. Do not adopt it if you need a hosted service, a packaged Python API, or a documented way to verify that a generated task is solvable, since the README does not describe any of those. Before committing, read instructions.md, run generate_tasks.py once and inspect the tasks.json it writes, and confirm your harness accepts the train and test structure the README shows.

Frequently asked questions

Official sources

1. Issues
2. License: MIT
3. pathwaycom/arc-task-gen on GitHub
4. README