# automlbenchmark exists because task difficulty is not a constant

> OpenML's AutoML Benchmark is a harness for comparing open source AutoML systems on curated regression and classification suites, and its founding argument is narrower than a leaderboard: subtle differences in how a search space or a time budget is defined can change a task's difficulty enough to make two papers incomparable. It is MIT licensed, pinned to Python 3.9, and last released in September 2023.

**openml/automlbenchmark** — OpenML AutoML Benchmarking Framework

- Repository: https://github.com/openml/automlbenchmark
- Website: https://openml.github.io/automlbenchmark
- Stars: 470 · Forks: 151
- Language: Python
- License: MIT
- Published: 2026-09-10 · Updated: 2026-09-10 · Language: en
- Canonical page: https://hysenlabs.com/projects/openml-automlbenchmark

## The second problem is the search space, not the dataset

The stated motivation has two halves and the second is the interesting one. The first is familiar: there is no standardised, easily accessible benchmarking suite of tasks that is simultaneously curated to reflect important problem domains, practical to use, and difficult enough to support a rigorous analysis. Those three criteria pull against each other, and most suites satisfy one at the expense of the others. The second half is about problem definition rather than problem selection. Subtle differences in how a task is defined, and the two examples given are the design of the hyperparameter search space and the way time budgets are defined, can drastically alter a task's difficulty. That is what makes published AutoML research hard to reproduce and results hard to compare across papers, and it is a reproducibility problem rather than a leaderboard problem. A benchmark that fixes both the tasks and the definitions turns an argument about systems into an argument about measurements, which is the whole design intent stated at the end of that section: standardized environments for in-depth experimentation.

## The suites are OpenML identifiers, so the data moves under you

The dataset story is deliberately thin, and that is a design decision rather than an omission. Rather than shipping corpora, the framework points at two curated suites hosted on OpenML: one for regression and one for classification, each identified by a suite number you reference rather than a folder you download. That keeps the repository small and means the benchmark is measuring current OpenML rather than a snapshot of what was current when the code was written. The trade is that your results are only reproducible against the state of those suites at your run time, which is a meaningful caveat for a project whose entire argument is about reproducibility. The other half of extensibility is the same in shape: new AutoML systems can be added, and the framework ships code to benchmark a number of popular systems on both regression and classification tasks already. So the extension point is the system under test rather than the data, and a new framework arrives as code and configuration rather than as a new dataset pipeline.

## Containers everywhere, and results that leave the machine

Execution is handled in two layers and both matter for a benchmark. Experiments can run in Docker or in Singularity containers, which is the mechanism that makes a comparison fair at all: each framework gets an isolated environment with its own pinned dependencies, so a framework does not win because your environment happened to suit it. On top of that, experiments execute either locally or on AWS, which is how a suite of tasks gets spread across machines without rewriting the harness. The dependency list shows what that costs. It includes boto3 and botocore for AWS, plus minio, s3fs and s3transfer for object storage, so results do not have to sit on the machine that produced them. The pinned set also carries the science stack the harnessed frameworks need, with scikit-learn, numpy, pandas, scipy, joblib and threadpoolctl, and the OpenML client itself at 0.13.1. `requirements.txt` is autogenerated by pip-compile for Python 3.9, with every version pinned and annotated by what pulled it in, which is what you want from a benchmark environment and is the opposite of what you want in an application.

## Python 3.9 is pinned in the tooling, not just the requirements

The target interpreter is fixed in three places and they agree. The packaging configuration sets a lint target of 3.9, the type checker is configured for a 3.9 language version, and the requirements file states it was generated by pip-compile with Python 3.9. That consistency is the point: a benchmark that type checks against a different interpreter than it runs is a benchmark whose guarantees are decorative. The dependency pins are also from that era rather than freshly resolved, with pandas at 1.5.3, numpy at 1.24.2, scikit-learn at 1.2.2 and requests at 2.28.2, all of which tell you when this project was last assembled rather than when it was last touched. Development dependencies live in their own requirement files, with one for reporting and one for development, and the presence of a reporting extra alongside the harness suggests the reporting side is separable from the run side, which matters because reporting tools tend to accumulate dependencies that the harness should not carry.

## The lint config records two failures it decided to live with

The packaging file is more interesting for its comments than its rules. Two lint codes are disabled at the top level and both carry an explanation. `E402`, imports not at the top of the file, is disabled because several modules need to set BLAS environment variables before importing, which is a real constraint rather than untidiness. `F403`, star imports, is disabled with a note to sort it out later, which is an honest admission rather than a justification. A file exclusion removes one framework's custom validation module from linting entirely, and the whole reporting package is excluded from lint as well. On the type checking side, attribute errors are switched off because the normal pattern of using namespaces produces them, and incremental mode is turned off because of a known interaction with the YAML library, referenced to a specific upstream issue. Coverage is scoped to three trees rather than the whole repository: the tests, the core package and the shared framework code.

## The root scripts describe a lifecycle with a recovery step

The top level tells you how a run actually proceeds. `runbenchmark.py` is the entry point for a single benchmark, `runall.py` sweeps the set, and `runstable.sh` is a shell wrapper for a stable configuration. Then there are the scripts that come after a run, which is where this repository is more candid than most. `recover_results.py` exists to pick up results, `upload_results.py` to send them somewhere durable, and `concat_csv.py` to merge outputs into one table. An AWS monitoring notebook sits alongside them, which tells you what the recovery path is for. The packaging configuration contains a note about the recovery script in its type-checking overrides, admitting that the special run-mode script makes some attributes optional and that the recovery script ends up abusing the benchmark class, with a to-do to refactor so it does not. That admission is worth reading: it tells you the recovery path exists because runs fail in ways the harness does not model, which is a normal fact about long batch experiments and an unusual thing to write down.

## Conclusion

Use this if your question is whether one AutoML system beats another under a budget you control, because the framework's value is that it fixes the task definitions that otherwise make the comparison unfair, and it lets you add your own framework rather than living with the list it ships. Do not reach for it to find a leaderboard number: the suites are identifiers into OpenML rather than a frozen corpus, and the ranking depends on the tasks, the budgets and the version of each framework you point at it. Three things to check before you plan a run. Which Python you are on, since both the lint target and the type checker are configured for 3.9 and the pinned requirement set was compiled for it. Whether the framework you want is in the list or needs the extension path. And when you last saw a release, because the newest tag is from September 2023 even though the repository is being pushed to, which tells you the project is maintained but versioned conservatively.

## FAQ

### What is the OpenML AutoML Benchmark?

An MIT-licensed framework for evaluating and comparing open source AutoML systems. It sets up standardized environments so that different systems are measured against the same task definitions, and its stated motivation is that differences in search space design or time budget definitions can drastically alter a task's difficulty, making results from different papers hard to compare.

### Can I add my own AutoML system to automlbenchmark?

Yes. The framework is described as extensible specifically so you can add your own AutoML frameworks and datasets. It already ships code to benchmark a number of popular AutoML systems on regression and classification tasks, so the extension path is the system under test.

### Where can automlbenchmark experiments run?

Experiments can run in Docker or Singularity containers, and they can execute locally or on AWS. The pinned dependency set includes boto3 for AWS and minio, s3fs and s3transfer for object storage, so results do not have to stay on the machine that produced them.

### What Python version does automlbenchmark target?

Python 3.9, consistently. The lint target, the type checker's language version and the pip-compile header in the requirements file all name 3.9, and the pinned dependency versions come from that period.

### Where does automlbenchmark get its datasets?

From curated suites hosted on OpenML, one for regression and one for classification, referenced by suite number rather than shipped in the repository. The suites are also the extension point for datasets, alongside the ability to add new AutoML systems.

## Sources

- [License: MIT](https://github.com/openml/automlbenchmark/blob/master/LICENSE)
- [openml/automlbenchmark on GitHub](https://github.com/openml/automlbenchmark)
- [Project website](https://openml.github.io/automlbenchmark)
- [README](https://github.com/openml/automlbenchmark/blob/master/README.md)
- [Releases](https://github.com/openml/automlbenchmark/releases)

---

Hysen Labs editorial analysis, written from the project's own repository and release notes. Cite the canonical page: https://hysenlabs.com/projects/openml-automlbenchmark