MLE-bench: What OpenAI's Kaggle-Derived Agent Benchmark Actually Measures
MLE-bench is a benchmark for measuring how well AI agents perform at machine learning engineering
At a glance
- What is it?
- MLE-bench packages Kaggle-style machine learning competitions into a graded evaluation harness for autonomous agents, and its leaderboard is currently closed to new entries. Here is the mechanism, the run commands, and where the design stops short.
- Who is it for?
- Adopt MLE-bench if you already build or operate an autonomous ML agent and want a graded, competition-derived score rather than a self-reported demo. Do not adopt it as a general code-agent benchmark or as a measure of research novelty; the tasks are Kaggle competitions with fixed submissions and held-out answers, and the README states the maintainers are not accepting new leaderboard submissions while they rework the fairness process.
- Can I use it commercially?
- Check first. The repository uses a licence we do not classify automatically, so read its LICENSE file before any commercial use.
- Is it still maintained?
- Yes. The repository last received commits 144 days ago.
- What is it written in?
- Mainly Python, according to GitHub's language statistics.
Answers come from the project's GitHub data, last synced on September 15, 2026, and from our analysis. They are not legal advice.
DEEP OPEN-SOURCE ANALYSIS
The gap MLE-bench fills: grading an agent that edits its own training script
Most agent benchmarks score a single response or a patch against a fixed test suite. Machine learning engineering does not look like that. An agent reads a competition description, explores a dataset, writes training code, runs it, reads its own validation output, and revises. The artifact is a submission file, and the score is a position on a held-out leaderboard. MLE-bench is built around that loop. The README describes it as a benchmark for measuring how well AI agents perform at machine learning engineering, and the repository ships the code used to construct the dataset, the evaluation logic, and the agents evaluated in the paper. The intended user is someone who has an agent harness and wants a number that reflects whether it can carry a modelling task from raw data to a scored submission, not whether it can answer a question about gradient boosting. The leaderboard rows make the framing concrete: entries are agents paired with specific LLMs, each with a running-time budget in hours and a date. The unit of comparison is the agent and model combination under a time limit, which is the honest unit for this kind of work.
How the harness works: competitions in, submission files out, grades against held-out answers
The repository is a Python package with a command-line entry point. The README's leaderboard section instructs submitters to organize grading reports in the runs/ folder by run groups. That tells you the shape of the pipeline even from a truncated README: a run is a named group, it produces a grading report, and the report is the artifact that gets aggregated into a score. The competition layer comes from Kaggle. Each task is a Kaggle competition with a training split the agent can see and a held-out split used for scoring, so the agent cannot simply inspect the answers. The grading logic is the part the repository claims to release explicitly, alongside the dataset construction code. Scores on the leaderboard are broken out by difficulty tier: Low == Lite, Medium, High, and All. The Low tier is presented as equal to Lite, which means the easiest subset doubles as a fast smoke-test tier. The Medium and High columns are where agents separate. The All column is the aggregate. Reporting all four is the right call, because a single aggregate number hides whether an agent is competent on straightforward tabular problems and collapses on the harder competitions, which is the pattern most practitioners actually care about. The README does not, in the material available, spell out the grading metric per competition, so treat the tier percentages as the documented unit and check the evaluation code for the per-task metric before you compare your own numbers to the table.
Getting it running: the commands the README gives
The README shows the install and run path in a small number of commands. Installation is a pip install of the package from the repository root: pip install -e . The prepare step downloads competition data: mlebench prepare -c <competition-id>. You can prepare more than one competition at a time by passing a comma-separated list to -c, and -c all prepares the full set. The run step executes an agent against prepared competitions: mlebench run --agent <agent> --competition <competition-id>, with --agent pointing at a registered agent and --competition at a prepared competition. Grading is a separate step: mlebench grade --competition <competition-id> --submission <path>. The README also mentions mlebench grade-sample <submission> <competition-id> for a single-submission check, and mlebench list for listing available competitions and agents. The split between prepare, run, and grade is the right design. Downloading Kaggle data is slow and storage-heavy, running an agent is expensive in tokens and wall-clock time, and grading is cheap. Separating them means a failed run does not force a re-download, and grading can be re-run against a saved submission without touching the agent. The one caveat visible in the README is that the run and grade steps key off competition identifiers, so a typo in the -c value fails at prepare time rather than at grade time, which is the cheaper place to catch it.
The leaderboard is closed, and that is a material limitation
The README carries an update dated 04-24-2026 stating that the maintainers are not taking new submissions to the leaderboard while they develop an improved process for ensuring submissions are fair and comparable. That is a direct statement about a known problem: comparability. Read the table and the reason is visible. Running times range from 12 to 36 hours. Some entries carry a note about test-set feedback, which means the agent had access to information about the held-out set, and those rows sit in a separate table explicitly marked as not directly comparable. One row's footnote describes light assistance from an ensemble of other models, distilled by a third. Another footnote notes that some scores were computed by padding incomplete seeds with failing scores, which changes what the number means. None of this is hidden; the README flags each case. But it means the top of the main table is not a clean ranking, and the maintainers have said so by pausing intake. If your goal is to place a number on a public board, that path is currently closed. If your goal is to run the harness locally and compare your agent against the published rows under your own conditions, the code is still there and still the point of the repository.
Where MLE-bench is the wrong tool
Three cases stand out. First, if you want to evaluate general coding ability, MLE-bench is the wrong instrument. The tasks are Kaggle competitions, and success depends on tabular preprocessing, feature construction, model selection and validation discipline more than on software engineering. An agent that writes clean, well-tested Python and cannot beat a baseline will score near zero here, and that tells you nothing about its value as a coding assistant. Second, if you want to measure research novelty or the ability to propose a new method, the format works against you. Competition scoring rewards fitting the held-out split, and the ceiling is set by the competition, not by the agent. Third, if your compute budget is small, the running-time column is the warning. Entries at 12 and 24 hours are the norm, and the README's own rows show 36. That is per run, per competition, and the All column aggregates across tiers, so a full evaluation is a multi-day, multi-machine exercise before you have a single comparable number. The prepare step is also a storage commitment: competition datasets are downloaded locally, and preparing all of them is a different order of magnitude from preparing one. Start with a single competition in the Lite tier, confirm the whole prepare-run-grade loop closes, and only then scale out.
AIDE and the agent-as-baseline comparison
The most useful comparison in the README is not between two leaderboard leaders; it is between the same agent on different models. AIDE appears four times in the table with the same 24-hour budget and four different LLMs: o1-preview at 17.12 percent on All, gpt-4o-2024-08-06 at 8.63, claude-3-5-sonnet-20240620 at 7.56, and llama-3.1-405b-instruct at 3.33. Same harness, same time limit, same grading. That is a clean model comparison, and it is the kind of row MLE-bench is built to produce. The alternative approach, and the one most teams reach for first, is a bespoke internal eval: a handful of past projects with a script that checks whether the pipeline runs and whether the metric clears a threshold. That approach is cheaper and fits your domain, but it is not comparable across teams or across time, because the tasks and thresholds are yours. MLE-bench trades that flexibility for a fixed task set with held-out answers and a documented grading step. The trade is real in both directions. You get comparability and a published baseline; you lose the ability to test the specific modelling problems your team actually ships. If your agent's value is concentrated in a narrow domain, a bespoke eval will tell you more, and MLE-bench will tell you something adjacent.
Maintenance, licence status, and what to check before you build on it
The repository is not archived and the last push date is 2026-04-24, the same date as the leaderboard freeze notice, so the project is active but the submission process is paused. There are no retrieved releases, which means there is no tagged version to pin against. You are tracking main, and the grading logic lives in the same tree as the dataset construction code. For a benchmark, that matters more than usual: if the grading code changes, your historical numbers may no longer be comparable to new ones, and without releases there is no changelog to tell you when that happened. Pin a commit hash in your own harness and record it alongside every score you produce. The licence field reads NOASSERTION, which means GitHub could not map the LICENSE file to a known identifier. That is a signal to open the LICENSE file yourself rather than an answer. There is a second layer: the competitions are Kaggle competitions, and each carries its own terms, which may restrict redistribution of the data even if the harness code is permissive. If you plan to publish results, check both. If you plan to redistribute prepared datasets, check the per-competition terms specifically, because the repository's own licence cannot grant you rights over data it merely downloads.
Editorial conclusion
Adopt MLE-bench if you already build or operate an autonomous ML agent and want a graded, competition-derived score rather than a self-reported demo. Do not adopt it as a general code-agent benchmark or as a measure of research novelty; the tasks are Kaggle competitions with fixed submissions and held-out answers, and the README states the maintainers are not accepting new leaderboard submissions while they rework the fairness process. Before committing compute, verify three things: that the competition data downloads and the prepare step completes under your storage budget, that your agent's output can be written as a submission file in the expected format, and that the grading report lands in the runs/ layout the leaderboard tooling expects. The licence field reads NOASSERTION, so read the LICENSE file and the per-competition data terms before redistributing anything.
Community notes