mlcommons/training: What the MLPerf Reference Implementations Actually Are, and What They Are Not

The problem is reproducibility, not training speed

MLPerf exists so that training results can be compared across hardware and software stacks. That comparison only works if everyone trains the same model on the same data to the same target quality. This repository supplies that shared definition: for each benchmark it provides code implementing the model in at least one framework, a Dockerfile, dataset download instructions, a script that runs and times training, and documentation covering dataset, model and machine setup. The audience is people preparing a submission, or people who need to understand exactly what a benchmark model is before they build their own version of it. It is not an audience of people looking for a fast training loop. The README says the implementations are valid as starting points but are not fully optimized, and it describes them as still very much alpha or beta quality. That sentence should be read as the scope statement for the whole repository.

What one benchmark directory contains

The layout is deliberately uniform. Every benchmark ships the same five pieces: model code in at least one framework, a Dockerfile, dataset download instructions, a timing script, and documentation. The framework column in the release tables tells you which stack the reference uses, and the README notes that submitters are free to use their own frameworks. That distinction matters. The reference implementation defines the model and the target quality; it does not constrain the submission. So when you read the v6.1 table and see NeMo next to llama3.1_8b, Primus next to gpt_oss_20b, or torchtitan next to flux.1, you are reading a statement about what the reference code uses, not a requirement. The parameter counts in the table come with a footnote that they are not the same as the active parameters being trained in the benchmark, which is a useful warning for the mixture-of-experts entries where the gap is large.

The run sequence: docker, dataset, verify, build

The README gives a four-step sequence that applies generally. First, set up docker and dependencies using a shared script, install_cuda_docker.sh, with the caveat that some benchmarks need additional setup described in their own READMEs. Second, download the dataset from mlcommons-storage, and the README is specific that this runs outside docker, on the host machine, from the directory the download script lives in, because it may make assumptions about the current working directory. Third, optionally run verify_dataset.sh to confirm the download succeeded. Fourth, build and run the docker image, with the exact command included in each benchmark. The timing behaviour is also stated: each benchmark runs until the target quality is reached and then stops, printing timing results. There is no config key list in the material supplied here, so any claim about specific configuration files would be guesswork; the per-benchmark READMEs are where those live.

The v6.1 model list, and the retirement pattern

The current table is MLPerf Training v6.1, with a submission deadline of Oct 16, 2026. It lists seven benchmarks: flux.1 (text_to_image, torchtitan, CC12M subset, 11.9B), llama3.1_8b (small_llm_pretraining, NeMo, C4, 8b), llama2_70b_lora (pytorch, SCROLLS GovReport, 70B), gpt_oss_20b (small_llm_moe_pretraining/primus, Primus, C4, 20B), deepseekv3 (llm_moe_pretraining/nemo, NeMo, C4, 671B), qwen35_397b_grpo (llm_post_training, NeMo-RL / NeMo-Gym, SWE tasks, 397B), and dlrm_v4_hstu (recommendation, pytorch, Yambda-5b). Comparing v6.0 and v5.1 shows how fast the set turns over: llama3.1_405b and dlrm_dcnv2 appear in older tables but point into retired_benchmarks, and retinanet appears in v5.1 but not in v6.0 or v6.1. If you pin to this repository for a long project, expect the model you integrate to be retired within a few cycles, and expect the path to move under retired_benchmarks rather than disappear.

Where this repository is the wrong tool

The README states plainly that these implementations are not intended to be used for real performance measurements of software frameworks or hardware. Anyone who wants a number to publish about their accelerator, their compiler, or their distributed training library is looking at the wrong artifact, and the repository says so before you clone it. A second limitation is speed: the README notes that some benchmarks are rather slow or take a long time to run on the reference hardware, and that significant improvements are expected with more hardware and optimized implementations. A third is quality, described as alpha or beta with an open invitation to file issues and pull requests. Add the dataset burden: C4, a CC12M subset, SCROLLS GovReport, Yambda-5b and SWE tasks are not small, and the download runs on the host outside docker, so storage and transfer time land on you before any training starts. If your goal is a working model for a product, or a fast iteration loop on a small dataset, this repository adds ceremony without giving you either.

The alternative is your framework's own examples, and the difference is the target quality

If you want to train a Llama-style model, a diffusion model or a recommendation model, the obvious alternative is the example or recipe collection shipped by the framework itself, for instance NeMo, torchtitan, Primus or PyTorch's own training examples. The difference in approach is what each one optimizes for. Framework recipes aim at a working model with reasonable defaults, and they change when the framework changes. This repository aims at a fixed definition: a named model, a named dataset, and a stopping condition based on target quality, with the same five artifacts in every directory so that two implementations can be compared. That fixed target is the entire value. It is also why the reference code is allowed to be unoptimized. A framework recipe that is 30 percent faster but trains a slightly different configuration is not a substitute if your purpose is comparison.

Maintenance cost, release cadence and the licence

The cost of tracking this repository is dominated by the benchmark cycle, not by dependency churn in the usual sense. The release history shows a long gap: v0.5 in April 2019, then v4.0 in April 2024, with the version tables in the README now running ahead to v6.1. The last push to the default branch is dated 2026-08-17. Practical consequence: if you build on a specific benchmark directory, budget for the possibility that it moves into retired_benchmarks in a later cycle, as llama3.1_405b and dlrm_dcnv2 already have. The repository is Apache-2.0, which is permissive and includes an explicit patent grant; that is a statement about the licence text, not legal advice, and if you redistribute a modified benchmark or ship it inside a product you should read the licence and the MLPerf submission rules separately, since the two govern different things. The README also asks that academic use of the benchmark, reference implementations or submissions cite the MLPerf Training Benchmark paper, arXiv 1910.01500.

Editorial conclusion

Adopt this repository if you are building a submission, studying how a benchmark model is defined, or need the dataset download and verification scripts as a starting point. Do not adopt it if you want a training framework, a tuned implementation, or numbers you can quote as hardware performance: the README states these implementations are not fully optimized and are not intended for real performance measurements. Before committing, verify three things in the specific benchmark directory you care about: whether the model you need is in the current v6.1 table or only in retired_benchmarks, what the per-benchmark README says about extra setup beyond install_cuda_docker.sh, and whether the dataset named there (C4, CC12M subset, SCROLLS GovReport, Yambda-5b, SWE tasks) is one you can actually obtain and store.

Official sources

1. License: Apache-2.0
2. mlcommons/training on GitHub
3. Project website
4. README
5. Releases