MLPerf Inference: the reference implementation repo behind the benchmark, not a runtime you ship
Reference implementations of MLPerf® inference benchmarks
At a glance
- What is it?
- mlcommons/inference holds the reference apps, datasets loaders and accuracy harnesses that define MLPerf Inference. It is a submission and reproducibility toolchain, and it is a poor fit as a production serving stack.
- Who is it for?
- Adopt mlcommons/inference if you are preparing an MLPerf Inference submission, reproducing a published result, or need the exact preprocessing and accuracy checks that define a benchmark scenario. Do not adopt it as your serving layer; the LoadGen harness and reference backends exist to make results comparable, not to be fast or operable in production.
- Can I use it commercially?
- Yes. Apache-2.0 is a permissive licence: you can use, modify and sell software built on it, as long as you keep its copyright and licence notices.
- Is it still maintained?
- Yes. The repository last received commits 13 days ago.
- What is it written in?
- Mainly Python, according to GitHub's language statistics.
Answers come from the project's GitHub data, last synced on September 15, 2026, and from our analysis. They are not legal advice.
DEEP OPEN-SOURCE ANALYSIS
What problem the repo actually solves
MLPerf Inference answers a narrow question: given a fixed model, a fixed dataset and a fixed latency constraint, how many queries per second can a system sustain while still producing answers that pass an accuracy check? The repository is the machine that makes that question answerable the same way at every vendor. It ships the reference application for each benchmark model, the data loaders that feed them, and the LoadGen harness that issues queries under a scenario such as SingleStream or Server. The README describes it as a suite for measuring how fast systems can run models in a variety of deployment scenarios, and the accompanying paper (arXiv 1911.02549) is cited as the source for the motivation and guiding principles.
The audience is therefore not application developers. It is benchmark submitters, hardware vendors preparing entries for a submission round, and researchers who need to reproduce a published number rather than trust it. The README is explicit that submitters may use their own frameworks; the frameworks listed in the model table describe the reference implementation only. That single sentence defines the repo's role: it is the specification made executable, not the code you are expected to ship.
The loader, backend, harness split
The architecture visible in the repository is a three-layer split repeated across benchmark directories. A dataset loader in Python handles preprocessing and, where the scenario requires it, tokenization or image decoding. A backend wraps the model and exposes a predict-style call. LoadGen sits above both and decides when queries are issued, how they are batched, and what latency percentiles are recorded. Accuracy evaluation is a separate pass: the harness collects outputs, and a checker compares them against a reference using the metric defined for that benchmark.
This is why the same model appears under several directories with different framework choices. vision/classification_and_detection lists tensorflow, onnx, tvm and ncnn for resnet50-v1.5; language/bert lists tensorflow, pytorch and onnx for the same model. The benchmark definition is the model plus the dataset plus the scenario, and the backend is swappable underneath it. If you are evaluating this repo to understand what MLPerf measures, the loader and accuracy-checker code is the part worth reading closely, because that is where the definition lives. The backend directories are the least interesting part and the most likely to be replaced by whatever you already run.
The v6.1 model table and what changed
The README carries two model tables, one for v6.1 with a submission deadline of July 31 2026 and one for v6.0 with a deadline of February 13 2026. Comparing them is the fastest way to see the benchmark's drift toward generative and agentic workloads. The v6.1 table adds stable-diffusion-xl under text_to_image, Wan2.2-T2V-A14B-Diffusers under text_to_video, GPT-OSS under language/gpt-oss-120b, a VLM entry (qwen3-vl-235b-a22b) under multimodal/qwen3-vl, qwen3.6-27b under language/edge-agentic, and e2e-rag with qna and db variants.
The table also states plainly that pointpainting, llama3.1-405b and mixtral-8x7b are not part of the v6.1 model list. That matters more than it looks. If your submission pipeline was built around one of those three, the reference app may still sit in the tree, but it is not a v6.1 target, and the README does not offer a migration path. Note also the category column: llama3.1-8b and whisper are listed for both edge and datacenter, while dlrm-v3, rgat, llama2-70b, deepseek-r1, GPT-OSS, the VLM entry and e2e-rag are datacenter only. A model being present in the repo tells you nothing about which division you can enter it in.
Getting a benchmark running from this repo
The README does not put runnable commands in the repository front page. It points to docs.mlcommons.org/inference/benchmarks/ as the documentation site, and says that site includes automated commands for running MLPerf inference benchmarks using different implementations. That is the intended entry point, and it is worth being direct about the consequence: the repository alone is not a quickstart. You get a tree of reference apps and a pointer to a separate documentation site.
For a submission, the README is more specific. Use the master branch and any commit since the v6.1 seed release, with the stated preference for the latest commit on master. The seed release is identified by commit b0662794362bc2c22c33c3e4a0c66ef83411ecf0, and the v6.0 seed is f131a0d29ccae9a967d93ffe96f66b1be3537d3b. Power submissions have a different dependency chain: SPEC PTD 1.11.1 from the mlcommons/power repository, which the README notes needs special access, plus any commit of mlcommons/power-dev after the code-freeze commit c4b3ad8202fbd8ac28d77149e5e7aeadb725bbf2. Per-benchmark configuration keys, dataset paths and backend flags live in the individual reference app directories rather than in a single top-level config, so expect to read the README inside the directory for the model you care about.
Where this repo is the wrong tool
The most common mistake is treating the reference implementation as a performance baseline. It is not. The README says submitters are free to use their own frameworks, which exists precisely because the reference backends are chosen for clarity and portability, not for throughput. If you run a reference backend and compare its numbers to a vendor's submitted result, you are comparing two different programs.
The second failure mode is dataset access. Several v6.1 entries depend on data that is not bundled and not trivially replaceable: OpenOrca for llama2-70b, CNN-Daily Mail for llama3.1-8b, IGBH for rgat, FRAMES for e2e-rag, a COCO safe subset for yolo v11, and for deepseek-r1 a set spanning AIME, MATH500, gpqa, MMLU-Pro and livecodebench. A loader that cannot find its data fails, and the accuracy metric is defined against that specific data, so substituting a similar corpus invalidates the check. The third is scope: if you want a serving benchmark for your own model, on your own traffic mix, this repository gives you a benchmark definition you did not choose. LoadGen's scenarios are fixed, and bending them to a custom workload means you are no longer running MLPerf.
Alternatives and the actual difference
The obvious alternative for latency measurement is a general load generator such as Locust or k6 driven against your own HTTP endpoint. The difference is not tooling quality, it is what is being held constant. Locust lets you define the request mix, the arrival pattern and the payloads, which is what you want for capacity planning against real traffic. MLPerf Inference fixes all three so that two vendors' numbers can be placed side by side. If your question is how many concurrent users your service survives, Locust answers it; MLPerf answers a question you did not ask.
A second alternative is the vendor's own benchmark harness, typically a script that drives the vendor's runtime directly. Those are usually faster to start and closer to the hardware's real behaviour, and they are also tuned by the party being measured. The reason to prefer mlcommons/inference is that the accuracy check and the scenario semantics come from a third party, and the README's seed-release and code-freeze commit references exist so that a specific run can be tied to a specific revision of those semantics.
Maintenance, releases and the licence
Release cadence follows the submission rounds rather than a software schedule. The recent tags are v5.1 and v5.1.1 on 2025-10-28, then v6.0.0pre on 2025-12-23, ahead of the v6.0 deadline of February 13 2026. A pre-release tag sitting in the history is a signal about how the tree is used: submitters pin to a commit, not to a tag, and the README's instruction to use any commit since the seed release reflects that. The practical cost is that upgrading between rounds is not a version bump. The model table changes, entries are removed, and the seed commit moves.
The licence is Apache-2.0, which permits commercial use and modification, and the README asks that any use of the benchmark, including reference implementations and submissions, cite the 2019 paper. Apache-2.0 does not grant rights to the MLPerf name or to the datasets the benchmarks load; those are separate questions with separate answers, and the repository does not address them. That is a licensing question for your own counsel, not something this review can settle.
Editorial conclusion
Adopt mlcommons/inference if you are preparing an MLPerf Inference submission, reproducing a published result, or need the exact preprocessing and accuracy checks that define a benchmark scenario. Do not adopt it as your serving layer; the LoadGen harness and reference backends exist to make results comparable, not to be fast or operable in production. Before committing, verify three things against the v6.1 seed release or a later master commit: that your target model appears in the v6.1 model table, that the accuracy dataset named there (OpenOrca for llama2-70b, CNN-Daily Mail for llama3.1-8b, FRAMES for e2e-rag) is one you can actually obtain, and that your power numbers, if you report them, come from SPEC PTD 1.11.1.
Community notes