MMMU: Running the Multimodal College-Exam Benchmark Locally

The gap MMMU fills: subject breadth with image types that break naive pipelines

Most visual question answering benchmarks ask about scenes, objects or documents. MMMU asks college exam questions. The README describes 11.5K questions collected from college exams, quizzes and textbooks, spread over six core disciplines (Art & Design, Business, Science, Health & Medicine, Humanities & Social Science, Tech & Engineering), 30 subjects and 183 subfields. The image side is deliberately heterogeneous: 32 image types, including charts, diagrams, maps, tables, music sheets and chemical structures. That mix is the point. A model that scores well on natural photographs can still fail on a music sheet or a circuit diagram, and MMMU is built to expose exactly that split. The intended audience is model developers and evaluation engineers who need a number that reflects domain knowledge plus deliberate reasoning, not just captioning ability. The README also frames the benchmark as a step toward expert AGI, which is a research framing rather than a deployment criterion; treat the paper's motivation as context, not as a claim about any specific model you are testing.

MMMU-Pro changes the input contract, not just the difficulty

MMMU-Pro is a separate variant built in three documented steps. First, questions that can be answered from text alone are filtered out, so the remaining items require the image. Second, additional plausible candidate options are added to the multiple-choice sets. Third, a vision-only input setting embeds the question inside the image, so the model has to read and see in one pass. That third step is the one that changes your integration work: if your pipeline assumes a text prompt plus an image URL, the vision-only setting does not fit it without modification. The README reports that model accuracy on MMMU-Pro ranges from 16.8% to 26.9% across the models evaluated, well below MMMU numbers, and that OCR prompts had minimal impact while chain-of-thought generally improved results. Those figures come from the paper's evaluation, not from this repository's code, and the README does not enumerate which models produced the low end of that range. If you plan to report MMMU-Pro numbers, read the paper for the model list before comparing your own run.

Splits, sizes and what each one is for

The README states the suite contains 150 development samples, 900 validation samples and 10,500 test samples. The intended use is explicit: development for few-shot or in-context learning, validation for debugging models, selecting hyperparameters and quick evaluations. The test set is the one that matters for published comparisons. Until February 2026 the test answers were withheld and predictions had to be submitted through EvalAI, which the README now shows struck through. The current state is that answers and explanations for the test set have been released, and the README says you can evaluate locally using answer_dict_test.json with the same pipeline as the validation set. That is a meaningful change in workflow: a full 10,500-sample test run is now something you can do on your own hardware, but it also means the test set is no longer a held-out set in the strict sense for anyone who downloads the answers. If you are comparing against leaderboard entries submitted before that release, be aware the conditions differ.

Getting it running: the repository path and the EvalScope path

There are two documented routes. The first is the repository's own code, split into two folders: mmmu for the base benchmark and mmmu-pro for the variant. The README points to those folders for detailed instructions rather than reproducing the commands, so the exact entry points live in the folder READMEs, which are not included in the material available here. The second route is the community-maintained EvalScope integration, added in July 2026. It targets OpenAI-compatible endpoints and the README gives this command:

evalscope eval \ --model YOUR_MODEL \ --api-url OPENAI_API_COMPAT_URL \ --api-key EMPTY_TOKEN \ --datasets mmmu \ --limit 10

The integration notes state that EvalScope uses the ModelScope mirror AI-ModelScope/MMMU, defaults to the validation split with 900 examples across 30 subjects, applies 0-shot prompting, handles both multiple-choice and open-ended questions, and reports accuracy in EvalScope's standard report format. The README is direct about the boundary: leaderboard-equivalent results require the official MMMU evaluation pipeline and the released test-set answer files, not the EvalScope wrapper. The --limit 10 flag in the example is a smoke test, not a real evaluation.

Where the pipeline is the wrong tool

The EvalScope route is a convenience layer, and the README itself draws the line at leaderboard equivalence. If your goal is a number you can put next to published results, the wrapper is the wrong instrument, because it runs validation, not test, and it applies its own prompt format. A second limitation is structural: MMMU is a static question set with a fixed answer key. It cannot tell you whether a model fails because of perception, because of subject knowledge, or because of answer-format parsing, unless you go read the per-question outputs yourself. The README's own reporting on MMMU-Pro (that OCR prompts barely moved results while chain-of-thought helped) is the kind of finding that only comes from inspecting failures, not from the aggregate accuracy. Third, the subject mix is uneven by construction: 30 subjects across 11.5K questions means some subfields are represented by a small number of items, and the README does not publish per-subject counts. A model that happens to be strong in one heavily weighted discipline can move the headline number. Do not use MMMU as a fine-grained diagnostic per subject without checking the counts in the Hugging Face dataset first.

Alternatives and the actual difference in approach

The most direct alternative is MMMU-Pro, which ships in the same repository. The difference is not difficulty tuning; it is a different input contract. MMMU-Pro removes text-only answerable questions, adds distractor options, and offers a vision-only setting where the question itself is rendered into the image. If your model consumes a text prompt plus an image, MMMU-Pro's vision-only mode tests a capability your production pipeline may never exercise, and the README's reported 16.8% to 26.9% range suggests it is a much harder target. The other documented alternative is EvalScope itself, which is not a competing benchmark but a competing harness: it trades the official pipeline for an OpenAI-compatible endpoint wrapper with saved predictions and reports. Choosing between the repository code and EvalScope is a choice about what you need the number for. Reproducing a published result points to the repository. Quick iteration against a hosted endpoint points to EvalScope, with the caveat that the README calls the resulting number non-leaderboard-equivalent.

Licence, maintenance and the cost of staying current

The repository is Apache-2.0. The dataset itself is a separate artifact hosted on Hugging Face, and the README's disclaimer says annotators were instructed to follow copyright and licensing rules of the source material, avoiding sites that forbid copying and redistribution, with a contact address for removal requests. That is a takedown process, not a guarantee of provenance for every item. If you redistribute MMMU data or derived predictions, check the dataset card on Hugging Face rather than assuming the Apache-2.0 licence on the code covers the questions and images. On maintenance: the repository has no releases listed in the material available here, and the last push is dated 2026-07-28, the same date as the EvalScope news entry. The upgrade surface is therefore not versioned tags but the main branch and the two evaluation folders. Pinning to a commit is the only way to make a run reproducible, since there is no release to pin to. The February 2026 test-answer release is the kind of change that silently invalidates a comparison against older leaderboard submissions, so record the date you pulled the data alongside any score you publish.

Editorial conclusion

Adopt MMMU if you need a subject-diverse, college-level check on a multimodal model and you are willing to wire the repository's own inference loop into your serving stack. Do not adopt it as a general-purpose harness for arbitrary chat endpoints, and do not treat a validation-set number as leaderboard-equivalent. Before quoting a score, confirm which split you ran (validation has 900 samples, test has 10,500), check that you used answer_dict_test.json rather than a validation answer file, and read the mmmu and mmmu-pro folders to see how the prompt and parsing path differ from the EvalScope wrapper.

Official sources

1. Issues
2. License: Apache-2.0
3. MMMU-Benchmark/MMMU on GitHub
4. Project website
5. README