MultimodalOCR: three OCR benchmarks for large multimodal models in one repository

Three suites that answer three different questions

The repository is described as the home of OCRBench, OCRBench v2 and MDPBench, each of which has its own directory and its own readme, and the top level listing matches that: a LICENSE.txt, the three benchmark directories and the root README.

They are three generations of the same idea, and each one widens the scope of what is being measured.

OCRBench is the original, titled On the Hidden Mystery of OCR in Large Multimodal Models, with an arXiv identifier of 2305.07895. It is described as a broad evaluation benchmark designed to assess the OCR capabilities of Large Multimodal Models, and the description begins to enumerate its five components: Text Recognition, SceneText-Centric VQA, Document-Oriented VQA, and two further components that the excerpt cuts off. The framing of the title is the interesting part. If a large multimodal model can read text, that is supposed to be one of its most basic capabilities, and the paper is about the gap between what models appear to do and what they actually do.

OCRBench v2 is the middle generation, and its subtitle is more specific: an improved benchmark for evaluating large multimodal models on visual text localization and reasoning. Where OCRBench asked whether a model can read, v2 asks whether it can find the text and reason about it. The documentation gives three concrete measures of that widening: it has 4 times more tasks than the previous multi-scene benchmark OCRBench, it covers 31 diverse scenarios including street scene, receipt, formula and diagram, and it holds a total of 10,000 human-verified question-answering pairs with a high proportion of difficult samples.

MDPBench is the third, and it exists because of a specific gap. Its title is a benchmark for multilingual document parsing in real-world scenarios, and the reasoning in the documentation is that document parsing has made remarkable strides, yet almost exclusively on clean, digital, well-formatted pages in a handful of dominant languages, and that no systematic benchmark existed to evaluate how models perform on digital and photographed documents across diverse scripts and low-resource languages.

MDPBench's real subject is the gap between open and closed models

The findings MDPBench reports are more interesting than the dataset itself, and they are stated in the root readme rather than hidden in a subdirectory.

The dataset comprises 3,400 document images spanning 17 languages, and the languages are named: Simplified Chinese, Traditional Chinese, English, Arabic, German, Spanish, French, Hindi, Indonesian, Italian, Japanese, Korean, Portuguese, Russian, Thai and Vietnamese, along with diverse scripts and varied photographic conditions.

The annotations came from a rigorous pipeline of expert model labeling, manual correction and human verification, which is a deliberate description. Expert model labeling scales; manual correction fixes the errors; human verification is what makes it a benchmark rather than a dataset.

The evaluation design is also deliberate. To ensure fair comparison and prevent data leakage, the maintainers keep separate public and private evaluation splits. That matters more than it might sound. A benchmark whose answers are all public is a benchmark that models can be trained on, and a private split is the only mechanism that keeps a leaderboard meaningful over time.

The finding is the headline. Evaluating both open-source and closed-source models uncovers a striking result: while closed-source models, and Gemini3-Pro is named in particular, held up comparatively well, open-source alternatives suffer dramatic performance collapse, particularly on non-Latin scripts and real-world photographed documents, with an average drop of 17.8% on photographed documents and 14.0% on non-Latin scripts.

The stated conclusion is that these results reveal significant performance imbalances across languages and conditions, and point to concrete directions for building more inclusive, deployment-ready parsing systems.

That is a result with consequences for anyone choosing a model for a document pipeline outside English and Chinese, and it is also a caution about how earlier benchmarks were built: an evaluation set of clean digital pages will not surface a gap that only appears on a photograph of a page.

Three ways to get the data, and how the leaderboards are hosted

Each benchmark is distributed through the usual model hubs, and the distribution is part of what makes them usable.

For MDPBench, the documentation links an arXiv paper at 2603.28130, a Hugging Face dataset at Delores-Lin/MDPBench, a ModelScope dataset at DeloresLin/MDPBench, and a leaderboard space at Delores-Lin/MDPBench-leaderboard. The ModelScope presence is not incidental for a benchmark whose stated subject is low-resource languages, because ModelScope is where a large number of researchers outside the English-language ecosystem publish and host their work, and a benchmark that claims multilingual coverage but is only mirrored on one platform would be unevenly accessible.

For OCRBench v2, the links are an arXiv paper at 2501.00321, a Hugging Face dataset at ling99/OCRBench_v2, a Google Drive download, and a leaderboard hosted as a static site at 99franklin.github.io/ocrbench_v2. The Google Drive link is the odd one out in a set of modern releases, and the documentation does not explain why, so whether it is still maintained is worth checking before you rely on it.

The original OCRBench links an arXiv paper at 2305.07895, a Hugging Face dataset at echo840/OCRBench, a leaderboard space at echo840/ocrbench-leaderboard, and a Google Drive download under the OCRBench data directory.

So the pattern is consistent across the three. Papers on arXiv, data on Hugging Face, and leaderboards in one of two forms: a Hugging Face Space for the original and for MDPBench, and a plain static site for v2. The two forms are not equivalent. A Space is a hosted application that can be run against a model and submitted, whereas a static page is a table that someone updates.

The documentation states the update dates, which is a useful signal of how fast this space moves: MDPBench was updated on July 20, 2026, and OCRBench v2 in June 2026.

A benchmark collection with no inference code

It is worth being clear about what this repository is, because the name invites a wrong assumption. MultimodalOCR is not a library you call to make a model better at OCR, and it is not a toolkit for running a model over an image.

The top level listing is LICENSE.txt, MDPBench, OCRBench, OCRBench_v2 and README.md. There is no src directory, no package manifest, no requirements file, no evaluation harness script and no test suite in the listing. The primary language on the project record is Python, which is what you would expect from the evaluation and scoring code that lives inside the benchmark directories, and what the root readme's use of a Python-style link to a data directory implies, but the top level gives you no entry point to install or run.

What the documentation gives you instead is pointers. For MDPBench it says more details can be found in the MDPBench README. For OCRBench v2 it says more details can be found in the OCRBench v2 README, with a link to the subdirectory. The original OCRBench section points at its own subdirectory readme for details as well. Each benchmark is documented in its own directory, and the root readme is the index.

That structure has a consequence for a reader. The root document is a landing page with the headline findings, and the substance, the task definitions, the scoring method, the answer formats and the exact data layout, lives in the subdirectories. Anyone citing a number from a leaderboard should read the subdirectory readme for that benchmark, because the definition of a correct answer is where the difficulty actually lives.

The author list on the papers gives a sense of scale. The MDPBench paper lists eleven authors including Zhang Li, Zhibo Lin, Qiang Liu, Xiang Bai and Yuliang Liu, and the OCRBench v2 paper lists twenty-five. These are the groups that built the original OCRBench, and the v2 paper in particular shows how much effort went into a benchmark that is, at heart, a carefully designed question set.

MIT licensed datasets that other people have to maintain the hosting for

The licence situation is worth understanding precisely, because it is not the usual arrangement for a benchmark.

The project record lists the licence as MIT, and the repository carries a LICENSE.txt. The datasets themselves are distributed under their own terms through Hugging Face, ModelScope and Google Drive, and the readme links those download pages with badges labelled as dataset downloads, which indicates that each host applies its own dataset licence rather than the repository licence covering the data.

The practical consequence is that the code and the data are licensed separately, and in practice the data terms are the ones a Hugging Face or ModelScope dataset carries, which is a non-commercial or research use arrangement in many cases. For a benchmark that the whole point of which is to measure commercial models, a research-only data licence is a friction point for a company that wants to run the evaluation internally on a model they are about to deploy.

There is a second practical point about the hosted leaderboards. The MDPBench leaderboard is a Hugging Face Space, and the OCRBench leaderboard is another one, so both are running applications maintained by third parties. The v2 leaderboard is a static site at 99franklin.github.io. None of these is the same as the benchmark code, and the readme does not describe who operates them or what happens to a leaderboard when a maintainer moves on.

The MIT licence on the repository is straightforward and useful in the way you would expect: you can read the benchmark, fork it, extend it with your own tasks, and use it as the licence permits, with the notice retained. The datasets and the leaderboards are the parts you should check the terms on individually before building a dependency on them.

Where these benchmarks are the wrong instrument

Three situations rule them out, and the third is the one that gets people.

The first is a system you are building rather than evaluating. There is no inference code here and no pipeline to reuse. If your problem is that a model misreads a specific kind of document your business actually receives, a general benchmark is a poor diagnostic, because it tells you that the model is bad at receipts rather than telling you which field it misread on your receipt. For that, you build an evaluation set from your own data, and these benchmarks are a design reference for how to structure one.

The second is a single-language, clean-input workload. OCRBench and OCRBench v2 are heavily weighted towards the scenarios those benchmarks were built for, and MDPBench exists specifically because that coverage was a problem. If every document you process is a clean digital PDF in English, then the measured differences between models may be smaller than the differences you would see on photographed pages and low-resource scripts, and you would be picking on the wrong axis.

The third, and the sharpest, is reading a leaderboard as a product claim. Every published score comes from a particular split, a particular task set and a particular prompt format, and the readme is explicit that the maintainers keep separate public and private splits precisely because public answers can leak into training. A number on a leaderboard page is a score on the public split, produced by the submitter's harness. It is not a promise about your workload.

The honest use of this repository is comparative and directional. MDPBench's own finding is the model for it: the 17.8% and 14.0% average drops tell you that the choice of open or closed model matters, and under what conditions. They tell you a great deal less about which particular model is best for your documents.

Editorial conclusion

MultimodalOCR fits a team that has to decide whether a multimodal model can read a document, a receipt or a street scene, and wants a number that separates models on hard, real inputs rather than on clean rendered text. It is the wrong tool if you are building rather than evaluating, because there is no inference code here, and it is the wrong tool if your inputs are all clean digital pages in one dominant language, which is exactly the case OCRBench v2 and MDPBench were built to move away from. Before you use a score, read the paper for the benchmark you are about to cite, because the three suites measure different things, and check whether the model you are evaluating is one the benchmark authors had in mind, since a published score and a score you reproduce on the public split are not the same thing.

Frequently asked questions

Official sources

1. Issues
2. License: MIT
3. README
4. Yuliang-Liu/MultimodalOCR on GitHub