Versatile-OCR-Program: A Config-Driven OCR Pipeline That Annotates Figures and Tables for ML Datasets

The problem is annotation, not character recognition

Plain OCR returns text. A model trained on exam papers needs more than that: it needs to know that a region is a figure rather than a paragraph, that a block of glyphs is a formula rather than prose, and what the figure depicts. Versatile-OCR-Program targets that gap. The README describes it as designed to extract structured data from complex educational materials, with exam papers named as the primary case. The output samples show why this is a different job from running Tesseract over a scan. In the biology example, a table of stage labels is emitted as a Markdown table and then followed by a summary, an educational value note, and a list of related topics. That surrounding commentary is generated, not extracted. The intended user is someone building an ML dataset from PDFs dense with formulas and diagrams, who would otherwise hand-write captions for every figure. It is a poor fit for anyone who just wants searchable text out of a contract.

A modular pipeline wrapping four external engines

The v3.0 release note describes a refactor into a modular, config-driven architecture, with the older v1.0 and v2.0 trees moved under legacy/ and explicitly not maintained. The README names the components: DocLayout-YOLO for layout detection, Google Vision API, Gemini Pro Vision, MathPix OCR and the OpenAI API, with OpenCV in the stack. The division of labour implied by that list is a layout stage that segments a page into text, figure, table and formula regions, followed by per-region routing to whichever engine handles that region type. MathPix is the obvious candidate for formulas, Gemini Pro Vision for the natural-language descriptions the samples show, and Google Vision for plain text. The repository layout, with a setup guide, a usage manual and a changes/ directory holding release notes, is consistent with the config-driven claim. What the supplied material does not give is the routing rules themselves, so which engine handles which region is inference from the tool list rather than something stated.

What the generated annotations actually look like

The sample outputs are the most concrete thing in the README, and they are worth reading closely because they define the product. A math question from a 2014 University of Tokyo paper is followed by an [Image Start] block containing an image description, an educational value paragraph, a related topics list, and a numbered exam relevance list. The biology sample from a 2017 EJU paper follows the same shape and adds a [Table Start] block with a Markdown table and its own summary. Both samples are labelled as English-translated outputs, which tells you the source material was Japanese and that translation is part of the pipeline. Note the structure: the model is being asked not just to describe a figure but to situate it in a curriculum. That is a deliberate choice for dataset construction, since downstream training benefits from captions that share vocabulary with the questions. It also means the annotations are model-generated text, and the README offers no evaluation of their correctness. A wrong phase assignment in a mitosis caption would propagate silently into your corpus.

Getting it running: the documented path and its gaps

The README's getting-started block points to three files rather than listing commands inline. The setup guide covers the sequence clone, API keys, Docker, first run. The usage manual is described as covering CLI flags, scenarios, config reference and FAQ. The release notes file is changes/2026-05-13_v3.0_release.md. So the documented path is a Docker-based run driven by a config file, with credentials for the external services supplied as environment or config values. I cannot give you the literal command line or the config key names, because the README does not contain them and I have not run the project. That is a real friction point for evaluation: you cannot judge the interface from the README alone, and you have to open setup_guide.md and usage.md before you know whether the CLI matches how you work. Budget for reading three documents, not one, and expect the config reference to be the one that matters, since the architecture is described as config-driven.

Accuracy claims and the missing evaluation

The README states over 90 to 95 percent accuracy on real-world academic datasets such as EJU Biology and UTokyo Math. Treat that as an author's claim about the two sample papers shown, not as a benchmark. There is no described test set, no per-element breakdown separating text accuracy from table accuracy from figure-caption quality, and no error analysis. The distinction matters here because the pipeline produces two very different kinds of output. Character-level text extraction accuracy is a well-understood quantity. Whether a generated caption correctly identifies metaphase in a micrograph is a different question with no metric attached in the material. If your use case depends on the annotations rather than the raw text, the 90 to 95 percent figure tells you almost nothing. The honest position is that the sample outputs look plausible and the accuracy claim is unverified.

The cost model is per-page API calls to four vendors

This is the limitation that will decide adoption for most teams, and the README does not discuss it. The pipeline depends on Google Vision API, Gemini Pro Vision, MathPix OCR and the OpenAI API. Every page you process incurs calls to one or more of those, at whatever their current rates are, and MathPix in particular is priced for formula extraction rather than bulk throughput. There is no described local-only mode and no caching layer mentioned. Compare that with a self-hosted stack such as Marker or MinerU, which run layout and text extraction on your own hardware with no per-page billing, at the cost of model download size and GPU time. The trade is straightforward: Versatile-OCR-Program buys you formula handling and generated figure captions, and you pay per page for them. For a few hundred exam papers that is likely fine. For a corpus of a hundred thousand pages, run the arithmetic before you start, because the architecture gives you no way to opt out of the paid engines.

Maintenance, licence and the abandoned predecessor

The last push is 2026-05-13, the same date as the v3.0 initial release, and no releases have been retrieved. So the project is at the start of a major refactor, not at a stable point. The README states plainly that v1.0 and v2.0 are preserved under legacy/ for reference but not maintained, which means anyone who built on the older architecture has no upgrade path other than a rewrite against the new config format. The licence field reports NOASSERTION, meaning GitHub could not map the repository's licence file to a known identifier. That is not a statement that the project is unlicensed, but it does mean you have to open the licence file and read it yourself, particularly if you intend to redistribute the generated datasets or ship the pipeline inside a product. The README also carries a note about a separate AI pipeline release being delayed past June because of an exam on June 15, which tells you the project has a single maintainer working alongside academic commitments. Plan for that bus factor.

When a plain extractor is the better call

If your documents are text-dominant and you do not need figure captions, the extra machinery here is overhead. A tool like PyMuPDF extracts text and tables from PDFs with no API keys, no network calls and no per-page cost, and it will handle a scanned-paper corpus badly but a born-digital one well. The difference in approach is that PyMuPDF reads the PDF content stream directly, so it never needs to decide what a region is; Versatile-OCR-Program rasterizes, segments with DocLayout-YOLO, and routes each region to a vision model. That routing is exactly what you want when a page mixes a formula, a micrograph and a data table, and exactly what you do not want when every page is a column of prose. The other case for a plain extractor is reproducibility: a deterministic text-layer read gives you the same output every run, while a pipeline built on generative vision models will not.

Editorial conclusion

Adopt it if you are assembling a training corpus from exam-style PDFs and you already hold API credentials for the vendors it wraps, because the value here is the annotation layer rather than the OCR engines. Do not adopt it if you need a general-purpose document digitizer, a permissively licensed dependency set, or a project with a published release history. Verify three things before committing: the exact licence text, since the repository reports NOASSERTION; the current cost of Gemini Pro Vision, MathPix and Google Vision calls at your document volume; and whether the v3.0 setup guide still matches the CLI flags in usage.md, because the refactor landed on 2026-05-13 and older v1.0 and v2.0 code now sits unmaintained under legacy/.

Official sources

1. Issues
2. raphael-seo/Versatile-OCR-Program on GitHub
3. README