Vision models on documents: when OCR is still the better tool

AI Tools & ModelsEvie3 replies

EvieMember#1

Vision models got good at reading documents, and the result is that a lot of people stopped using OCR. For a whole class of documents that was a mistake.

What a vision model is good at

Layout. A vision model sees the page, so it understands that a figure caption belongs to the figure and that a table has headers. That is a genuine advantage over OCR, which produces a stream of characters and no structure.

Reading a table. Where the structure is visual and irregular, a vision model often gets it right where a text-based converter produces a mangled pipe table.

A page that is mostly a picture. A scanned form, a chart with text in it, a page with a diagram and a paragraph. OCR gives you the paragraph and nothing else.

What OCR is still better at

Speed and cost. A page through OCR is a fraction of the cost and a fraction of the time of a page through a vision model, and for a corpus of 50,000 pages that difference is the whole project.

Exactness. OCR gives you the characters. A vision model gives you what it read, which is usually right and occasionally confidently different. For a serial number, a figure in a table, a version string, a legal citation — you want the characters.

Auditability. You can point at the word an OCR engine was unsure about. A vision model does not have a per-word confidence in a form you can act on.

Working without the page layout. If you are extracting text for a search index and you do not need the structure, the structure is overhead.

The rule

Use OCR to get the text, and use a layout-aware converter or a vision model on the pages where the structure matters. The expensive tool is for the pages that need it, not for the corpus.

The tool chain here that does this in the browser: /tools/ocr-image-to-text for the OCR, and /tools/pdf-to-text and /tools/pdf-to-images for getting there. /learn/ocr has the background and /compare/paddleocr-vs-tesseract compares the two engines.

3 replies

AamirMember#2

The auditability point is the one that decides it for anything regulated.

A scanned invoice where the amount has to be exact, or a licence plate in a photo, or a page of a contract that goes to a lawyer: you need the characters and a confidence you can look at. A vision model's reading of a page is an answer, and an answer is not the same thing as a transcription.

JudyMember#3

In reply to Aamir (#2)

The cost difference is worth quantifying for anyone planning a crawl, because it changes the order of operations.

A corpus of 50,000 pages through a vision model is a real bill and a real wait. OCR first, then a second pass over the pages that turned out to be table-heavy, is usually a tenth of the cost for most of the quality.

EvieAuthorMember#4

In reply to Judy (#3)

That is the right way to phase it, and the second pass is the part people skip because it requires deciding which pages need it.

The cheap test for "is this page table-heavy" is to look for long runs of digits in the OCR output with few spaces, which is what a table looks like once the structure is gone. That is a heuristic, but it is a very good one.