What is OCR?

How a page becomes text

There are three generations of approach, and they fail in different places.

The classical pipeline. Preprocess the image: convert to grayscale, binarise so ink is black and paper is white, deskew so text is horizontal. Then analyse the layout to find blocks, columns and lines, and recognise each line. Older systems segmented further into words and characters and classified them one at a time; modern ones train a recurrent network over a whole line, so the segmentation boundary that used to break everything is gone. tesseract-ocr/tesseract is the long-lived example: our analysis describes it as an Apache-2.0 engine plus a command line program with LSTM line recognition, UTF-8 support and more than a hundred languages, and the page segmentation mode remains the setting that decides how a complicated page is divided.

Sequence-to-sequence recognition. One model reads an image region and emits text directly, with no per-character classes. It removes the segmentation problem entirely and it is what made handwriting and unusual fonts workable.

Page-level models. A vision-language model is given the whole page image and asked for text, and it returns a structured document rather than a flat string. PaddlePaddle/PaddleOCR is now in this camp as well as the traditional one, with our analysis describing it as an Apache-2.0 Python toolkit that turns PDFs and images into Markdown or JSON, shipping a 0.9B document vision-language model alongside a smaller recognition family running from 1.5M to 34.5M parameters. baidu/Unlimited-OCR does one forward pass per document, and our analysis notes it runs through Hugging Face Transformers, vLLM or SGLang on CUDA hardware. datalab-to/chandra is the same idea with forms and merged cells in mind, and our analysis says Chandra OCR 2 turns PDFs and images into Markdown, HTML or JSON while keeping layout, tables and forms intact.

There is a fourth idea worth separating out. deepseek-ai/DeepSeek-OCR does not transcribe at all: our analysis describes it compressing document images into a small number of vision tokens which a language model then decodes, which treats a page as something to be compressed rather than read, and that is why it also works on things that are not text.

The part that is actually hard: layout

Characters are the easy part. A page is not a stream of text, and almost every bad OCR result is a layout error wearing a recognition error's clothes.

Read a two-column paper as one stream and the two columns interleave line by line, producing text that is individually correct and completely wrong. A form has labels in one place and values in another. A table has a header row that belongs to no single cell. So a serious system analyses layout first, recognises each region separately, then decides the reading order, and only then serialises.

The serialisation target is a real design decision, not a formatting preference. Plain text is the most compatible and the most lossy: a table becomes a run of numbers with no rows. Markdown keeps headings, lists and simple tables, and is what most people want to read. HTML can express a table properly but merges and spanning cells are exactly where it gets ambiguous. JSON is the most explicit, and it moves the burden onto whatever consumes it, because the schema is the producer's invention. Choosing Markdown when your downstream is a database is how a table ends up as a paragraph.

The same ordering argument explains why a page-level model changed the field. Giving a model the whole image means it can see that two numbers belong to a row, which no line-based recogniser can do, because the line is not the unit that contains the meaning. datalab-to/surya is the clearest expression of this, with our analysis describing it as Datalab's open OCR stack for more than ninety languages whose version 2 routes layout, recognition and table extraction through a single vision-language model that the library spawns on your behalf.

When you need it, and when the file has text already

OCR is for pixels. The most common wasted work in this area is running it on a document that already contains extractable text.

A born-digital PDF, produced by a word processor or a publishing tool, has a text layer. Extracting it is faster, exact and free. ocrmypdf/OCRmyPDF is built around the case where the layer is missing: our analysis describes it as a Python command line tool that runs Tesseract over scanned PDFs and writes the recognised text back as an invisible layer, producing a PDF/A by default, which is the right fit for batch conversion on a machine you control and the wrong fit if you want a hosted service or a desktop interface.

A document worth OCRing is a scan, a photograph, a screenshot, a fax, an archived paper record, or a photograph of a whiteboard. Handwriting is the case that separates the generations: it is where line-level recognition starts to struggle and where a page-level model with language priors does better.

There is a third case that surprises people. Many PDFs that look scanned already carry a text layer, left there by a previous OCR run of poor quality. That layer is worse than nothing, because search and copy-paste will silently return wrong text while the page looks correct. Detecting it is worth a few lines of checking before you decide you need OCR at all.

And a case where you do not want a model. A recogniser running over a clean printed archive processes hundreds of pages a minute on one machine, and a vision-language model costs a forward pass with a large context per page. The choice is throughput against layout competence, and a pipeline that does both is common because the two stages have different strengths.

Where it goes wrong

Six problems, and the first is the one people blame on the engine.

Resolution. Recognition accuracy depends on how many pixels each character has, and a scan at 150 dots per inch does not have enough. The practical floor for printed text is around 300, and below it the recogniser is guessing. This is the most common cause of a bad result on a good engine.

Rotation, skew and noise. A page photographed at an angle needs deskewing before anything else, and a low-contrast or speckled scan needs denoising. Every engine exposes these as options because the preprocessing is doing more work than the recogniser.

The silent failure. OCR does not usually error. It returns confident, plausible, wrong text, and a pipeline that only checks whether the output is non-empty will accept it. The only measure that works is a character error rate against a sample you have labelled yourself.

Wrong serialisation. Correct characters, wrong structure, and a table becomes a paragraph.

Language coverage. A model that handles ninety languages is not equally good at all ninety, and scripts it was not trained on produce plausible-looking nonsense rather than blanks. Check the language you have, not the count of languages advertised.

Model licences. datalab-to/chandra is the example to look at: our analysis says the code is Apache-2.0 while the model weights carry a separate OpenRAIL-M licence, and that split is the first thing to check before you build on it. Surya's model licence is a similar question, per our analysis. An Apache-2.0 repository is not a statement about the weights it downloads.

How the projects divide

Five groups, and the group tells you which problem you are actually solving.

The engine. tesseract-ocr/tesseract is the reference implementation most things are built on or compared against, and it remains the right answer for clean printed text at volume because it is small, fast and has a hundred languages behind it.

The same engine everywhere else. naptha/tesseract.js wraps a WebAssembly build of the same engine so recognition runs in JavaScript, in a browser tab or in Node, with no native binary to compile; our analysis describes the trade-offs as ones of scope and accuracy rather than packaging, which is the right way to think about it. T8RIN/ImageToolbox is the on-device case, with our analysis describing an Apache-2.0 Android app that does cropping, filtering, EXIF editing, OCR and background removal on a phone.

The toolkit. PaddlePaddle/PaddleOCR is the largest of these, and the interesting part is that it contains both generations: a small recognition family and a document vision-language model, behind one interface that outputs Markdown or JSON.

The document wrapper. OCRmyPDF is the one to reach for when the deliverable is a searchable PDF rather than a text file, because adding an invisible text layer to the original image keeps the document viewable.

The page-level models. Unlimited-OCR, chandra and surya all read the whole page and emit structure, and they differ in what they optimise: Unlimited-OCR in a single forward pass on CUDA hardware, chandra in forms and tables, surya in ninety-plus languages with one spawned model per stage. DeepSeek-OCR is the outlier that reframes the problem as compression.

Whichever you pick, the first thing to measure is your own character error rate on a hundred pages of your own documents, because the ranking of these systems on your paper is not the ranking on someone else's.

In practice

OCR is a solved problem for clean printed text and an unsolved one for layout, and the page-level models are what closed most of that gap. Decide by throughput and by document shape: tesseract-ocr/tesseract for volume and predictable print, PaddlePaddle/PaddleOCR when you want both a small recogniser and a document model behind one interface, one of the page-level stacks when tables and forms matter more than throughput, and OCRmyPDF when the output is a searchable file rather than text. Before any of it, check whether your documents already have a text layer, and after it, measure your own character error rate rather than reading a leaderboard, and read the model weights licence separately from the code licence.