ParseBench: a benchmark that scores parsed PDFs on what an agent can still do with them

The failure ParseBench is built to catch

Most document parsing evaluations compare extracted text to a reference string. ParseBench starts from a different premise: a parsed document is only useful if an agent can act on it. The README states the benchmark tests whether parsed output preserves the structure and meaning needed for autonomous decisions, not just whether it looks similar to a reference text. That distinction matters in practice. A parser can reproduce every word on a page and still return a table as a run of lines with no cell boundaries, or drop the coordinates that let a downstream step point back at a region of the page. The intended audience is engineers wiring parsers into agent workflows over enterprise PDFs, plus the parser vendors themselves. The corpus is drawn from insurance, finance and government documents, which is where the awkward layouts live: multi-page tables, footnoted charts, and forms where position carries meaning. If your input is clean, single-column text, this benchmark measures things you never hit.

Five dimensions, five ground-truth formats, five metrics

The dataset is stratified into five dimensions, and each has its own file, ground-truth format and metric. Tables use table.jsonl with GTRM, described as GriTS plus TableRecordMatch, over 503 pages from 284 documents. Charts use chart.jsonl with ChartDataPointMatch over 568 pages from 99 documents, carrying 4,864 rules. Content Faithfulness uses text_content.jsonl with a Content Faithfulness Score over 506 pages and 506 documents, with 141,322 rules, by far the densest rule set. Semantic Formatting uses text_formatting.jsonl with a Semantic Formatting Score over 476 pages and 476 documents and 5,997 rules. Visual Grounding uses layout.jsonl with Element Pass Rate over 500 pages from 321 documents and 16,325 rules. The totals are 2,078 unique pages, 1,211 documents and 169,011 rules. One detail worth flagging: Content Faithfulness and Semantic Formatting share the same 507 underlying text documents and are scored with different rule sets. That means those two numbers are not independent samples of parser behaviour. A parser that mangles a particular document will tend to lose points in both columns, which inflates the apparent spread between a strong and a weak pipeline when you read the Overall column alone.

How the harness is put together

A pipeline is the unit of evaluation: a parsing tool or a specific configuration of one. The README says there are 180+ pipelines, listed in docs/pipelines.md or via the parse-bench pipelines command. The paper baselines number 21 and span four kinds of provider. There are LlamaParse configurations (Agentic, Agentic Plus, Cost Effective), hosted VLMs (OpenAI GPT-5 Mini at two reasoning levels, GPT-5.4, Anthropic Haiku 4.5 with and without thinking, Opus 4.6, Gemini 3 Flash at minimal and high thinking, Gemini 3.1 Pro), cloud document services (Azure Document Intelligence, AWS Textract, Google Cloud Document AI), commercial APIs (Reducto in two modes, Extend, LandingAI) and open-weight or self-hosted models (Qwen 3 VL, Dots OCR 1.5, Docling). Several pipeline names end in _parse_with_layout_file, which suggests those runs feed a layout representation into the model rather than the raw page image, though the README does not spell out the prompt or the layout format. Extension points are documented separately: docs/extending.md covers adding providers, pipelines, products and rule types from another package. The inclusion criteria for the leaderboard are explicit and worth reading before you complain about an absent tool: public accessibility via open weights or a self-serve API, a run that finishes in roughly single-digit hours, and no custom framework changes, with concurrency adjustable to the provider's recommended settings.

Getting a run started

Installation is a Python package with per-provider extras. The README gives pip install "parse-bench[runners]" for every provider SDK, or a single extra such as parse-bench[llamaparse], parse-bench[openai], parse-bench[anthropic] or parse-bench[google]. From a repository checkout the equivalent is uv sync --extra runners. There is an optional fast extra, parse-bench[runners,fast], which pulls in numba for a JIT-accelerated TEDS table metric; the README states scores are identical to the default path and the extra only changes speed on large tables. Credentials go in a .env file, one API key per parsing tool you intend to evaluate. The three commands that matter are uv run parse-bench run llamaparse_agentic --test for a small run of three files per category, uv run parse-bench run llamaparse_agentic for the full set, and uv run parse-bench serve llamaparse_agentic to open interactive reports in a browser. Drop the uv run prefix if you installed from PyPI. The --test flag is the cheapest way to confirm your key, your network path and the provider's rate limits before committing to a full pass. Note that the CLI takes a pipeline name, not a provider name, so swapping in your own tool means either matching an existing pipeline or following docs/extending.md.

Reading the leaderboard without being misled

The published top ten ranks by Overall score. LlamaParse Agentic Plus sits first at 90.20, with 93.37 on Tables, 94.18 on Charts, 92.25 on Content Faithfulness, 87.12 on Semantic Formatting and 84.09 on Visual Grounding, at 5.62 cents per page. LlamaParse Agentic is second at 87.01 and 1.25 cents per page, roughly a quarter of the cost for about three Overall points. Third is Pulse Ultra 2 at 81.60, then LlamaParse Cost Effective at 80.61 at 0.38 cents per page, which is the cheapest entry on the board by a wide margin. Anthropic Fable 5.1 is fifth at 78.92 but costs 16.05 cents per page, the highest listed price. Several entries have no price at all, including oi-parser, rakedoc-nano, florin-parser-nano and KDL-Frontier-Parser-nano, which makes cost-per-quality comparison impossible for those rows. Two structural cautions apply. First, the top four rows are all LlamaParse products, and ParseBench is maintained by run-llama, so the benchmark is published by a participant in its own ranking. That is not an accusation of error, but it is a reason to read the per-dimension columns rather than the Overall rank. Second, the dimension columns disagree sharply: Anthropic Fable 5.1 scores 91.52 on Tables but 67.06 on Charts, while LlamaParse Cost Effective scores 83.77 on Visual Grounding against 67.29 on Semantic Formatting. A single Overall number averages away exactly the trade-off you are trying to make.

Where ParseBench is the wrong tool

The scope is PDFs. Nothing in the README suggests HTML, DOCX, spreadsheets or scanned image sets outside a PDF container are covered, so a pipeline that only ever sees those formats gets no signal here. The benchmark also grades parsing, not extraction against a schema. The README points to a companion benchmark, ExtractBench, for schema-guided enterprise document extraction, which is the right place to look if your question is whether a tool can fill a fixed set of fields rather than whether it preserves structure. Two further limits are visible in the material. The dataset is 2,078 pages, which is enough to separate pipelines that differ by several points but thin for resolving close calls, and the README itself notes that two of the five dimensions share the same 507 underlying documents, so the effective diversity is lower than the page count implies. Finally, the leaderboard's inclusion criteria exclude anything that needs custom framework changes or runs longer than roughly single-digit hours. A slow but accurate local model that would take a day is simply absent, and its absence is not evidence about its accuracy. There is no stated refresh cadence for the leaderboard, and the releases in the supplied material are patch versions (v1.0.2 through v1.0.4, all within September 2026), which tells you about the harness, not about how often the underlying pipeline results are regenerated.

The real alternative, and how it differs

The obvious comparison is Docling, which appears in ParseBench's own pipeline list as docling_parse. The difference is one of role, not quality. Docling is a parser you install and run; ParseBench is a harness that runs parsers and scores them. Choosing Docling means accepting one implementation's behaviour on your documents. Choosing ParseBench means you keep the option of swapping implementations, but you take on the cost of API keys, per-page fees and a network dependency for every hosted pipeline you evaluate. The second comparison is ExtractBench, the companion benchmark named in the README. ExtractBench asks whether a tool can produce a schema-conformant record; ParseBench asks whether the parsed representation preserves the tables, charts, text, formatting and layout that a downstream step needs. If your pipeline ends in a JSON object with fixed keys, ExtractBench is the closer fit. If it ends in a document tree that an agent browses, ParseBench is. A third reference point is building your own evaluation set from your own PDFs. That gives you distribution match and no vendor involvement, at the cost of the human verification that ParseBench claims for its pages and the per-dimension metrics it already implements.

Licence and maintenance cost

The repository is Apache-2.0, which permits commercial use and modification, and the README links the licence file directly. The licence covers the code, not the dataset: the data is hosted separately on HuggingFace under llamaindex/ParseBench, and the README does not state a licence for it in the supplied material, so check the dataset card before redistributing pages. The practical running cost is dominated by provider fees rather than the harness. At the listed rates, a full pass over 2,078 pages costs roughly 7.90 dollars for LlamaParse Cost Effective at 0.38 cents per page, about 26 dollars for LlamaParse Agentic at 1.25 cents, about 117 dollars for LlamaParse Agentic Plus at 5.62 cents, and about 334 dollars for Anthropic Fable 5.1 at 16.05 cents. Those are arithmetic on the published per-page figures, not measurements. Re-running the full board across every pipeline is therefore a recurring spend, and the --test flag exists precisely so you do not pay it while debugging credentials. On the maintenance side, the release history shows three patch releases inside a week, which suggests active work on the harness itself; pin a version if you depend on score comparability across runs.

Editorial conclusion

Adopt ParseBench if you are choosing between the 180+ pipelines it already wraps and you care about tables, charts and layout rather than plain text extraction. Skip it if your documents are not PDFs, or if you need a schema-guided extraction score rather than a parsing score; the companion ExtractBench covers that case. Before trusting a number, run the small dataset with parse-bench run <pipeline> --test, then read the per-dimension scores instead of the Overall column, because an Overall of 90.20 for LlamaParse Agentic Plus hides a Visual Grounding score of 84.09 and a Semantic Formatting score of 87.12 that are the two lowest of its five dimensions.

Official sources

1. License: Apache-2.0
2. Project website
3. README
4. Releases
5. run-llama/ParseBench on GitHub