Vectara's Hallucination Leaderboard: What HHEM Measures and What It Leaves Out

The Problem It Addresses: Summarization Hallucination Has No Common Yardstick

Ask three vendors how often their model invents facts and you will get three definitions. Some count any sentence not directly traceable to the source. Some count only contradictions. Some count nothing and report a refusal rate instead. The leaderboard exists to replace that with one fixed procedure applied to every model in the table. The README states the goal plainly: it evaluates how often an LLM introduces hallucinations when summarizing a document, using Vectara's Hallucination Evaluation Model, HHEM. The audience is anyone choosing a summarization model across vendor lines, plus researchers who need a comparable number to cite. It is published as a repository rather than a paper or a dashboard alone, which matters: the table is versioned in git, and the README points to two earlier snapshots, one based on HHEM-1.0 and one based on HHEM-2.3 with the previous dataset. That versioning is the most useful thing here. A leaderboard whose scoring model changes over time is only honest if the old numbers stay reachable, and in this repository they do, on separate branches.

How the Score Is Produced: One Summarization Task, One Judge Model

The pipeline is short. A short document goes in, the candidate LLM produces a summary, and HHEM scores that summary against the source for factual consistency. The table then reports four columns per model: Hallucination Rate, Factual Consistency Rate, Answer Rate, and Average Summary Length in words. The first two are complements of each other, so a 1.8 percent hallucination rate is a 98.2 percent factual consistency rate. The interesting columns are the other two. Answer Rate is the share of prompts where the model produced a summary at all. Average Summary Length tells you how much text the model emitted. Those two columns are what keep the ranking from being a single number you can sort and forget. A model at 3.7 percent hallucination with an 80.7 percent answer rate is not directly comparable to a model at 3.1 percent with a 100.0 percent answer rate, because the first one skipped roughly a fifth of the inputs and its rate is computed on the rest. The README also notes the intent to update regularly as the model and the LLMs change, and the table carries a last-updated date. Treat every row as a snapshot with an expiry, not a property of the model.

Reading the Columns Together: Why the Top of the Table Is Not the Shortlist

Sort by hallucination rate and the first row is antgroup/finix_s1_32b at 1.8 percent with a 99.5 percent answer rate and 172.4 average words. That is a clean result on all three axes. Now look at microsoft/Phi-4, fourth at 3.7 percent, with an 80.7 percent answer rate. Its hallucination rate covers only the prompts it answered. snowflake/snowflake-arctic-instruct at 4.3 percent has a 62.7 percent answer rate, meaning it declined or failed on more than a third of inputs, and google/gemma-3-4b-it at 6.4 percent answered 67.3 percent. If your application requires a summary for every document, those models are not interchangeable with the ones above them. The same logic applies at the other end. openai/gpt-5.1-high-2025-11-13 sits at 12.1 percent hallucination with a 254.4-word average summary, the longest in the visible table, while openai/gpt-5.4-mini-2026-03-17 sits at 5.5 percent with 54.7 words. A longer summary has more surface area for a factual-consistency scorer to flag. The ranking does not normalize for that, and the README does not claim it does. Whether that is a flaw depends on what you want: if you want a faithful short summary, the short-output models are being compared fairly against long-output ones only if you accept that length and hallucination rate are entangled in this setup.

Running It Yourself: What the Repository Actually Ships

This is the section where the material runs thin, and it is worth being direct about that. The README supplies no installation instructions, no requirements file listing, no CLI invocation, and no configuration keys. What it does supply is the rendered output: a markdown table between the LEADERBOARD_START marker and the corresponding end marker, a plot image referenced as ./img/top25_hallucination_rates_2026-04-20.png, and links to the interactive version on Hugging Face Spaces at huggingface.co/spaces/vectara/leaderboard. The repository is Python, so the generation path is presumably a script that calls the HHEM model and rewrites the table in place, but nothing in the supplied README confirms the entry point, the environment variables, or the model loading call. If you need to reproduce a row, the honest starting point is the Hugging Face Space and the HHEM model card, not this README. What you can do from the repository alone is read the table and diff it against the branch snapshots. The two archived branches, hhem-1.0-final and hhem-2.3-old-dataset, let you see how a given model moved when the scoring model or the dataset changed, which is often more informative than the current absolute number.

The Scope Boundary: Summarization of Short Documents Is Not RAG

The single largest limitation is in the title. This measures summarization of short documents. It does not measure hallucination during retrieval-augmented generation over a corpus, where the model must decide which retrieved passages are relevant, ignore distractors, and synthesize across several sources. It does not measure hallucination in multi-turn dialogue, where earlier turns can be misremembered. It does not measure tool-calling accuracy or structured extraction. A model that scores 3 percent here can still fabricate a citation in a RAG pipeline, because the failure modes are different. There is a second boundary that follows from the first. The judge is HHEM, Vectara's own model, and the leaderboard is published by Vectara. That is not a scandal, since the scoring model is named and its versions are exposed as branches, but it means the number is only as good as HHEM's agreement with human judgment on the same task, and the README does not present that validation. If you need a hallucination number for a task other than short-document summarization, this table gives you a prior about which models tend to invent things, and nothing more precise than that.

Alternatives and the Actual Difference in Approach

The closest alternative is a general-purpose evaluation harness such as lm-evaluation-harness, which runs many benchmarks across many tasks with a pluggable task registry. The difference is not quality, it is shape. A harness gives you a framework you configure and extend; this repository gives you a single fixed task, a single judge, and a published table. If you want to know how a model behaves on summarization specifically, the harness makes you define the metric and the judge yourself, and your numbers will not be comparable to anyone else's. This repository removes that choice, which is exactly why its rows can be compared to each other. A second alternative is to run your own evaluation on your own documents with your own judge. That is the only approach that answers the question you actually have, and it is also the most expensive, because you need labeled examples and a judge you trust. The practical reading is that this leaderboard is a screening step. It narrows a field of dozens of models to a handful worth testing. It does not replace the test.

Maintenance, Versioning, and the Licence Question

The repository is Apache-2.0, which permits commercial use, modification, and redistribution provided the licence text and notices are preserved. That covers the repository contents: the table, the plot, the README, and any generation scripts. It does not automatically cover the HHEM model weights or the interactive Space, which may carry their own terms, and the supplied material does not state what those are. Check the model card before you ship anything derived from the scores. On maintenance, the visible signal is the last-updated date of May 11, 2026, and the README's statement that updates are planned as the model and the LLMs change. There are no tagged releases in the retrieved metadata, so versioning happens through branches and through the table's own date stamp. For a consumer of the numbers, that means pinning to a branch or recording the last-updated date alongside any figure you quote. If you cite a row without the date, you are citing something that will silently change under you.

Who Should Adopt This and What to Check First

Use it if you are selecting a summarization model for short documents and want a defensible shortlist, or if you need a published, dated, versioned reference for the claim that hallucination rates differ by an order of magnitude across vendors. Do not use it as a general model-quality ranking, and do not use it to pick a model for RAG, chat, or extraction. The first thing to check on any row you care about is the Answer Rate, because a low hallucination rate paired with a low answer rate is a model that abstains rather than a model that is accurate. The second is Average Summary Length, because the scoring does not appear to normalize for output length. The third is the last-updated date, because the table is a moving target and the README says so. After that, take your top three candidates and run them on your own documents with your own definition of a hallucination. The leaderboard's job is to make that shortlist small. It cannot make the decision.

Editorial conclusion

Adopt this as a reading list, not as a gate. It is useful if you are picking a summarization model for short documents and want a starting shortlist, or if you need a citation for the claim that hallucination rates vary widely across vendors. It is the wrong tool if your task is retrieval-augmented generation over long inputs, multi-turn chat, structured extraction, or anything where the model must reason rather than condense. Before you rely on a row, check the Answer Rate column, because a model that declines to answer 37 percent of the time can post a low hallucination rate on the remainder, and check the Average Summary Length column, because a 254-word average and a 54-word average are not the same task. Then re-run your own top three candidates on your own documents.

Official sources

1. Issues
2. License: Apache-2.0
3. Project website
4. README
5. vectara/hallucination-leaderboard on GitHub