LettuceDetect: span-level hallucination detection for RAG, code and tool output

The gap LettuceDetect targets: answer-level verdicts hide which sentence failed

Most hallucination checks return one label for a whole answer. That is enough if you only want to block a response, and useless if you want to show a reviewer which sentence is unsupported, or trim the bad span and keep the rest. LettuceDetect is built around that second case. The README describes it as an "open span-level grounding verifier for AI outputs" that, given source evidence, localizes unsupported, contradictory or fabricated parts of an answer. The intended inputs are RAG answers, coding-agent responses and tool-grounded output, and the output is a set of character spans rather than a single score. The README also notes the models are inspired by the Luna paper, an encoder-based approach that works at token level. The audience is therefore teams that already have retrieval or tool context attached to a generation and want a check that points back into the text. If your pipeline discards the evidence after generation, there is nothing for this tool to compare against.

How the detector works: token classification over answer plus evidence

The v1 models are encoders. The README describes a token-level approach: the answer and its source evidence go in, and the model classifies tokens, which the library maps back to character spans. The published encoder families are ModernBERT (base and large, English) and EuroBERT (210M and 610M, multilingual), with long context windows of 4K and 8K respectively. The v2 line adds a generative path. According to the release notes for 0.2.0, the June 2026 release introduced code, tool and agentic models, typed spans, an encoder taxonomy cascade and automatic context chunking. Two v2 checkpoints are named in the README: lettucedect-v2-qwen-2b, a generative model that emits typed spans in one pass, and lettucedect-v2-mmbert-base, a fast encoder that returns binary spans. The typed output matters: the README says the detectors localize and type unsupported spans such as invented APIs or identifiers, wrong values, and behavior the request never asked for. The cascade is the part worth watching in practice. Chunking the context means span offsets have to survive that chunking, and the documentation does not spell out how offsets are reconciled across chunks.

Installing LettuceDetect and running a first detection

The package requires Python 3.11 or newer and installs from PyPI or from a checkout. The README gives both forms, and the repository pyproject.toml confirms the package name lettucedetect and the console script lettucedetect, which maps to lettucedetect.cli:main.

pip install lettucedetect -U

The dependencies pulled in are torch>=2.6.0, transformers>=4.48.3, scikit-learn, numpy, openai and rag-fact-checker. For a repository checkout the README uses an editable install.

pip install -e .

One pin needs attention before you build anything on top. The README warns that the EuroBERT models load remote code that is not yet compatible with transformers 5.x, and points to issue #33. Its stated workaround is to install a bounded transformers version when using those checkpoints. ModernBERT models are described as unaffected.

pip install "transformers>=4.48.3,<5"

Model loading is described as one-line through the Hugging Face integration. The README lists KRLabsOrg/lettucedect-base-modernbert-en-v1 and KRLabsOrg/lettucedect-large-modernbert-en-v1 for English, and a Hugging Face collection for the multilingual EuroBERT variants. The repository ships runnable entry points rather than only prose: demo/detection.ipynb and demo/detection_api.ipynb for the Python API, demo/streamlit_demo.py for a UI, and demo/code_hallucination_viewer.py for the code path. The README does not print the exact constructor arguments for a first detection, so read the notebook before writing your own call. An optional FastAPI surface exists behind the api extra (fastapi[standard], pydantic-settings, httpx); the README does not document its routes, so treat the notebooks and the lettucedetect_api package as the source.

Where the design gets awkward: chunking, offsets and model choice

Span-level output is only as good as the offsets. Automatic context chunking arrived in 0.2.0, and the README does not describe how a span found in the second chunk is reported against the original answer string. If you build a highlight-in-the-UI feature, verify that mapping yourself on a long answer before trusting it. The second trade-off is model selection. The encoder path is fast and local, but returns binary spans; the generative 2B path types each span, which is more informative and more expensive to run. The README's own framing is that the fine-tuned 2B detector substantially outperforms off-the-shelf detectors and large LLM judges on code-agent answers, which it says over-flag generated code. That claim comes from the project's benchmark, published at KRLabsOrg/lettucedetect-code-hallucination, not from an independent evaluation, and the README points to docs/benchmarks.md and the model cards under docs/code-hallucination/ for scoped results and stated limitations. Read those before assuming the code path fits your repository conventions. Finally, the CLI is registered in pyproject.toml, but the README does not document its flags or subcommands, so the Python API is the better-documented entry point.

LettuceDetect compared with an LLM judge and with RAGTruth-style evaluation

The obvious alternative is asking a general LLM whether the answer is grounded. That approach needs no local weights and handles arbitrary phrasing, but it returns a judgement you then have to map back to text yourself, and the README argues it over-flags generated code in the code-agent setting. LettuceDetect replaces that judgement with token-level classification, so the span comes out of the model rather than out of a second prompt. The second comparison is with RAGTruth. RAGTruth is a benchmark for hallucination in RAG, cited in the README as training data alongside the 14-language PsiloQA dataset. It is evaluation ground truth, not a detector you deploy in a request path. LettuceDetect is the runtime component; RAGTruth and the project's own code-hallucination dataset are what you would measure it against. A third option is the project's own LLM baseline mode: the README notes you can use LLM baselines for hallucination detection, and the 0.2.2 release notes mention a fix to method="llm" confidence validation. That path keeps you inside the same library while trading local inference for an API call.

Maintenance, releases and what the MIT licence covers

The repository is not archived, and the last push was on 2026-09-07. Releases have been frequent through 2026: 0.2.1 on 2026-07-02, 0.2.2 on 2026-07-05 and 0.2.3 on 2026-08-13. The changelog and a public roadmap are both in the repository, and the 0.2.2 notes describe a detector-hierarchy regression test, which suggests the library's own class hierarchy is covered by tests. The licence is MIT for both code and, per the README, the models. That is permissive for commercial use, but it says nothing about the provenance of the training data behind each checkpoint. The README cites RAGTruth, PsiloQA and the project's own code-hallucination dataset; if your compliance process requires dataset-level review, that work sits with you, not with the MIT text. Upgrade cost is concentrated in two places. The transformers upper bound for EuroBERT means a future transformers 5.x migration is blocked until issue #33 is resolved. And the v2 model line arrived in 0.2.0 with new output types, so code written against v1 spans will need review when you move to typed spans.

Editorial conclusion

Adopt LettuceDetect if you need character offsets for unsupported spans inside a RAG answer, a code-agent reply or tool output, and you can run PyTorch locally or call the API. Do not adopt it if an answer-level clean/flagged verdict from an LLM judge is enough, or if you cannot accept the EuroBERT transformers pin. Verify first that your Python is 3.11 or newer and that the model card for the checkpoint you pick covers your language and task.

Frequently asked questions

Official sources

1. KRLabsOrg/LettuceDetect on GitHub
2. License: MIT
3. Project website
4. README
5. Releases