uqlm: response-level uncertainty scores for LLM hallucination detection
[JMLR 2026] "UQLM: A Python Package for Uncertainty Quantification in Large Language Models"
At a glance
- What is it?
- UQLM is a Python package from CVS Health that turns an LLM's own outputs into a 0 to 1 confidence score. It ships four scorer families with very different cost and access requirements, and the README is explicit about which trade-off you are buying.
- Who is it for?
- Adopt uqlm if you already have a LangChain chat model wired up and you need a confidence number per response rather than a rewritten answer; the white-box path is the only one that adds no extra LLM calls, so start there if your model exposes token probabilities. Do not adopt it as a fact-checker: the README describes confidence scores, not ground-truth verification, and the high-accuracy configurations multiply your inference bill.
- Can I use it commercially?
- Yes. Apache-2.0 is a permissive licence: you can use, modify and sell software built on it, as long as you keep its copyright and licence notices.
- Is it still maintained?
- Yes. The repository last received commits 1 day ago.
- What is it written in?
- Mainly Python, according to GitHub's language statistics.
Answers come from the project's GitHub data, last synced on September 15, 2026, and from our analysis. They are not legal advice.
DEEP OPEN-SOURCE ANALYSIS
The problem: an LLM answer with no attached confidence
A chat model returns text. It does not return a number saying how much you should trust that text. Teams that put LLM output in front of users, or into a downstream automated step, usually need that number, because the alternative is treating a fluent sentence and a fabricated sentence as equally acceptable. UQLM targets exactly that gap. The README states it is a Python library for LLM hallucination detection using uncertainty quantification techniques, and that each scorer returns a confidence score between 0 and 1, where higher scores indicate a lower likelihood of errors or hallucinations. The scope is response-level scoring, not retrieval and not fact-checking against a knowledge base. The intended user is an engineer who has a model call in production or in evaluation and wants a per-response signal to threshold, rank or route on. The project is published by CVS Health and carries an Apache-2.0 licence.
Four scorer families and the cost each one buys
The README's comparison table is the most useful page of the documentation, because it frames the library as a set of trade-offs rather than a single method. Black-box scorers measure consistency across multiple generations from the same prompt. They work with any LLM, need no access to internal states, and are described as adding medium-to-high latency and high cost because they require multiple generations and comparisons. White-box scorers use token probabilities that the model already returns, so the table marks them minimal added latency and no added cost, with the caveat that they require access to token probabilities. LLM-as-a-judge scorers call a model to grade the response, adding low-to-medium latency and low-to-high cost depending on how many judges you configure. Long-text scorers work at claim level and are the most expensive row in the table, with high to very high latency and high cost. Ensemble scorers combine the others and inherit their cost profile. That last point matters: an ensemble is not a free accuracy upgrade, it is a way to spend several scorers' worth of budget on one answer.
The black-box path in practice
The README's worked example is short enough to read in full. You construct a LangChain chat model, pass it to BlackBoxUQ along with a scorer list and use_best=True, then await generate_and_score with a prompt list and num_responses=5, and call to_df() on the result. Two design decisions are visible in those four lines. First, the library is built on LangChain chat models rather than a bespoke provider client, so provider support is whatever LangChain supports. Second, use_best=True is described as mitigation: the uncertainty-minimized response is selected. So the same call can either score five candidate answers or return the one the scorer liked most. That is a meaningful difference from a pure measurement tool, and it means the scorer is in the output path, not just the logging path. The README also names the underlying methods for the black-box scorers, including discrete semantic entropy (Farquhar et al., 2024; Bouchard & Chauhan, 2025) and number of semantic sets (Lin et al., 2024; Vashurin et al., 2025; Kuhn et al., 2023), which is more citation than most libraries of this kind provide.
Installation and the API surface you actually touch
Installation is a single command from PyPI: pip install uqlm. The package requires Python 3.10 or later according to the badge in the README, and the example imports come from uqlm directly, as in from uqlm import BlackBoxUQ. The configuration knobs shown in the material are the llm argument, the scorers list, use_best, and the num_responses argument to generate_and_score. There is no config file and no CLI described in the README; this is a library you import. The repository links to per-family demo notebooks, including a black-box demo, and to hosted documentation at cvs-health.github.io/uqlm. Beyond the black-box example, the README does not spell out constructor signatures for the white-box, judge, ensemble or long-text classes, so plan on reading the API reference rather than guessing from the table. The README also notes the project uses uv and Ruff, which tells you the maintainers' toolchain but not what you need at runtime.
Where the approach breaks down
The white-box row carries an asterisk in the README's table, and the footnote says it does not apply to multi-generation white-box scorers, which have higher cost and latency. So the cheap path is only cheap for the single-pass variants. That is the sharpest limitation in the material. If your model does not expose token probabilities, or exposes them only through a provider that does not return them in the LangChain wrapper, the white-box family is unavailable and you fall back to the expensive rows. The second limitation is structural: consistency-based scoring assumes that a model's uncertainty shows up as variation across samples. A model that is confidently wrong in the same way every time will score high. Nothing in the README claims to catch that case, and no amount of num_responses fixes it. Third, the cost multipliers are real. Five generations per prompt is five times the inference, and the long-text row is described as high to very high latency, which rules it out for interactive use. Fourth, the library scores responses; it does not verify them against a source of truth, so a high confidence score is a statement about model behaviour, not about correctness.
How it compares with a guardrail or evaluation framework
The nearest alternative category is an LLM evaluation or guardrail framework, which typically scores outputs against a rubric, a reference answer, or a classifier trained to flag unsafe or off-topic content. The difference in approach is what supplies the signal. A rubric-based evaluator needs a definition of correct, either a labelled dataset or a judge prompt encoding your criteria, and it tells you whether an answer matched that definition. UQLM does not need a labelled set or a rubric. It derives the score from the model's own behaviour, either from sampling variance or from the token probabilities the model already produced. That makes it usable on day one with no annotation effort, and it makes it blind to errors the model is consistently confident about. A second practical difference is that uqlm sits inside the generation call, as the use_best example shows, whereas most evaluation harnesses run after generation on stored outputs. If your requirement is a pass/fail gate against written criteria, an evaluator is the better-shaped tool. If your requirement is a cheap per-response ranking signal with no labelling work, uqlm is aimed at that.
Maintenance, releases and licence
The release cadence visible in the material is roughly monthly, with v0.6.4 in late July 2026, v0.6.5 in mid August, and v0.6.6 in early September 2026, and the repository's last push date matches the latest release. Version numbers in the 0.6.x range indicate the API is still pre-1.0, so pinning a version in your requirements file is the sensible default. The project is not archived and has a CI workflow badge, so there is an active build. The licence is Apache-2.0, which permits commercial use and modification and includes an explicit patent grant; the usual obligations around preserving notices and stating changes apply, and if you redistribute it you should read the licence text rather than rely on this summary. There is also a JMLR 2026 paper and three additional linked publications, which gives you something to cite if the scoring method needs defending internally. The upgrade cost is mostly the usual pre-1.0 risk: constructor arguments and scorer names are the surface that would move, and the README's example is the part most likely to be copied into your codebase, so check it against the changelog on each bump.
Editorial conclusion
Adopt uqlm if you already have a LangChain chat model wired up and you need a confidence number per response rather than a rewritten answer; the white-box path is the only one that adds no extra LLM calls, so start there if your model exposes token probabilities. Do not adopt it as a fact-checker: the README describes confidence scores, not ground-truth verification, and the high-accuracy configurations multiply your inference bill. Before committing, confirm that your model returns token logprobs for the white-box scorers, and run BlackBoxUQ with scorers=["semantic_negentropy"], use_best=True, num_responses=5 on a sample of your own prompts to see the actual latency and cost multiplier on your workload.
Community notes