LLM-as-a-Verifier: pairwise scoring for agent trajectories, and where it stops being free

The problem: a pass/fail verifier throws away the ranking you need

If you generate five agent trajectories for one task, a unit test tells you how many passed. It does not tell you which of the failing ones was closest, and it does not distinguish a trajectory that fixed the root cause from one that patched a symptom and got lucky. That distinction matters for best-of-N selection, for progress tracking inside a single run, and for reward signals in reinforcement learning. The README frames the framework as providing fine-grained feedback for any agent without additional training, and the three named uses are test-time scaling, progress tracking and reinforcement learning. The intended user is someone running an agent harness that already produces a pool of candidates, or a sequence of steps, and who needs a scalar per candidate rather than a boolean. The README's own example is a five-trajectory pool for the task Fix the failing test in utils.py, with criteria for root cause and verification. That is the shape of the problem: you have more candidates than you can inspect by hand, and the cheap signal is too coarse.

How the score is produced: expectation over score-token logprobs

The mechanism is stated in three parts in the README: use fine-grained scoring granularity, take the expectation over the full logprob distribution of the LLM score tokens, and scale repeated evaluation and criteria decomposition. The practical consequence is a hard backend requirement. The quickstart says selection needs DEEPSEEK_API_KEY or VERTEX_API_KEY in a .env file, or an OpenAI-compatible server that returns logprobs, and gives vllm serve Qwen/Qwen3.5-9B with OPENAI_BASE_URL=http://localhost:8000/v1 as the example. A verifier that returns only text, or only a parsed integer, cannot feed this method. The fine-grained reward lives in the interval [0, 1]; the compare example prints 0.99994 and 0, and the select example prints scores of 0.73104, 0.38446 and 0.38449. Note those last two: two candidates that an integer scale would have tied are separated by 0.00003. Whether that gap is meaningful or is noise is something the README does not address, and it is the first thing I would want to check on my own data before trusting a ranking built from it.

select, compare and track: three entry points over one reward model

The public surface is small. compare takes a problem, two candidates and a criteria dict, and returns a pair of fine-grained rewards. select is described as built on that pairwise reward model, and returns an object with index, scores and ranking. track takes a problem, a list of step strings and a checkpoint_steps list, plus n_evaluations, and returns a score per checkpoint; the README example shows progress rising from 0.00106 after reading the problem statement to 0.99978 after the fix, with the jump concentrated between steps 3 and 4. That example is the clearest illustration of what fine-grained means here: a boolean reward would have shown zero progress for three of the five steps. The criteria argument is a plain dict mapping a criterion name to a question, and criteria decomposition is listed as one of the three scaling axes, so multiple criteria are expected rather than exceptional. n_evaluations is repeated evaluation per criterion, and the README's select example uses 4 while the track example also uses 4.

The pivot tournament: O(Nk) instead of O(N squared)

Ranking N trajectories by pairwise comparison naively costs a full round robin. The README states that select runs a Probabilistic Pivot Tournament to rank all N trajectories using O(Nk) pairwise verifications instead of a full O(N squared) round-robin, where k is the pivot count. The select example sets pivots=2 and notes pivots must be less than N, and that reduced verification cost is the reason. The README also states the trade-off directly: more pivots means more comparisons and higher accuracy. This is the most defensible design decision in the project, because it turns verification cost into a dial rather than a fixed property of the candidate pool. It is also the part most likely to bite you quietly. With pivots=2 and a candidate pool of five, the tournament is making very few comparisons, and the README does not describe what happens to ranking stability when two candidates are close. The published numbers carry error bars in one table (86.5% plus or minus 1.1%, 88.0% plus or minus 0.6%) and none in the other, which suggests the authors know variance matters but have not exposed a per-run uncertainty estimate through the API.

Getting it running, and the commands that actually matter

Installation is pip install llm-verifier, or pip install -e . from a clone. Credentials go in .env. The reproduction scripts are the fastest way to confirm the pipeline works before you point it at your own data: python scripts/run_bo3.py and python scripts/run_bo5.py for the Terminal-Bench 2.1 self-verification runs, which read trajectories from data/terminal_bench_2.1_trajs/ and need only DEEPSEEK_API_KEY. For the cross-benchmark table, python scripts/run.py with no argument lists the available benchmarks, and python scripts/run.py terminal_bench, swe_bench or medagentbench runs one by name. Tournament defaults are overridable on the command line, as in python scripts/run.py swe_bench --pivots 2 --n-evaluations 8 --seed 0 --max-workers 50. Benchmarks live in llm_verifier/benchmarks.py, which the README says is where you add or tweak one. For your own task the README describes a three-step path: copy trajectories into data/task_name_trajs/, update something the truncated README does not finish describing, and let Claude Code generate criteria and a runner. That last step is a Claude Code plugin hosted in a separate repository, llm-as-a-verifier/TurboAgent, so the adaptive path depends on a tool outside this package.

Version 0.2.0 and what it tells you about cost

The changelog summary in the README lists four items for 0.2.0: prefix-cache optimization reported as roughly 3.4x fewer uncached input tokens on trajectory-heavy benchmarks, a Terminal-Bench 2.1 self-verification benchmark, a deepseek-v4-flash verifier backend, and token accounting through llm_verifier.token_usage(). The prefix-cache item is the informative one. Trajectory-heavy verification means repeatedly sending long, largely identical prefixes (the problem statement, the criteria) with a short varying suffix (the candidate), which is exactly the access pattern prefix caching exists for. That the project needed this optimization is a fair signal that naive verification is expensive. The token accounting function is the corresponding control: it is the only documented way to see what a run cost. There are no releases retrieved for this repository, so the version number in the README is the only version marker I can point at, and I cannot confirm how 0.2.0 relates to any tagged release.

Where it is the wrong tool

The logprob requirement is the sharpest constraint. If your verifier is a hosted model that does not expose logprobs, or an internal classifier, or a rule engine, this framework's core method does not apply, and the OpenAI-compatible path exists precisely because you may need to self-host something like vllm serve Qwen/Qwen3.5-9B. Second, if a deterministic check exists, use it. A test suite that passes or fails is cheaper and more reliable than a sampled reward, and the framework's advantage appears only when the check is absent or partial. Third, the published results are all selection over an existing pool. The README's headline claims are about picking the best of N, with oracle columns showing the ceiling. Nothing in the supplied material shows the reward being used to train a policy, despite reinforcement learning being listed as a use. Fourth, the framework judges trajectories, and the self-verification table is the interesting case: the same model that produced the rollouts acts as the verifier, and selection still lands above Pass@1 (79.4% to 86.5% at best-of-3, 78.7% to 88.0% at best-of-5). That is a real result, but it is also a warning. Self-verification works well enough to select, and it is not evidence that the reward is calibrated for anything more demanding.

Alternatives, and the actual difference in approach

The obvious alternative is an execution-based verifier: run the candidate code against a test suite and rank by tests passed, then by runtime or diff size. That approach is deterministic, needs no API key, and cannot be gamed by a verifier model that shares the generator's blind spots. Its limitation is the one this project targets: it produces no signal on tasks without tests, and no graded signal on trajectories that all fail. A second alternative is a scalar LLM judge that returns an integer score. It is simpler, needs no logprob support, and works with any chat model. The difference is granularity, and the README's own numbers show it: the select example separates two candidates by 0.00003, which an integer scale collapses to a tie, and the track example shows a graded progress curve where an integer judge would likely report zero for the first three steps. The cost of that granularity is the logprob dependency and the repeated evaluations needed to make the expectation stable. A third option is to write the tournament yourself over an existing judge. The pivot scheme is described in enough detail to reimplement, but the surrounding pieces (criteria decomposition, repeated evaluation, token accounting) are the parts you would end up rebuilding.

Licence and what to verify before you commit

The repository is MIT licensed, which permits commercial use, modification and redistribution provided the copyright notice and permission notice are retained. I am not giving legal advice, and you should read the LICENSE file in the repository rather than this summary. Two licence-adjacent points are worth checking yourself. First, the package depends on verifier backends you supply credentials for, and the terms of DeepSeek, Vertex or your self-hosted vLLM endpoint are separate from the MIT grant. Second, the Terminal-Bench reproduction data ships in data/terminal_bench_2.1_trajs/, and the benchmark results in the README come from base models including GPT-5.5, Opus 4.5, Opus 4.6, Gemini 3 Flash and Claude Opus 4.8; the licence on the framework says nothing about the terms attached to those models or their outputs. On maintenance: the last push recorded for the repository is 2026-08-20, the README documents a 0.2.0 changelog, and no releases were retrieved, so there is no version history to read for a stability signal. The API surface shown here (select, compare, track, token_usage) is small, which limits upgrade exposure, but the benchmark definitions in llm_verifier/benchmarks.py are the file most likely to change under you if you build on the shipped benchmarks rather than on your own data.

Editorial conclusion

Adopt LLM-as-a-Verifier if you already generate multiple candidate trajectories per task and your verifier backend exposes logprobs, because the pairwise tournament in select is the part that is genuinely hard to reimplement well. Do not adopt it if you have a deterministic check available, such as a test suite that either passes or fails, since a logprob-scored reward adds model calls and an API key without adding information you do not already have. Before committing, verify two things against your own backend: that it returns logprobs for the score tokens, and that llm_verifier.token_usage() reports a number you consider acceptable at your chosen n_evaluations and pivots, since those two parameters are the only direct cost controls the framework exposes.

Official sources

1. Issues
2. License: MIT
3. llm-as-a-verifier/llm-as-a-verifier on GitHub
4. Project website
5. README