Judgeval: tracing and agent judges for LLM applications

The gap Judgeval targets: production traces that never turn into fixes

Most agent debugging stops at a trace viewer. You find a bad run, read the span tree, change a prompt, and hope. The evidence that motivated the change is not stored anywhere reusable, so the next regression looks like a new problem. Judgeval's README frames the product as a "Continuous-Improvement Stack for Agents" with three verbs: detect failures, triage root causes, ship fixes backed by production data. The unit that carries the fix is the judge, a prompt-based scorer that produces what the README calls structured behaviors: scored, labeled outputs describing how the agent acted. Those behaviors accumulate into what the project describes as a searchable record of agent behavior over time. The audience is teams running LLM-powered applications in production, particularly agent-style systems with tool calls and multi-step reasoning, where a single request can fail in several distinct ways and a flat success rate hides which one. If you are still prototyping and have no traffic, the tracing half is useful but the judge half has nothing to score.

How tracing and judging fit together

The mechanism has two layers. The instrumentation layer is OpenTelemetry-based. You call Tracer.init(project_name=...) once, then decorate functions with @Tracer.observe(span_type="tool") or @Tracer.observe(span_type="agent"). The README states that this automatically captures inputs, outputs, and LLM token usage, and that being built on OpenTelemetry gives compatibility with existing observability stacks. The wrap() helper instruments a client, shown with OpenAI, so calls made through the wrapped client are captured without decorating each call site. The evaluation layer sits on top of the resulting traces. Judges are prompt-based scorers you define, and they can run against live production traffic or be replayed on historical traces. Replay is the part that changes workflow: instead of validating a prompt change on a fresh sample, you point the judge at traces you already collected and compare outcomes. Online monitoring runs judges server-side against live traffic, which the README says happens with no latency impact, and detected behaviors surface as structured signals that can be routed to Slack alerts. The data flow is therefore trace in, behavior out, with behaviors as the queryable artifact rather than raw spans.

Getting it running: install, credentials, and the two-line setup

Installation is a single pip command: pip install judgeval. Two environment variables carry credentials: JUDGMENT_API_KEY and JUDGMENT_ORG_ID. The README's quickstart then does Tracer.init(project_name="my-project") and client = wrap(OpenAI()), after which decorated functions and wrapped client calls are traced. Note that the API key and org ID are required for the basic path, which means trace data is sent to Judgment's service rather than staying in your process. That is a deployment decision, not just a config detail. The CLI is a separate repository, linked as JudgmentLabs/cli, and the README describes it as covering agents, traces, judges, behaviors and evaluations from the terminal, including querying trace history, deploying judges, inspecting detected behaviors, and running evals against production data. There is also an MCP server for connecting Judgment to MCP-compatible tools and IDEs. For JQL, the README is specific about scoping: queries use the same API key, organization and project configuration as the rest of Judgeval, and tenant identifiers are not part of the query payload. A minimal query looks like client.query(spans().rows(), trace_ids=["trace-123"]) after constructing Judgeval(project_name="my-project").

JQL's trace_ids and session_ids rules are stricter than they look

JQL is the query surface over the trace store, and its scoping rules are where the design has teeth. trace_ids and session_ids are mutually exclusive, and they live outside the JQL query object rather than inside it. Trace IDs narrow the query directly. Session IDs behave differently: the service resolves them within the authenticated organization and project, then narrows every part of the query to their traces. If no session resolves, the request fails rather than falling back to the whole project. That last decision is the right one for evaluation work, because a silent fallback would return a project-wide result set that looks like a scoped one and would quietly corrupt any judge comparison built on it. The README also notes that both options work with present() and discover(), and that offline_traces() and offline_spans() query traces captured by an offline test. That offline path matters for the replay workflow: it is how you separate traces produced by a test run from traces produced by real traffic, which you want to keep apart when you are comparing judge output before and after a change. The README does not spell out the full JQL grammar, so treat the query builder as something to learn from the linked docs rather than from the quickstart.

Where Judgeval is the wrong tool

The clearest limitation is architectural: the primary path assumes a hosted backend. Tracer.init and the credential pair point at Judgment's service, and online monitoring is described as server-side scoring, so the judge execution you care about most does not happen inside your process. Teams with data residency constraints, or teams that cannot send prompts and completions to a third party, will find that the tracing and judging halves are not separable in the documented setup. Second, the SDK is Python. The README lists integrations with OpenAI, Anthropic, Google GenAI, Together AI, LangGraph, OpenLit and Claude Agent SDK, but a TypeScript service would be integrating at the HTTP or MCP boundary rather than through the SDK. Third, prompt-based judges are only as good as the prompts. The README describes judges as producing scored, labeled outputs but does not describe calibration, agreement measurement against human labels, or drift detection for the judges themselves. If your evaluation problem is that you do not yet know what correct behavior looks like, adding a judge does not answer that question; it encodes your current guess and then scores traffic against it. Finally, offline-only evaluation is a second-class path here. If your workflow never touches production traffic, a library that runs scorers locally over a dataset is a better fit than a platform built around a trace store.

Alternatives and the actual difference in approach

The obvious comparison is LangSmith, which also pairs tracing with evaluation and is named in adjacent ecosystems, though Judgeval's own integration list names LangGraph rather than LangSmith, and the README does not discuss the relationship. The substantive difference is where the evaluation artifact lives. In a dataset-centric tool, you curate examples into a dataset, run an experiment, and get a scored report per experiment; the dataset is the thing you version and the trace is an input to it. Judgeval inverts that. Behaviors are attached to traces, and the trace store is the primary record, with replay as the way you re-score history after changing a judge or a prompt. That suits teams whose failure cases arrive as production traffic and are hard to enumerate in advance. It suits them less if you already have a stable golden set, because you will be rebuilding that set as queries over traces instead of as a file. A second alternative is a plain OpenTelemetry collector plus your own scoring script. Judgeval's tracing is already OpenTelemetry-based, so the instrumentation half is not a lock-in; what you would be replacing is the judge definition format, the behavior store, and the server-side online monitoring. Whether that is worth it depends on how much you value running judges on live traffic without adding latency to your own service.

Maintenance, versioning and licence

The release history is worth reading before you pin a version. The most recent releases listed are v1.3.2 from 2026-09-01, v1.3.1 and v0.32.2 both from 2026-08-03. A 0.x line and a 1.x line receiving releases on the same day suggests either parallel maintenance or a migration in progress, and the material here does not say which. For a library that sits inside your request path through wrap() and decorators, that ambiguity is a real upgrade cost: you need to know whether v0.32.2 is a backport for teams that have not moved, or a legacy branch being wound down. The README does not include a changelog, a deprecation policy, or a compatibility statement between the two lines, so that is something to confirm in the release notes before pinning. The licence is Apache-2.0, which permits commercial use and modification and includes an explicit patent grant; it also means any modified files you distribute carry attribution and notice requirements. That is a description of the licence terms, not legal advice, and if you plan to redistribute a modified SDK or embed it in a product, have counsel read the notice and patent clauses rather than relying on a summary. The CLI lives in a separate repository, so it versions independently of the SDK and needs its own pinning decision.

Editorial conclusion

Adopt Judgeval if your agent already runs in production and you want a trace store you can score repeatedly with prompt-based judges, and if sending trace data to Judgment's servers is acceptable. Do not adopt it if you need an offline-only evaluation runner, if your stack is not Python, or if you object to a hosted control plane for traces and judges. Before committing, verify three things against the docs: which span types the judge definitions actually consume, whether the Slack alerting path covers the behaviors you care about, and what the retention and export story is for traces already sent. Then run one replay over historical traces and compare the judge output against a hand-labeled sample before you trust it on live traffic.

Official sources

1. JudgmentLabs/judgeval on GitHub
2. License: Apache-2.0
3. Project website
4. README
5. Releases