Tracely: turning failing agent traces into CI gates

The dataset problem Tracely is built around

Evaluation tooling for LLM applications usually starts from a dataset. Someone sits down, writes questions, writes ideal answers, and keeps the file current as the product changes. The README calls this out directly: every eval tool asks you to hand-author a dataset, and that dataset is a guess about what might break. The guess is the weak point. It reflects what the team imagined, not what users actually did.

Tracely inverts the source of truth. Its claim is that the recorded run is the test. A production trace already contains the exact input, the exact tool calls and the exact model responses of a run that failed. Tracely grades traces as they land, groups the failures, and freezes a failing run into a regression case. There are no hand-authored datasets anywhere in the loop. Everything the product shows (quality scores, failure clusters, CI verdicts, alerts) is derived from traces.

The audience is teams running agents in production who already have some tracing in place and who feel the gap between seeing a failure on a dashboard and preventing it from shipping again. It is not aimed at teams who have not instrumented their agents at all, because there is nothing to grade without a trace.

How a trace becomes a blocking test

The mechanism has five stages, and the README maps them to five pages in the app: production trace, failure detection, regression test, CI gate, alert.

Traces arrive over plain OTLP. Tracely promotes agent semantics (agent.id, conversation.id, turn, step) to first-class indexed columns, so runs group into conversation threads rather than a flat list of spans. The waterfall renders agent, thinking, skill, generation and hand-off spans, with the failing span marked and its input and output alongside it.

Grading happens online, as runs land. Evaluators are columns on the trace table rather than a separate view. Each one grades at conversation, run or span level and writes a verdict into the grid. Scores stream in over SSE as judges finish. Alongside LLM-as-judge evaluators there are structural checks that need no model at all, which matters for cost and for determinism.

Failures then cluster, both structurally and semantically, so that many broken runs become one issue with an occurrence count. The README gives the example of 31 broken runs collapsing into one issue. A cluster can also suggest the evaluator that would have caught it, which is a way of turning a discovered failure mode into a permanent check.

Promotion is the step that turns observation into testing. One click converts a failing trace into a hermetic case: recorded input, tool outputs and LLM outputs bundled as fixtures, plus a fail-to-pass contract. The case must fail against the old code and pass against the fix, or the promotion is not trusted. For multi-turn behaviour, Tracely uses scenarios instead: either a scripted conversation or an adversarial goal that a red-team model improvises against.

What runs in CI, and why it costs nothing

The suite replays against the recorded fixtures. That is the whole reason the README can claim $0 replay cost: the case carries the tool and LLM outputs with it, so CI does not call a live model and does not need API keys. Replay is deterministic and offline.

The gate command is tracely gate. According to the README it exits non-zero on failure, posts a commit status, and upserts a PR comment. The comment carries per-case verdicts and the judge's reason for a failure, which is the difference between a red X and something an engineer can act on.

The alerting layer is drawn as a flow with two halves. The when side covers a gate failing, a live conversation breaking on a judge, a failure mode appearing that nobody has seen before, or a rate crossing a line. The what-happens side includes conditions that gate the rest of the flow, Slack, email and a custom webhook. Notifications are part of the same product rather than a separate integration to wire up.

Getting the stack running

The README gives two paths. The first is a one-click deploy on Railway that stands up the whole stack: API, worker, UI, Postgres, ClickHouse, Redis and MinIO. That is a lot of infrastructure for a tool that markets itself as a CI gate, and it is the honest shape of the product: Tracely is a self-hosted platform, not a CLI you drop into a workflow file.

The second path is the Python package, tracely-ai on PyPI, which requires Python 3.10 or above. The README does not spell out the exact install command or the full set of environment variables in the portion available here, so treat the package name and the version floor as the confirmed facts and read the docs site for the rest.

The repository also ships an agent skill, described under the heading teach your coding agent Tracely, plus a guided tour in guides/OVERVIEW.md, a two-minute demo in guides/DEMO.md, and a design dossier under design/. Those are the files to read before deciding, because they are where the operational detail lives.

One configuration point is worth flagging early. The indexed columns depend on specific span attributes: agent.id, conversation.id, turn and step. If your instrumentation does not emit those names, the conversation threading and the agent waterfall will not have the structure the product assumes.

Where the trace-as-test approach breaks down

The central bet is that the recorded run is the test, and that bet has a boundary. A fixture replays recorded tool and LLM outputs. That makes CI free and deterministic, and it also means the case cannot detect a regression that lives in a live dependency. If a downstream API changes its response shape, or a model provider updates a model behind the same name, the recorded fixture will not notice. The case tests your orchestration against frozen data, not the world.

The fail-to-pass contract is the mitigation, and it is also a constraint. A promoted case must fail on the old code and pass on the fix. Cases that do not satisfy that are not trusted, which is the right call, but it means promotion is not a fire-and-forget click. Someone has to land the fix for the contract to be verifiable.

Clustering is the other soft spot. Grouping failures structurally and semantically is what keeps the issue list readable, and any clustering heuristic will sometimes merge two distinct bugs into one issue or split one bug across several. The README does not describe the clustering algorithm in the material available here, so the accuracy of that step is something to test against your own failure data rather than assume.

Finally, the deployment footprint is real. Postgres, ClickHouse, Redis and MinIO is a platform to operate. A team that wants a single binary or a hosted SaaS will find this heavier than expected.

How it differs from LangSmith and Braintrust

The closest alternatives are dataset-first evaluation platforms such as LangSmith and Braintrust. Those products also support tracing and also run evaluations in CI, so the difference is not capability, it is where the test cases come from.

In a dataset-first tool you author the cases. You decide what good looks like, you write the examples, and you maintain them as the product changes. That gives you control and lets you test behaviour you have not seen fail yet. It also means your CI suite covers the failures you anticipated.

Tracely takes the opposite position. Cases are promoted from real failing traces, so the suite grows from observed production behaviour rather than from a planning session. You get fidelity to the exact failing run, at the cost of only testing what has already gone wrong. A team with a mature, curated eval dataset and a good process for maintaining it will not gain much from switching. A team whose eval dataset went stale three releases ago is the target user.

The replay model is the second difference. Dataset-first tools typically replay against live model calls, which costs money per CI run and introduces nondeterminism. Tracely replays against recorded fixtures at no model spend. The trade is coverage of live behaviour for reproducibility and cost.

Licence, upkeep and what to check before adopting

Tracely is MIT licensed. That is permissive: you can use it commercially, modify it and redistribute it, provided the copyright notice and licence text are preserved. It also means there is no vendor obligation to you. Nothing here is legal advice, and if you are embedding the project in a product you should read the LICENSE file and, where it matters, talk to a lawyer.

Maintenance cost is the part the README understates. The package is Python 3.10 or above and the self-hosted stack includes four stateful services. Upgrades mean coordinating the API, the worker, the UI and the datastores, and ClickHouse schema changes are the kind of thing that needs a migration plan rather than a pull and restart. The repository lists no releases in the material available here, so there is no version history to inspect for upgrade cadence. Check the tags and the CI workflow before you commit.

The first thing to verify is instrumentation. Confirm that your spans carry agent.id, conversation.id, turn and step. Without them the conversation threading that makes the trace view useful does not exist, and you are back to a flat span list.

The second is the fail-to-pass contract. Promote one real failure end to end, confirm the case fails on the pre-fix commit and passes on the fix, and confirm tracely gate exits non-zero and writes the PR comment. If that loop works on one case, the rest is volume. If it does not, the gate is decoration.

Editorial conclusion

Adopt Tracely if your agents already emit OTLP traces and you want production failures to become blocking CI tests without paying for live model calls. Do not adopt it if you have no trace pipeline, or if your test needs live tool or model behaviour that recorded fixtures cannot reproduce. Before committing, verify that your OTLP spans carry the agent.id, conversation.id, turn and step attributes the README lists, and confirm that a promoted case actually fails on the old code and passes on the fix, since that fail-to-pass contract is what the gate trusts.

Official sources

1. Issues
2. Jwuthri/Tracely-ai on GitHub
3. License: MIT
4. Project website
5. README