# Tracely turns a failing agent trace into the test that blocks the next pull request > Tracely is a self-hostable Python platform that grades production agent traces, clusters failures into issues, promotes a broken run into a hermetic replayable case, and gates pull requests on it. The interesting part is the fail-to-pass contract: a promoted case must fail on the old code and pass on the fix. **jwuthri-cloud/Tracely-ai** — Trace-native CI/CD for AI agents: grade production traces, cluster failures, freeze them into replayable regression tests, and block PRs that regress. Self-hostable. - Repository: https://github.com/jwuthri-cloud/Tracely-ai - Website: https://tracely-ai.com - Stars: 978 · Forks: 91 - Language: Python - License: MIT - Published: 2026-10-20 · Updated: 2026-10-20 · Language: en - Canonical page: https://hysenlabs.com/projects/jwuthri-cloud-tracely-ai ## The recorded run is the test, and there is no dataset to maintain Tracely's central claim is a rejection of how evaluation tools usually work. The project describes the problem as observability stopping at the dashboard: you can see that your agent broke, and then what? Every eval tool asks you to hand-author a dataset, sitting down to invent questions, write ideal answers, and keep them current as the product changes. That dataset is a guess about what might break. The alternative is to start from what already happened. Production has handed you the real thing, a trace of the exact run that failed, with the exact input, the exact tool calls and the exact model responses. Tracely freezes that trace into a hermetic regression case and replays it on every pull request. Quality scores, failure clusters, suggested fixes, CI verdicts, trends and alerts are all derived from the trace. The comparison table in the project documentation makes the trade-off explicit. Where a dataset-first tool has a guess at fidelity to production, Tracely claims the exact failing run. Where a dataset tool spends live model calls to replay, Tracely claims zero cost using recorded tool and LLM fixtures. Where a regression moves a dashboard number, Tracely blocks the pull request. The audience is engineers running agentic systems in production, and the differentiator is a closed loop from observation to gate. ## Traces arrive over plain OTLP and get promoted to indexed columns The ingestion path is deliberately plain. Traces arrive over OTLP, and agent semantics are promoted to first-class indexed columns: agent.id, conversation.id, turn and step. That promotion is what lets runs group into conversation threads rather than remaining a flat span soup. Evaluators are columns on the trace table rather than a separate tab. Each one grades at conversation, run or span level and writes its verdict into the grid, with scores streaming in live over SSE as judges finish. Tokens, cost, latency, metadata and a rolling per-turn summary are columns in the same place. The waterfall shows agent, thinking, skill, generation and hand-off spans, with the failing span marked in red and its input and output beside it. Two alternative readings of a conversation ship with it. Replay walks the conversation event by event on a timeline, and Fleet renders the same conversation as a pixel-art office where each agent is a character, hand-offs are walks across the floor, and tools are objects on the wall. The project claims that second view is the fastest way to explain a multi-agent system to someone who does not read waterfalls, which is a specific claim about communication rather than about correctness. ## Failure clustering turns 31 broken runs into one issue with a count Online evaluators grade every run as it lands, using LLM-as-judge at conversation, run or span level, plus structural checks that need no model at all. Those structural checks matter for cost: a rule that can be decided without a model call is a rule that does not need an API key and does not need spend. Failures then cluster, structurally and semantically, so the project documentation puts it as 31 broken runs becoming one issue with a count rather than 31 rows to read. Each cluster can also suggest the evaluator that would have caught it, which is a useful pointer when the failure was previously invisible. The cluster view carries occurrence counts and a taxonomy, and the design dossier and guided tour live in the repository under design/ and guides/. The one-click promotion step is the bridge to testing: a failing trace becomes a hermetic case with recorded input and bundled tool and LLM output fixtures, and it carries a fail-to-pass contract. The case must fail on the old code and pass on the fix, or the promotion is not trusted. Multi-turn behaviour is handled differently, through scenarios that are either a scripted conversation or an adversarial goal a red-team model improvises against. ## The gate replays recorded fixtures, so no API keys are needed in CI The shipping step is the part that changes team behaviour. The suite replays in CI against recorded fixtures, which the documentation describes as deterministic, offline, with no API keys and no model spend. The tracely gate command exits non-zero, posts a commit status and upserts a pull request comment. A failing gate run shows the FAIL verdict, per-case verdicts and the judge's reason. Because the replay is offline, a pull request can be blocked without a CI runner holding provider credentials, which removes an obvious objection to putting LLM behaviour in the merge path. The alerting step is described as a flow with two halves: when and then. A rule fires when a gate fails, when a live conversation breaks on a judge, or on a fail-to-pass case. The repository exposes an OPS_ALERT_WEBHOOK setting, and the documentation notes that a blank value means log only, while the run still logs a selfcheck heartbeat. The delivery channels named in the project documentation are Slack, email and your own webhook, with the alert arriving rather than waiting for someone to open the dashboard. ## Self-hosting the stack means Postgres, ClickHouse, Redis and MinIO Tracely is self-hostable, and the docker-compose file at the repository root describes the full stack: API, worker, UI, Postgres, ClickHouse, Redis and MinIO. The compose file documents the one-liner it expects, with the comment noting the host port 8000 must be free, and giving the override for when it is busy: ```bash docker compose up -d --build --wait ``` ```bash TRACELY_BACKEND_PORT=8088 docker compose up -d --build --wait ``` The internal wiring is visible in the environment block. The API and the Alembic migrations use different Postgres drivers, asyncpg for the application and psycopg for Alembic, pointing at the same database. ClickHouse holds events and scores with a plain HTTP port 8123, Redis is the Celery broker and result backend, and S3_ENDPOINT_URL points at MinIO where raw OTLP blobs are stored as the source of truth. That last point is an architectural decision worth noticing: the blob store holds the raw payload, so ClickHouse holds the queryable projection. The compose file also sets a Celery pool of solo with a concurrency of one for local use, with a comment explaining that numba, UMAP and HDBSCAN are fork-fragile and that production sets prefork and two or more workers. Clustering, in other words, is what constrains the worker configuration. ## The local Makefile runs the three data stores, then the three services For development, the Makefile at the repository root is the entry point, and the target names are self-documenting. Infrastructure comes up first: ```bash docker compose up -d --wait ``` Then dependencies are synced. The install target runs uv across all workspace packages and extras, then installs the frontend dependencies with pnpm: ```bash uv sync --all-packages --all-extras ``` ```bash cd frontend && pnpm install ``` Migrations are split three ways. ClickHouse migrations run as a module, Postgres migrations use Alembic from the backend directory, and a third target creates the checkpoint tables for LangGraph so judge conversations are durable: ```bash uv run python -m tracely.infrastructure.clickhouse.migrations ``` ```bash cd backend && uv run alembic upgrade head ``` ```bash uv run python -m tracely.infrastructure.llm.checkpointer ``` The three services then run separately, with the API on port 8000 and the worker as a Celery worker on the solo pool: ```bash uv run uvicorn tracely.api.main:app --reload --port 8000 ``` ```bash uv run celery -A tracely_workers.worker worker --pool=solo --loglevel=info ``` A seed target creates the default project and an ingest key named tracely_dev_key, and a demo target populates traces, clusters, cases and gates in one pass. ## A blank OpenRouter key quietly degrades every judge to a no-op The .env.example file contains a setting that deserves careful reading. Every judge, failure-intelligence and meta-analysis call goes through LangChain create_agent on OpenRouter, with embeddings coming from its OpenAI-compatible endpoint, and the comment states that the OpenRouter key alone is enough. The failure mode is the problem: a blank key means the pipeline still runs, but evaluators degrade to no-ops. That is a significant design choice, and a risky one. A pipeline that starts, accepts traces, draws a dashboard and quietly stops evaluating is worse than one that refuses to start, because the failure is visible in the UI rather than in the logs. Anyone running this in production should set OPENROUTER_API_KEY and then confirm the judges are producing verdicts, rather than assuming a running system is a working one. Legacy direct-endpoint fallbacks such as OPENAI_API_KEY and LLM_JUDGE_API_KEY exist but are used only when no OpenRouter key is set. The other notable defaults are about housekeeping rather than evaluation. ClickHouse has its own telemetry tables that ship with no TTL and are never read by Tracely, so a nightly sweep caps them, and CH_SYSTEM_LOG_TTL_DAYS defaults to 3 with 0 turning the sweep off. The same care applies to the OpenRouter key that a workspace sets in Settings, which is encrypted at rest. ## An active repository with no tagged releases yet The repository is not archived and the last push was on 2026-10-19, so the code is moving. There is, however, no GitHub release published, which means there is no versioned artifact to pin, and the pyproject.toml at the workspace root is deliberately a non-package with the version 0.0.0 and the note that it is a uv workspace root and not itself a package. The real packages are the workspace members backend, workers and sdk, plus a separate tracelyai-sdk published on PyPI. Python 3.12 or newer is required. The repository carries a uv.lock, a Dockerfile.backend, a docker-compose.yml, a Makefile, a server.json and a skills directory, along with a .agents directory and a .claude directory of vendored agent-skill packs, which the ruff configuration explicitly excludes from linting because the project does not own that third-party code. The licence is MIT. The ruff configuration in pyproject.toml records a policy choice that is worth reading as a maintainer signal: the classic rule set is pinned explicitly because a Dependabot patch bump from Ruff 0.15 to 0.16 switched on a dozen new rule families and produced 465 findings nobody wrote, and the comment states that a lint-policy change should be a commit rather than a side effect of a version bump. A project that pins its lint policy against tool churn is one that expects to be maintained for a while. ## Conclusion Tracely fits teams running agentic systems in production who already emit OTLP traces and want a failing production run to become a CI gate without hand-writing a dataset. It does not fit a team that has no production traffic yet, since the whole chain starts from traces that already landed. Before adopting it, confirm that your agents emit OTLP with the agent.id, conversation.id, turn and step attributes, set OPENROUTER_API_KEY in .env so judges are not silently degraded into no-ops, and read the fail-to-pass rule, because a case that cannot be validated will not be trusted as a regression test. ## FAQ ### What does Tracely do with production agent traces? Tracely grades every trace as it lands using online evaluators, clusters the failures into issues with occurrence counts, and can promote a failing trace into a hermetic regression case with recorded fixtures and a fail-to-pass contract. The suite then replays in CI and blocks the pull request that would ship the failure again. ### How do I self-host Tracely locally? The docker-compose file at the repository root starts the full stack with `docker compose up -d --build --wait`, which brings up the API, worker, UI, Postgres, ClickHouse, Redis and MinIO. For development, the Makefile provides infra-up, install, migrate, backend, workers and frontend targets on top of that. ### Does Tracely need API keys to run its evaluators? Yes, in practice. Every judge call goes through OpenRouter, and the .env.example states that a blank OPENROUTER_API_KEY means the pipeline still runs but evaluators degrade to no-ops. Replaying recorded cases in CI needs no keys at all, since it uses recorded fixtures. ### Is Tracely released under an open source licence? Yes, the project is MIT licensed, and it is designed to be self-hosted. The workspace root pyproject.toml is a uv workspace root rather than a published package, while the SDK is published separately on PyPI as tracelyai-sdk. ## Sources - [Issues](https://github.com/jwuthri-cloud/Tracely-ai/issues) - [jwuthri-cloud/Tracely-ai on GitHub](https://github.com/jwuthri-cloud/Tracely-ai) - [License: MIT](https://github.com/jwuthri-cloud/Tracely-ai/blob/main/LICENSE) - [Project website](https://tracely-ai.com) - [README](https://github.com/jwuthri-cloud/Tracely-ai/blob/main/README.md) --- Hysen Labs editorial analysis, written from the project's own repository and release notes. Cite the canonical page: https://hysenlabs.com/projects/jwuthri-cloud-tracely-ai