PandaProbe: A Self-Hosted Trace, Eval and Metrics Stack for Python Agents

The debugging gap PandaProbe targets in multi-step agents

A single agent turn can fan out into dozens of model calls, tool invocations and handoffs between subgraphs. When the final answer is wrong, the failure is usually somewhere in that middle layer: a tool returned a truncated payload, a router picked the wrong branch, a retry loop fired twice. Print statements and a generic application logger do not reconstruct that sequence, and a plain LLM observability tool that only records the outermost request will not either. PandaProbe's stated purpose is to let teams "collaboratively trace, evaluate, monitor, and debug AI agents", which places it in the agent-engineering category rather than general application monitoring. The intended user is a Python engineer who already has an agent running and now needs to see inside it, plus whoever on the team reads the dashboard to decide whether the last prompt change helped. The topic list on the repository names the frameworks it cares about: LangGraph, CrewAI, Claude Agent SDK, OpenAI Agents SDK. That is a narrow audience by design. If your agent is a single prompt-and-response call with no tool loop, the trace tree this project builds has nothing to show you.

How a trace moves from SDK to PostgreSQL in PandaProbe

The architecture diagram in the README splits the API into two planes. The management plane authenticates with an identity provider token (Supabase or Firebase are named) sent as an Authorization: Bearer header, then resolves users, organizations and projects through an identity service that reads and writes PostgreSQL. The data plane is separate and uses an X-API-Key header plus an X-Project-Name header, which the identity service resolves into a project context. That split matters operationally: agent processes carry a project-scoped key, not a user token, so a leaked key from a running agent does not grant access to organization membership or project settings. Ingestion is asynchronous. A POST /traces call does not write to PostgreSQL directly. The trace service pushes the payload onto Redis, the API returns 202 Accepted to the caller, and a Celery worker picks the job up and persists the trace and its spans. Reads work the other way: GET /traces and GET /sessions query PostgreSQL with filters and return a paginated response. Evaluations follow the same queue pattern. A POST /evaluations enqueues a job, a worker makes an LLM-as-a-judge call through LiteLLM, and the verdict and score are written back to PostgreSQL. The consequence of this design is that a 202 from the ingestion endpoint confirms the trace was queued, not stored. If the worker or Redis is down, the client sees success while the dashboard shows nothing.

Running the PandaProbe stack with start.sh

The README gives one path for self-hosting, and the only stated prerequisite is that Docker is installed and running. The commands are a clone, a cd, and the start script: git clone https://github.com/chirpz-ai/pandaprobe.git, then cd pandaprobe, then ./start.sh. Two endpoints come up afterwards. The dashboard is at http://localhost:3000 and the API reference, rendered by Scalar, is at http://localhost:8000/scalar. The services table lists six components with their ports: frontend (Next.js) on 3000, app (FastAPI) on 8000, a Celery worker with no port, a Celery beat scheduler with no port, PostgreSQL 16 on 5432, and Redis 7 on 6379, described as both broker and cache. The README does not enumerate environment variables, compose file names, or how the Supabase or Firebase identity provider is configured for a local run, so treat the first start as an exercise in reading the compose files and the CONTRIBUTING guide rather than a copy-paste setup. On the client side, the two headers you will need are X-API-Key and X-Project-Name for trace ingestion, and the docs site at docs.pandaprobe.com is where the README points for integration-specific instrumentation.

The LLM-as-a-judge loop and what it costs you

Evaluation in PandaProbe is not a static assertion library. The worker takes an enqueued eval job and makes a model call through LiteLLM, which is the abstraction layer that lets the same judge code target different providers. The result stored is a verdict plus a score. Two things follow from that. First, every evaluation consumes tokens from whichever model you point the judge at, and the README does not describe a budget cap, a sampling policy or a dry-run mode, so a job that fans out across many traces will bill accordingly. Second, judge quality is your problem, not the platform's. A verdict from a model call is a noisy label, and the repository material does not show a calibration workflow, a human review queue, or an agreement metric against labelled data. Teams that need deterministic scoring (exact match, schema validation, a unit-test-style assertion on tool arguments) will find the LLM-as-a-judge path indirect. The honest reading is that PandaProbe gives you the plumbing to run judges at scale, not a validated judge.

What the README does not tell you about running this in production

Several operational questions are unanswered in the supplied material. There is no mention of retention policies, so trace tables in PostgreSQL 16 will grow with every span until something deletes them, and nothing in the README describes what that something is. Backpressure is also unspecified: Redis is the broker, and a burst of traces beyond what the Celery worker can drain sits in the queue, with no documented dead-letter path or queue-depth alert. The identity service depends on Supabase or Firebase, which are external managed providers, so a genuinely air-gapped deployment needs a substitute that the README does not describe. The frontend and API expose ports 3000 and 8000 with no mention of TLS termination or reverse-proxy configuration, which is fine for a laptop and not fine for a shared internal host. None of this is disqualifying, but each item is work you take on when you choose self-hosting over the managed cloud option the project also offers.

PandaProbe against LangSmith and Langfuse

The nearest alternatives are LangSmith and Langfuse, and the difference is in where the data lives and how broad the framework support is. LangSmith is a managed service from the LangChain team, so you get hosted storage and UI without running PostgreSQL, Redis or a Celery worker, at the cost of sending traces off your network. Langfuse is the closer comparison: it is also open source and self-hostable, and its tracing model is built around OpenTelemetry-style spans with token and cost accounting as first-class fields. PandaProbe's differentiator in the supplied material is the explicit agent-framework integration list (LangGraph, CrewAI, Claude Agent SDK, OpenAI Agents SDK) and the bundled eval service with its Celery queue, rather than a generic span store you wire up yourself. The trade-off is maturity of the surrounding surface: Langfuse has a longer documented history of cost tracking and dataset management, while PandaProbe's README shows a trace, session and evaluation API and leaves cost accounting unmentioned. If you are already standardized on LangChain and want zero infrastructure, LangSmith is the lower-effort choice. If you want open source and cost dashboards, look hard at Langfuse before committing.

Release cadence, Apache-2.0 terms and upgrade exposure

The release history shows v0.5.3 on 2026-08-10, v0.6.0 on 2026-08-25 and v0.6.1 on 2026-08-26. A minor version followed by a patch one day later is normal for a project at this stage, and it also means the API surface is still moving. Anything you build against the trace or evaluation endpoints should be pinned to a version and re-checked on each minor bump, because the README gives no compatibility promise. The licence is Apache-2.0, which permits commercial use, modification and distribution provided you keep the licence and notice files and state significant changes; it also includes an explicit patent grant. That is a permissive arrangement, and it is not legal advice. Note the split between the open source repository and the hosted PandaProbe Cloud offering: self-hosting gives you the code under Apache-2.0, while the cloud free tier is a separate managed service with its own terms, and the README does not describe a data-processing agreement for either path. The maintenance cost of self-hosting is the six-service stack itself: PostgreSQL, Redis, the FastAPI app, the worker, the beat scheduler and the Next.js frontend all need upgrades, and the beat scheduler implies periodic jobs whose schedule the README does not document.

Editorial conclusion

Adopt PandaProbe if your agents are written in Python against LangGraph, CrewAI, Claude Agent SDK or OpenAI Agents SDK and your traces must stay inside your own network, since the whole stack runs from a single ./start.sh against Docker, PostgreSQL 16 and Redis 7. Do not adopt it if you need per-span token accounting and cost rollups, because the README and architecture diagram describe trace and span persistence without ever listing a cost or usage field. Before committing, verify three things against the docs at docs.pandaprobe.com: the exact OpenTelemetry or SDK instrumentation path for your framework, whether the eval service lets you register a custom judge instead of the built-in LLM-as-a-judge call, and what the Supabase or Firebase identity dependency means for a fully air-gapped deployment.

Official sources

1. chirpz-ai/pandaprobe on GitHub
2. License: Apache-2.0
3. Project website
4. README
5. Releases