LangWatch: An Open-Core Platform for Agent Testing and LLM Evaluation
The platform for LLM evaluations and AI agent testing
At a glance
- What is it?
- LangWatch combines trace-based observability, offline evaluation, and agent simulation in one Apache-2.0 core, with an optional AI gateway. This review covers its architecture, setup paths, and the trade-offs of running it locally.
- Who is it for?
- Adopt LangWatch if you need an integrated loop of tracing, dataset management, offline evaluation, and agent simulation without writing glue code, and you accept running a multi-service stack. Skip it if you only need lightweight tracing or you cannot tolerate the resource footprint of local evaluators.
- Can I use it commercially?
- Yes. Apache-2.0 is a permissive licence: you can use, modify and sell software built on it, as long as you keep its copyright and licence notices.
- Is it still maintained?
- Yes. The repository received new commits within the last day.
- What is it written in?
- Mainly TypeScript, according to GitHub's language statistics.
Answers come from the project's GitHub data, last synced on September 15, 2026, and from our analysis. They are not legal advice.
DEEP OPEN-SOURCE ANALYSIS
What LangWatch Actually Solves
LangWatch targets teams that build LLM-powered agents and need more than ad-hoc prompt testing. The README frames it as a platform for regression testing, simulations, and production observability, all without building custom tooling. The specific pain point is the disconnect between development-time evaluation and runtime monitoring. LangWatch tries to close that gap by putting traces, datasets, and evaluations in one loop. The intended user is an engineering team that ships agents with multiple tools and state, where a single bad decision can break a whole run. It is less useful for someone who only wants a simple prompt playground or a pure metrics dashboard, because the platform assumes you are willing to collect traces and manage datasets as first-class objects.
The Trace-to-Evaluate-to-Optimize Loop
The core mechanism is a closed loop: trace, then dataset, then evaluate, then optimize, then re-test. Traces come in through an OpenTelemetry/OTLP-native integration, which the README stresses as an open standard with no lock-in. Those traces feed into datasets, which are then used for offline evaluation. The evaluation step produces results that guide prompt or model optimization. The loop is meant to replace the usual sprawl of separate tools for logging, evaluation, and prompt management. The documentation describes this as a single loop with no glue code. That is a strong claim, but the architecture appears to support it by making datasets the intermediary between observed traffic and test cases. Whether that loop works smoothly across all frameworks depends on the quality of the OTLP exporters, which the README does not detail.
Agent Simulations Against the Full Stack
One of LangWatch's distinguishing features is end-to-end agent simulation. The README describes running realistic scenarios against the full stack, including tools, state, a user simulator, and a judge. The goal is to pinpoint where agents break, down to each decision. This is not a simple replay of recorded traces. It involves a user simulator that acts as an interlocutor and a judge that scores the agent's behavior. The simulation runs against your actual tools and state, which means it can catch failures that unit tests of individual prompts would miss. The trade-off is complexity: setting up a realistic simulator and judge requires defining scenarios that match your production use cases. The README links to a getting-started guide, but does not explain how to author those scenarios, so expect a learning curve.
Local Setup: One Command, Many Services
The fastest local path is npx @langwatch/server. That command installs uv, postgres, redis, clickhouse, an AI gateway binary, and the Langy assistant runtime into ~/.langwatch/, creates a .env with generated secrets, and starts all services in parallel. The dashboard appears at http://localhost:5560. Everything lives under ~/.langwatch/, so rm -rf ~/.langwatch is a clean reset. The README flags three environment variables that control optional components. LANGWATCH_ENABLE_LANGY defaults to true, adding about 45MB for a runtime that runs unsandboxed as your user. LANGWATCH_ENABLE_PRESIDIO defaults to false because it pulls in roughly 670MB of language models. LANGWATCH_ENABLE_LINGUA also defaults to false, adding about 95MB. The presidio model is larger than the rest of the evaluator environment combined, and the README notes that LangWatch's own secret redaction does not depend on it. That is a useful warning: you can skip the heavy PII evaluator unless you specifically need it.
Docker and Development Paths
If you prefer containers, the README shows a docker compose route: clone the repo, go to platform/app, copy platform/app/.env.example to platform/app/.env, then run docker compose up -d --wait --build. The service still lands on port 5560. For local development without full docker, you can start only the databases with docker compose up redis postgres opensearch, then run make install and make start in another terminal. The development path assumes you want to contribute, since it requires a source checkout and a Makefile. The docker compose file is the more reproducible option for a deployment, but the npx command is clearly the fastest for a trial. One thing to note: the npx server installs a lot of infrastructure, so the first run will take time and disk space. The README's table of optional evaluators gives you a way to trim that, but only for three components.
The AI Gateway: A Separate Binary with a Cost
LangWatch includes an AI Gateway that acts as an OpenAI/Anthropic-compatible proxy. It offers virtual keys, hierarchical budgets, inline guardrails, automatic fallback across providers, and Anthropic cache_control passthrough. The README claims a hot-path overhead of about 700 nanoseconds. That number is specific, but it comes from the project's own materials, and I have not benchmarked it independently. The gateway ships as a separate Go binary under services/aigateway/ and has its own Helm sub-chart at charts/gateway/. That separation means you can run the gateway without the full platform, or deploy it in a different scaling domain. The trade-off is operational complexity: you now have a second service to monitor, and its proxy behavior must be tested against your existing provider calls. The fallback feature is valuable, but it also introduces a failure mode where a fallback to a different provider might change model behavior, so you need to design guardrails carefully.
Licensing and Maintenance Realities
The repository is licensed under Apache-2.0, but the README badge says 'Apache 2.0 + Enterprise', indicating an open-core model. That means the core platform is open source, while some features likely fall under a separate Enterprise extension. The README does not enumerate which features are enterprise-only, so you must check the docs before building on a specific capability. Maintenance appears active, with recent releases across the Go SDK, TypeScript SDK, and skills packages. The last push date is September 2026, which suggests ongoing work. The project is written in TypeScript, but the gateway is Go, so contributions span two ecosystems. For upgrade cost, the npx installer manages dependencies under ~/.langwatch, but you are responsible for updating the server package and any evaluator models. The optional presidio model adds a large download that will recur on version changes. There is no mention of a migration path for datasets or traces between versions, so verify that before upgrading a production instance.
Where LangWatch Is the Wrong Tool
LangWatch is not a lightweight tracing library. If you only need to log a few prompts and see latency, the full platform is overkill. The local setup pulls in postgres, redis, clickhouse, and an optional gateway, which is a heavy footprint for a small project. The README's warning that the Langy assistant runs unsandboxed as your user is a genuine security consideration for local development. Running that on a shared machine or a CI environment is risky. Also, the evaluation ecosystem depends on external evaluator models, some of which are large. If your team lacks the disk space or the patience for a 670MB model download, you will stick with the defaults and miss some evaluator types. The platform's focus on agents with tools and state means it may not add much value for simple single-prompt classification tasks, where a spreadsheet and a few API calls might suffice.
A Real Alternative: OpenTelemetry Alone with Custom Evaluation
The most direct alternative is to skip LangWatch and build your own loop on top of OpenTelemetry. LangWatch itself is OTLP-native, so you could collect traces with any OTLP backend like Jaeger or Grafana Tempo, export them to a data warehouse, and write your own evaluation scripts. The difference in approach is that LangWatch provides the dataset management, the evaluation runner, and the simulation harness as integrated products. A custom build gives you full control over the evaluation logic and no dependency on LangWatch's UI or its evaluator models. The cost is the glue code that LangWatch claims to eliminate. For a team with strong data engineering skills, a custom pipeline might be more flexible, especially if you need to evaluate on proprietary metrics that LangWatch does not support. But you will lose the annotation queues and the prompt versioning integration out of the box.
Editorial conclusion
Adopt LangWatch if you need an integrated loop of tracing, dataset management, offline evaluation, and agent simulation without writing glue code, and you accept running a multi-service stack. Skip it if you only need lightweight tracing or you cannot tolerate the resource footprint of local evaluators. Before adopting, verify which evaluators you actually need, set LANGWATCH_ENABLE_PRESIDIO and LANGWATCH_ENABLE_LINGUA to false unless required, and confirm that your LLM provider and framework work with the OpenTelemetry integration. The project's open-core license means enterprise features are not in the Apache-2.0 core, so check the docs for what falls under the Enterprise extension before relying on any feature in production.
Community notes