LangWatch: An Open-Core Platform for Agent Testing and LLM Evaluation

What LangWatch Actually Solves

LangWatch targets teams that build LLM-powered agents and need more than ad-hoc prompt testing. The README frames it as the open-source platform for AI in production: trace, test, route and govern every LLM call in your company, from the agents you build to the coding assistants your engineers use. The specific pain point is the disconnect between development-time evaluation and runtime monitoring. The intended user is an engineering team that ships agents with multiple tools and state, where a single bad decision can break a whole run. It is less useful for someone who only wants a simple prompt playground or a pure metrics dashboard, because the platform assumes you are willing to collect traces and manage datasets as first-class objects.

The Trace-to-Evaluate-to-Optimize Loop

The README groups observability, agent testing, evaluations and prompt management under LLM Ops, and lists OpenTelemetry among its framework integrations alongside LangChain, LangGraph, Vercel AI SDK, Mastra, CrewAI, Google ADK and DSPy.

Agent Simulations Against the Full Stack

The README says you can create simulation tests for evaluating or benchmarking your agents and voice agents, and links to the agent-testing documentation for details. Expect to read those docs before writing scenarios.

Local Setup: One Command, Many Services

The fastest local path is npx @langwatch/server. The README says only Node.js 24 or newer is required, on macOS or glibc Linux, and points to the Self-hosting LangWatch documentation for production.

Docker and Development Paths

For contributors, the README says to read CONTRIBUTING.md and then run make quickstart. The Makefile's help lists quickstart presets such as all-local (local ClickHouse, Postgres, Redis, app and workers, no NLP), all-local-nlp, dev-infra, frontend-only, migration and full-local, plus make service svc=<name> for running a Go service such as aigateway. For production, the README points to the self-hosting documentation.

The AI Gateway: A Separate Binary with a Cost

LangWatch includes an AI Gateway that the README describes as one OpenAI and Anthropic compatible endpoint with virtual keys, budgets and routing, and it suggests creating virtual keys with budgets to hand out to customers or employees. The Makefile runs the gateway as a Go service (make service svc=aigateway), so using it means operating a Go service alongside the TypeScript platform, and its proxy behavior must be tested against your existing provider calls.

Licensing and Maintenance Realities

The repository is licensed under Apache-2.0, but the README says the enterprise modules under enterprise/ need a commercial license in production, and the SDKs are MIT. It points to the Editions and licensing page to check which features fall under which license. Maintenance appears active, with langwatch v3.22.0 released on 2026-11-02. The last push date is 2026-11-02, which suggests ongoing work. The project is written in TypeScript, but the gateway is Go, so contributions span two ecosystems. Verify the upgrade path in the self-hosting documentation before upgrading a production instance.

Where LangWatch Is the Wrong Tool

LangWatch is not a lightweight tracing library. If you only need to log a few prompts and see latency, the full platform is overkill. The contributor quickstart presets run ClickHouse, Postgres, Redis, the app and workers, which is a heavy footprint for a small project. The platform's focus on agents with tools and state means it may not add much value for simple single-prompt classification tasks, where a spreadsheet and a few API calls might suffice.

A Real Alternative: OpenTelemetry Alone with Custom Evaluation

The most direct alternative is to skip LangWatch and build your own loop on top of OpenTelemetry. LangWatch lists OpenTelemetry among its integrations, so you could collect traces with any OTLP backend like Jaeger or Grafana Tempo, export them to a data warehouse, and write your own evaluation scripts. The difference in approach is that LangWatch provides the dataset management, the evaluation runner, and the simulation harness as integrated products. A custom build gives you full control over the evaluation logic and no dependency on LangWatch's UI or its evaluator models. The cost is the glue code you write yourself. For a team with strong data engineering skills, a custom pipeline might be more flexible, especially if you need to evaluate on proprietary metrics that LangWatch does not support. But you will lose the built-in agent testing and prompt management.

Editorial conclusion

Adopt LangWatch if you need an integrated loop of tracing, dataset management, offline evaluation, and agent simulation without writing glue code, and you accept running a multi-service stack. Skip it if you only need lightweight tracing or you cannot tolerate the resource footprint of local evaluators. Before adopting, verify which evaluators you actually need, and confirm that your LLM provider and framework work with the OpenTelemetry integration. The project's open-core license means enterprise features are not in the Apache-2.0 core, so check the Editions and licensing page for what falls under the enterprise modules before relying on any feature in production.

Frequently asked questions

Official sources

1. langwatch/langwatch on GitHub
2. License: Apache-2.0
3. Project website
4. README
5. Releases