Open-source alternatives to LangSmith: 4 LLM tracing and evaluation tools compared
Moving off LangSmith means the trace collector, the store and the evaluation runner live on infrastructure you own, so the trade is a hosted service you do not operate for a database, a queue and an upgrade schedule that are yours. Most readers should start with langfuse/langfuse, the one candidate here that covers tracing, prompt management and evaluations in a single self-hosted stack.
What LangSmith does, and why people look elsewhere
LangSmith is a hosted service that records the traces, prompts and datasets of an LLM application and scores them against test sets, so a team can compare changes before shipping them.
- every prompt, completion and trace is stored on the vendor's servers
- the tracing SDK reports to the vendor's collector, so an application needs a path out to it
- the evaluation datasets and their scores live in the vendor's project, and what leaves is a file rather than the same store
The candidates at a glance
| Project | Licence | Maintenance | Language | GitHub stars | Install |
|---|---|---|---|---|---|
| langfuse/langfuse | Custom licenceCustom licence: read the LICENSE file | Commits in the last six monthsLast push October 11, 2026 | TypeScript | 35,650 | See analysis |
| Helicone/helicone | Apache-2.0Permissive: commercial use allowed | Commits in the last six monthsLast push October 20, 2026 | TypeScript | 6,235 | See analysis |
| Arize-ai/phoenix | Custom licenceCustom licence: read the LICENSE file | Commits in the last dayLast push October 22, 2026 | Python | 11,953 | See analysis |
| traceloop/openllmetry | Apache-2.0Permissive: commercial use allowed | Commits in the last six monthsLast push October 19, 2026 | Python | 7,503 | Command documented |
Each alternative in detail
langfuse/langfuse🪢 Open source AI engineering platform: LLM evals, observability, metrics, prompt management, playground, datasets. Integrates with OpenTelemetry, LangChain, OpenAI SDK, LiteLLM, and more. 🍊YC W23
- Best for
- You want one self-hosted platform to take over the tracing, prompt management, evaluation and dataset work LangSmith does for you, and you are willing to run a ClickHouse-backed stack on machines you control.
- What you give up
- The licence is a custom one GitHub cannot classify, and our analysis of Langfuse says it is a poor fit for anyone who cannot accept a licence that is not OSI-approved, so the LICENSE file has to be read before a deployment. Self-hosting also means you own the database, the supporting services and the upgrade path, and our analysis asks you to confirm that the self-hosted build carries the features you need, since some exist only in the hosted offering.
Helicone/helicone🧊 Open source LLM observability platform. One line of code to monitor, evaluate, and experiment. YC W23 🍓
- Best for
- You run production LLM traffic through an OpenAI-compatible SDK and want the gateway, the request logs, cost and latency tracking and prompt versioning to live in one Apache-2.0 stack you host yourself.
- What you give up
- The stack is the largest on this page: the README describes a web frontend, a proxy-logging worker, a dedicated Jawn server for collecting logs, Supabase for the application database and auth, ClickHouse for analytics and Minio for object storage, and our analysis of Helicone tells you to skip it if you have no tolerance for operating five services. The production Helm chart is not publicly downloadable, since the README says to contact the team for enterprise access, and manual deployment is marked as not recommended.
Arize-ai/phoenixAI Observability & Evaluation
- Best for
- You already emit OpenTelemetry and want tracing, LLM-as-a-judge evaluations, versioned datasets and experiments inside one self-hosted service that stays neutral about which model provider you use.
- What you give up
- The repository reports a licence GitHub cannot classify, and our analysis of Phoenix says to verify that this status meets your legal requirements before you deploy rather than assume a permissive licence. It also expects a server component that someone has to run and update, and the OpenInference instrumentation has to cover the exact framework versions you run, so a mismatched agent framework means a different integration job.
traceloop/openllmetryOpen-source observability for your GenAI or LLM application, based on OpenTelemetry
- Best for
- Your LLM calls already speak OpenTelemetry somewhere else and you only need spans, so the traces can go to a backend you already run instead of standing up a full evaluation platform.
- What you give up
- It gives you traces, not judgements about model output, so the dataset registry, the scoring functions and the side-by-side comparison of runs that LangSmith provides are yours to build. The README also asks you to pick a destination from its tested list rather than assuming any collector-compatible backend works, and on a local run you have to disable batch sending or spans sit in a queue while you debug an empty screen.
- First install step
- Full setup guide
pip install traceloop-sdk
Four different answers to the same trace store
LangSmith records the traces, prompts and datasets of an LLM application, scores them against test sets and keeps the results side by side. The reason teams look elsewhere is specific rather than ideological: every prompt, completion and trace is stored on the vendor's servers, the tracing SDK reports to the vendor's collector, so the application needs a path out to it, and the evaluation datasets with their scores live in the vendor's project instead of in files the team holds.
The four candidates answer that problem in four different ways, and the difference between them matters more than any feature list.
langfuse/langfuse has the closest shape to what LangSmith does. The README describes observability, prompt management with version control, evaluations, datasets and an LLM playground in one platform, and it names OpenTelemetry, LangChain, the OpenAI SDK and LiteLLM among the integrations. It also says the project has been part of ClickHouse since January 2026 and runs on the ClickHouse open source database.
Helicone/helicone is a different animal: an AI gateway that sits in front of your model calls and records them. The README describes access to many providers through one OpenAI-compatible endpoint with routing and fallbacks, plus traces and sessions for agents, cost and latency metrics, a playground, prompt versioning and a fine-tuning path through named partners. If your traffic already goes through an OpenAI-compatible SDK, the gateway is a natural place to put the logging, and if it does not, you are adding a hop to every request just to get your traces.
Arize-ai/phoenix is a platform rather than a library, and it is deliberately provider-neutral. Tracing uses OpenTelemetry-based instrumentation, and the README lists LLM evaluations, versioned datasets, experiments, a prompt playground, prompt management and a remote MCP server in the same product. The auto-instrumentation lives in a separate OpenInference project.
traceloop/openllmetry is the narrowest of the four. It is a set of OpenTelemetry instrumentations for LLM providers and vector databases, plus a convenience SDK, and the README stresses that it emits standard OpenTelemetry data that connects to existing observability stacks such as Datadog or Honeycomb. What it does not do is judge anything.
Choosing by scenario: one developer, a small team, a company with residency rules, a team embedding tracing
The split that matters is how much of the platform you actually need, and who holds the pager when it breaks.
A solo developer with one application and a personal API key should look at traceloop/openllmetry first. One instrumentation package, standard OpenTelemetry output, and the destination list in the README is long enough that most people will find a backend they already run. Nothing to operate and nothing to migrate. What you give up is precisely the part LangSmith is good at: datasets, scores and prompt versions.
A small team that wants one shared place for traces, prompts and evaluation sets should look at langfuse/langfuse or Arize-ai/phoenix. Phoenix leans toward teams that already think in OpenTelemetry spans, and its README lists supported frameworks from LangGraph to DSPy. Langfuse leans toward teams that want prompt versioning and a playground in the same place, and our analysis of Langfuse puts it squarely with people who are comfortable running a ClickHouse-backed stack.
A company with data residency or audit requirements asks a different question. None of the four removes the need for someone to keep the software patched, but self-hosting moves the trace data, the prompt text and the evaluation scores inside your own perimeter, and that is usually the requirement that actually matters.
A team embedding tracing into a product it ships should first ask whether it needs a server at all. Helicone/helicone puts the collection point in the request path, which is convenient for an OpenAI-compatible codebase, and it also brings routing, fallbacks and cost analytics that have to be sized and monitored.
What self-hosting each stack costs in operations
The install commands on this page are the quickest way to separate a library from a platform.
traceloop/openllmetry is a single pip install of the SDK and one call in your code. What it needs on the other side is a collector or an observability backend, which most teams already have. On a local run you need to disable batch sending, or traces sit in the queue while you watch an empty screen.
langfuse/langfuse asks for more. Its self-host path is a git clone and docker compose up on a local machine, a single virtual machine, or Helm on Kubernetes, which the README calls the preferred production deployment. Underneath sit the application, its database and the supporting services, and our analysis of Langfuse warns that this is a real operational commitment.
Helicone/helicone is the heaviest of the four. The README describes a stack made of a web frontend, a proxy-logging worker, a dedicated Jawn server for collecting logs, Supabase for the application database and auth, ClickHouse for analytics and Minio for object storage. Object storage for logs grows with your traffic, and ClickHouse is a database with its own memory profile that someone has to watch.
Arize-ai/phoenix sits in the middle. The README shows a pip install and a serve command, a container option, and a Docker image and Helm chart for deployment, and it says the server can run on your machine. A server component is still a server component: someone has to run it, back it up and upgrade it.
Licences: Apache-2.0 for two, a custom file for two
Two of the four are unambiguous for a company that wants to run a modified copy internally. Helicone/helicone is Apache-2.0 and traceloop/openllmetry is Apache-2.0. Both let you host a modified version for your own use without publishing the changes, and neither attaches conditions to the data you push through them.
The other two need a lawyer before a deployment, and that is a scheduling problem rather than a technical one. langfuse/langfuse carries a licence GitHub cannot classify, and our analysis of Langfuse says plainly that it is a poor fit for anyone who cannot accept a licence that is not OSI-approved. Arize-ai/phoenix is in the same position, and our analysis of Phoenix says to confirm that the licence status meets your legal requirements before you deploy.
That is a difference in kind, and it should be settled early rather than discovered during a security review. The Apache-2.0 pair can be adopted and modified behind your own perimeter with no obligation to publish your changes. The custom-licence pair may carry terms about hosting a modified version that a company has to read and accept.
One practical point closes this. Whatever you choose, the licence covers the platform. Your prompts, your evaluation data and your traces stay under your own terms, and what you are accepting is the terms of the tool that holds them.
Where the weaker choices are, stated plainly
Two of the four will disappoint a reader who expects a LangSmith replacement and gets something narrower.
traceloop/openllmetry is the clearest case. It replaces the collector, not the product. You get spans with the right attributes, which is most of what a dashboard needs, but the datasets, the scoring functions and the comparison of one run against the next are yours to build. The README's own list of supported destinations is a list of observability backends, and that is the tell: this is instrumentation, not a workflow. Our analysis of OpenLLMetry says the same thing, that what it gives you is traces rather than judgements about model output.
Helicone/helicone has the opposite problem. It does more than most people asking for a LangSmith replacement need, because a gateway with routing and fallbacks is network infrastructure that has to be sized, monitored and versioned. If you only need traces and evaluations, a platform is a better fit than a gateway.
Phoenix and Langfuse carry a different criticism: both assume you want a server. Their README descriptions point at self-hosted deployments, and both analyses say the same thing, that you have to plan for integration and maintenance overhead instead of assuming it disappears. Neither point is a reason to skip them. It is a reason to name the team that will own the box before you promise a date.
Bottom line
Pick traceloop/openllmetry when you want spans and already run an observability backend, langfuse/langfuse or Arize-ai/phoenix when you want one shared place for traces, prompts and evaluation sets, and Helicone/helicone when your traffic already passes through an OpenAI-compatible gateway. Before committing, read the LICENSE file of Langfuse and of Phoenix, run a pilot with your own workload to measure trace volume and query latency, and check whether your framework already ships OpenTelemetry support of its own before adding a second instrumentation.
Sources
Was this useful?