Open-source alternatives to Braintrust: 4 LLM evaluation and prompt operations tools compared
Moving off Braintrust means the evaluation runner, the prompt registry and the dataset store all live on infrastructure you control, so the trade is a hosted workspace you do not operate for a process, a database and a comparison you have to keep yourself. Most readers should start with promptfoo/promptfoo, whose command line fits inside a pull request check.
What Braintrust does, and why people look elsewhere
Braintrust is a hosted workspace for LLM application work: it stores prompts and datasets, runs experiments against them and keeps the score of each run side by side.
- prompts, datasets and experiment results are stored on the vendor's side
- the evaluation runner runs on the vendor's infrastructure
- the scoring functions and the prompt registry are the vendor's product, so a workflow built around them does not move as files
The candidates at a glance
| Project | Licence | Maintenance | Language | GitHub stars | Install |
|---|---|---|---|---|---|
| promptfoo/promptfoo | MITPermissive: commercial use allowed | Commits in the last six monthsLast push October 21, 2026 | TypeScript | 26,245 | Command documented |
| confident-ai/deepeval | Apache-2.0Permissive: commercial use allowed | Commits in the last six monthsLast push October 2, 2026 | Python | 18,624 | See analysis |
| Arize-ai/phoenix | Custom licenceCustom licence: read the LICENSE file | Commits in the last dayLast push October 22, 2026 | Python | 11,953 | See analysis |
| langfuse/langfuse | Custom licenceCustom licence: read the LICENSE file | Commits in the last six monthsLast push October 11, 2026 | TypeScript | 35,650 | See analysis |
Each alternative in detail
promptfoo/promptfooTest your prompts, agents, and RAGs. Red teaming/pentesting/vulnerability scanning for AI. Compare performance of GPT, Claude, Gemini, DeepSeek, and more. Simple declarative configs with command line and CI/CD integration. Used by OpenAI and Anthropic.
- Best for
- You are choosing between prompt variants or between models and want a number instead of an opinion, with the evaluation running on a machine you control and the check wired into your pull requests.
- What you give up
- It has no shared workspace for a team, so results live in files and in a terminal rather than somewhere a colleague can open, and our analysis of Promptfoo says it does not suit a team that needs a vendor-neutral evaluation story. The first line of the README announces that the project is now part of OpenAI, and the same tool also ships a scanner that reviews other people's pull requests, so the surface you inherit is wider than the evaluation half you asked for.
- First install step
- Full setup guide
npm install -g promptfoo promptfoo init --example getting-started
confident-ai/deepevalThe LLM Evaluation Framework
- Best for
- You already write tests and want evaluation to live in the same repository as your prompts and retrieval code, and you are willing to learn the metric catalogue before you write the first assertion.
- What you give up
- There is no hosted comparison dashboard in the library, and our analysis of DeepEval says that a team needing one is looking at the Confident AI platform rather than the library. The metric surface is wide and the documentation site is the real product, so you must check which metric class matches your artefact, which model you pass as the judge and whether a metric can compute without retrieval context.
Arize-ai/phoenixAI Observability & Evaluation
- Best for
- You already emit OpenTelemetry and want tracing, LLM-as-a-judge evaluations, versioned datasets and experiments to live in one self-hosted product that stays neutral about which provider you use.
- What you give up
- The repository reports a licence GitHub cannot classify, and our analysis of Phoenix says to confirm that this status meets your legal requirements before you deploy. It also expects a server component that someone has to run and update, and our analysis adds that the OpenInference instrumentation has to support the exact framework versions you run.
langfuse/langfuse🪢 Open source AI engineering platform: LLM evals, observability, metrics, prompt management, playground, datasets. Integrates with OpenTelemetry, LangChain, OpenAI SDK, LiteLLM, and more. 🍊YC W23
- Best for
- You want one self-hosted platform to take over the prompt registry, the datasets, the evaluations and the traces, and you are willing to run a ClickHouse-backed stack on machines you control.
- What you give up
- The licence is a custom one GitHub cannot classify, and our analysis of Langfuse says it is a poor fit for anyone who cannot accept a licence that is not OSI-approved, so the LICENSE file has to be read before a deployment. Self-hosting means the database, the supporting services and the upgrade path are yours, and our analysis asks you to confirm that the self-hosted build carries the features you need, since some exist only in the hosted offering.
Four answers to the same question: what scores a prompt change
Braintrust is a hosted workspace for LLM application work. It stores prompts and datasets, runs experiments against them and keeps the score of each run side by side. The reasons teams look for something else are structural rather than ideological: prompts, datasets and experiment results sit on the vendor's side, the evaluation runner runs on the vendor's infrastructure, and the scoring functions and the prompt registry are the vendor's product. A workflow built around those does not move as files.
The four candidates differ in what they treat as the unit of work.
promptfoo/promptfoo is a command line tool and a library. The README describes declarative configuration files, model comparison across providers, red teaming and vulnerability scanning, an integration for CI/CD, and a code scanning action that reviews pull requests for LLM security issues. It is the only one of the four designed to run inside a pull request rather than to be looked at in a browser.
confident-ai/deepeval has the opposite shape. The README calls it similar to Pytest but specialized for unit testing LLM apps, and its metric surface is wide: G-Eval for custom criteria, a graph-based deterministic judge called DAG, and a family of agentic metrics covering task completion, tool correctness, goal accuracy, step efficiency, plan adherence, plan quality, tool use and argument correctness. The README also says the LLM-as-a-judge and other NLP models run locally on your machine.
Arize-ai/phoenix and langfuse/langfuse are platforms, and both are broader than evaluation. Phoenix adds tracing, versioned datasets, experiments, a playground and prompt management on top of OpenTelemetry-based instrumentation. Langfuse adds observability, prompt versioning, datasets, evaluations and a playground, and its README says the project has been part of ClickHouse since January 2026.
The difference that shows up when you pick one: a command line gives you a number, a library gives you an assertion, and a platform gives you a place where several people look at the same results.
Why Humanloop and PromptLayer are not on this list
Humanloop and PromptLayer are the two names readers arrive with alongside Braintrust, and neither is here. Both are hosted: they run the evaluations on their own infrastructure and keep the prompts, datasets and results in their own project. That is exactly the property the people arriving at this page are trying to remove.
This is not an argument against them. If nobody on the team wants to run a process and hold a database, a hosted workspace is the right answer and every project on this page will make things worse. Braintrust sits in the same category, and the reader who keeps using it is keeping a good product.
What is also absent is the category of a general-purpose test framework with a plugin for language models. That is a different need, and none of the four here is that.
One more omission is worth naming: a shared workspace where a colleague can open a run without installing anything. Langfuse's prompt management with version control and Phoenix's experiments come closest, and both still need a server that somebody maintains.
Choosing by scenario: one developer, a small team, a company with compliance needs, a team embedding evaluation in delivery
The useful split is how the results have to travel.
A solo developer with a handful of prompts should look at promptfoo/promptfoo. The README shows a global package install, an init step that writes a starter configuration, and then an eval run with a companion view step for the results, and it asks for a Node.js runtime of 22.22.0 or later on the npm and npx paths. Everything happens on the machine in front of you, and the README says the evals run locally so your prompts never leave it.
A team that already writes tests should look at confident-ai/deepeval. The README compares it to Pytest on purpose, and our analysis of DeepEval says it suits people who think in test cases and want evaluation to live in the same repository as their prompts and retrieval code. The cost of that shape is reading: our analysis says the documentation site is the real product, and it tells you to check which metric class matches your artefact, which model you pass as the judge, and whether a metric needs retrieval context or an expected output before it can compute at all.
A company with compliance or residency requirements has to decide how much of this runs inside its own network. Phoenix and Langfuse keep the prompts, the datasets and the scores in a store you host. What none of them removes is the need to answer, in writing, which provider sees your evaluation data, because the judge model is still a service you choose.
A team embedding evaluation into delivery should start from where the check belongs. promptfoo ships an integration for CI/CD and a code scanning action for pull requests. Phoenix ships a remote MCP server that lets coding agents query traces, datasets and experiments from an existing instance. Those are different integration points answering different questions.
What each one costs to keep running
Only one of the four has nothing to run.
promptfoo/promptfoo and confident-ai/deepeval are the two that are just processes. The install command on this page for promptfoo is a global npm install, with a Python index and a container image as alternatives, and our analysis of Promptfoo notes it is distributed that way for a reason: the eval is expected to run on a laptop or in a CI runner. deepeval takes the same shape, as a library inside your test suite.
The two platforms are a different commitment. Phoenix shows a pip install and a serve command in its README, along with a container image and a Helm chart, and it can also be run with no install at all through uvx. Our analysis of Phoenix says plainly that its breadth means you have to plan for integration and maintenance overhead, and that you should not adopt it if you cannot commit to running and updating a server component.
langfuse/langfuse asks for the most. Its self-host path is a git clone and docker compose up on a local machine, a single virtual machine, or Helm on Kubernetes, which the README calls the preferred production deployment. Underneath sit the application, its database and the supporting services.
Two practical details are worth reading before you promise a date. Phoenix's fast path is a coding agent running a setup command that detects your framework and provider and installs the instrumentation, which is quicker than hand-wiring but adds a dependency on that tool. And our analysis of Langfuse warns that a ClickHouse-backed stack is a real operational commitment, so the upgrade path is part of what you are buying.
Licences and who owns the tool
Two of the four are unambiguous. promptfoo/promptfoo is MIT and confident-ai/deepeval is Apache-2.0. Both let you run a modified copy inside your own perimeter without publishing the changes, and neither attaches conditions to the data you push through them.
The other two need a decision before deployment. langfuse/langfuse carries a licence GitHub cannot classify, and our analysis of Langfuse says it is a poor fit for anyone who cannot accept a licence that is not OSI-approved. Arize-ai/phoenix is in the same position, and our analysis of Phoenix says to confirm that the licence status meets your legal requirements before you deploy. That is a scheduling item rather than a technical one, and it is better settled in the first week than during a security review.
Ownership is the second half of the question, and it carries more weight in this category than in most, because evaluation logic ends up inside your release process. The first line of the promptfoo README announces that the project is now part of OpenAI and states that it remains open source and MIT licensed. Our analysis of Promptfoo says it does not suit a team that needs a vendor-neutral evaluation story and is uneasy about who owns the tool, and it tells you to check whether your prompts can stay local for the runs that matter. The same README also ships a scanner that reviews other people's pull requests, so the tool you inherit is bigger than the evaluation half you may need.
One practical point closes this. Whatever you pick, the licence covers the tool. Your prompts, your datasets and your scores stay under your own terms, and what you accept are the terms of the tool that holds them.
Bottom line
Take promptfoo/promptfoo when the check belongs in a pull request, confident-ai/deepeval when you want assertions sitting next to your existing tests, langfuse/langfuse when you need one shared place for prompts, datasets and scores, and Arize-ai/phoenix when you already run OpenTelemetry and want the traces in the same product. Before committing, read the LICENSE file of Langfuse and of Phoenix, confirm that the model you use as judge handles your data the way your policy allows, and check whether the metric you picked can compute from your artefacts at all.