Opik: self-hosted LLM tracing and evaluation from Comet

The gap Opik fills between logging and evaluation

Most teams building on LLMs end up with two disconnected habits. They log requests somewhere for debugging, and they judge output quality by eyeballing a handful of examples. Opik targets that gap directly. It is an open source platform from Comet for tracing LLM calls and agent activity, running evaluations against datasets, and monitoring the result in dashboards, all under Apache-2.0.

The intended audience is narrow enough to be useful: engineers building LLM applications, RAG systems and multi-step agents in Python or JavaScript. The README lists the scope as AI agent tracing, LLM evaluation, prompt management and production monitoring, with a PyTest integration for running evaluations in CI. That combination matters because it means the same dataset and metric definitions can be reused from a local experiment through to a CI check, rather than being rewritten per stage.

Where the pitch gets thin is in operational detail. The README states the full platform is free to self-host, but it does not spell out resource requirements or expected throughput. Treat that as something to measure in your own environment rather than something the project promises.

How tracing, datasets and judge metrics connect

The architecture visible from the repository splits into a client SDK and a server. The SDK is what your application imports; it sends traces and spans to an Opik backend, which can be the hosted Comet service or a self-hosted deployment. The repository root carries separate top-level directories for apps, deployment, extensions and sdks, which reflects that split: the server, the deployment tooling and the client libraries ship from the same repository but are consumed differently.

On the evaluation side, the flow is dataset first. You define a dataset, run your application over it as an experiment, and attach metrics. The README points to LLM-as-a-judge metrics for hallucination detection, moderation and RAG assessment, including Answer Relevance and Context Precision. Those metrics are model-graded, which means each evaluation run consumes calls to a judge model of your choosing. That is a real cost consideration the README does not quantify.

Tracing and evaluation share the same storage, so a trace captured in development can be annotated with feedback scores through the Python SDK or the UI, then reused as evaluation material. That is the design decision worth noting: Opik is not a pure tracing tool with evaluation bolted on, and it is not an evaluation harness with tracing bolted on.

Installing the Opik SDK and running a first evaluation

The Python SDK installs from PyPI. The README shows the package name as opik, and the badge at the top of the repository links to the PyPI project of the same name.

pip install opik

After installing, the SDK needs to know where to send data. The README documents a configuration step that sets the API key and workspace, and the repository ships an .env.template at the top level as the reference for the variables involved. The exact keys are not reproduced in the README excerpt, so read that template before writing your own configuration file rather than guessing variable names.

Once configured, the typical first use is wrapping a function call so it produces a trace. The README describes tracking LLM calls and traces with detailed context, and points to a Quickstart page for the concrete call. The integration list is where most teams start instead of hand-instrumenting: the README names native integrations with LangChain and LlamaIndex among the topics, and mentions recent additions including Google ADK, Autogen and Flowise AI.

For evaluation, the documented path is to create a dataset, run an experiment over it, and attach metrics. The README links separate documentation pages for managing datasets and for evaluating your LLM application. Expect the first real experiment to require a judge model, since the metrics listed are LLM-as-a-judge rather than string-matching heuristics.

Self-hosting the Opik server is the real commitment

The client SDK is a small dependency. The server is not. The repository contains a deployment directory and the README carries a dedicated server installation section, which tells you the project expects self-hosting to be a deliberate operation rather than a one-liner. The README does not publish minimum CPU, memory or storage figures in the excerpt available, and it does not document a rollback procedure for a failed upgrade. That silence is the main thing to resolve before committing a production workload.

There is a second limitation that follows from the design. Because traces, datasets and experiments live in the same backend, the server becomes a piece of infrastructure your evaluation pipeline depends on. If the self-hosted instance is down, CI evaluation jobs that rely on it will fail. Teams used to evaluation harnesses that run entirely in-process will notice the difference.

The language coverage is also narrower than the feature list suggests. The primary language is Python, and the repository has an sdks directory implying more than one client, but the README's examples and integration discussion centre on Python. If your stack is Go, Java or Rust, verify client support in the SDK documentation before assuming parity.

Opik compared with an in-process evaluation harness

The closest alternative in kind is a library-only evaluation framework that runs in your test process and writes results to stdout or a file. The difference is architectural, not cosmetic. An in-process harness has no server, no dashboards and no shared trace store; you get metrics and a report, and you own the storage question yourself.

Opik moves that storage and the UI into a service. In exchange for running it, you get trace trees for multi-step agents and tool calls, annotation of spans with feedback scores, and a prompt playground for iterating on prompts and models. You also get online evaluation rules, which is a capability an in-process harness cannot offer at all, because there is no long-running component to apply rules to production traffic.

The trade-off is honest: if all you need is a pass or fail on a fixed test set during CI, a library-only harness is less machinery. Opik earns its server when you also want to inspect why a specific production trace went wrong, and when the same dataset should drive both the local experiment and the CI check.

Licence, releases and what an upgrade actually costs

Opik is licensed under Apache-2.0, and the repository includes a LICENSE file and a CLA.md for contributors. Apache-2.0 permits commercial use and modification, but this is a description of the licence identifier, not legal advice; if you redistribute the platform or embed it in a product, read the licence text and the CLA yourself.

Release cadence is high. The three most recent releases listed are 2.2.55 on 2026-09-08, 2.2.54 earlier the same day, and 2.2.53 on 2026-09-07. Patch-level releases landing within hours of each other suggest active iteration, and the last push to the repository was on 2026-09-09. The practical consequence is that pinning versions matters more than usual: if you self-host the server and the SDK separately, an unpinned upgrade on one side can outpace the other.

The repository also carries a Makefile whose help target documents developer tooling rather than deployment, including hooks for the pre-commit framework and targets that sync AI editor configuration from a .agents directory into .cursor, .claude and Codex layouts. That is contributor ergonomics. It tells you the project invests in its own development workflow, but it is not part of the runtime you deploy.

Editorial conclusion

Opik fits teams already instrumenting Python LLM apps who want traces, datasets and judge metrics in one Apache-2.0 stack, and who accept running the server themselves for full control. It is the wrong pick if you want a hosted-only product with no Docker in your stack, or if you need a language other than Python and JavaScript today. Before adopting, verify that your Python version and framework integration appear in the SDK documentation, and confirm the licence terms for the components you deploy.

Frequently asked questions

Official sources

1. comet-ml/opik on GitHub
2. License: Apache-2.0
3. Project website
4. README
5. Releases