Coze Loop: self-hosted prompt debugging, evaluation and tracing for AI agents

The gap Coze Loop targets between prompt editing and production monitoring

Most agent teams end up with three disconnected tools: a scratchpad for prompt iteration, a spreadsheet or script for scoring outputs, and a logging stack for what happens after deployment. Coze Loop's pitch is that these are one workflow. The README describes it as providing "full lifecycle management capabilities from development, debugging, evaluation, to monitoring," and the feature table splits that into four rows: prompt debugging with a playground and version management, evaluation with evaluation sets, evaluators and experiments, observation with SDK trace reporting and trace data observation, and model integration supporting OpenAI, Volcengine Ark and others. The intended user is a developer who is already building an agent and wants the iteration loop self-hosted. That is a narrower audience than the tagline suggests. This is not a framework for building agents. It is the tooling around an agent you build elsewhere.

What the open-source edition actually contains relative to the commercial product

The README is unusually direct about the split. Coze Loop is described as being "based on the commercial version," with the open-source edition offering "free access to core foundational feature modules." That phrasing matters more than the feature table. The table lists capabilities, not the boundaries between editions, so a reader cannot tell from the README alone which evaluator types, which model providers, or which observability retention settings are present in the open-source build versus the commercial one. The stated reason for open-sourcing is customization and extension, and the README frames it as letting developers "customize and extend according to business needs." Treat that as the design intent: you are expected to read and modify Go source, not just operate a binary. The README's Coze Loop link points to coze.cn/loop.

How traces, prompts and evaluations fit together in the architecture

The observable mechanism is trace reporting through an SDK. The README says Coze Loop records "every stage from user input to AI output, including key stages such as prompt parsing, model invocation, and tool execution, and automatically capturing intermediate results and exceptions." The reporting side is an SDK you add to your agent; the observation side is the platform UI. Prompt parsing is named as a traced stage, which implies the platform expects to know which prompt version produced a given run, tying the prompt management module to the trace view. Evaluation then operates on those captured outputs: the README describes "automated multi-dimensional testing of prompts and Coze agents' output, such as accuracy, conciseness, compliance," organized through evaluation sets, evaluators and experiments. The dependency chain is worth stating plainly. Without SDK instrumentation you get the playground and prompt versioning but no trace data, and without trace data the evaluation module has fewer inputs to score. The README's overview does not cover the SDK's language bindings or the trace payload schema, so check those before planning the integration.

Deploying with Docker Compose and the model_config.yaml keys

The README gives a Docker Compose path and a Helm path. For Compose: clone the repository, then edit release/deployment/docker-compose/conf/model_config.yaml. Two fields are named, api_key and model, configured with a Volcengine Ark API key and the Ark endpoint ID respectively. The README notes that users in mainland China and users outside it follow different documentation links for obtaining both values, which is a useful signal that the default path assumes a ByteDance-operated model endpoint. After editing, run make compose-up from the coze-loop directory and open http://localhost:8082. The default is described as development mode. For Kubernetes, pull the chart with helm pull oci://docker.io/cozedev/coze-loop --version 1.0.0-helm, extract it, and edit release/deployment/helm-chart/umbrella/conf/model_config.yaml with the same two fields. The Helm path additionally requires Nginx Ingress, kubectl and Helm, and asks you to edit templates/ingress.yaml, setting ingressClassName and the class, instance and host values for your cluster. Note the version skew: the chart is tagged 1.0.0-helm while the latest release listed is v1.5.1. The README does not say whether the chart tracks the application version, so pin both deliberately.

Where the setup will bite you: model coupling, ports and the missing SDK reference

The model configuration is the sharpest constraint. Both deployment paths route through a single model_config.yaml with one api_key and one model field, illustrated with Volcengine Ark. The feature table claims support for OpenAI and "other models," but the README's worked example is Ark-only, and it does not show the YAML shape for multiple providers or for routing different agents to different endpoints. If your evaluation plan involves comparing outputs across two model families, verify from the source that model_config.yaml supports more than one entry before you build a workflow around it. Two smaller issues: the Compose path serves the UI on port 8082, so check which other ports the stack opens on a host with existing services. And trace data comes in through SDK trace reporting. That is the first thing to read in the repository, not the last.

How Coze Loop differs from wiring Langfuse or Phoenix into your agent

The obvious alternative for a team that wants tracing plus evaluation is Langfuse. The architectural difference is where the prompt lives. Coze Loop treats the prompt as a first-class versioned object inside the platform, with the playground, the version history and the trace view all referencing it, and evaluation sets scored against those same prompt versions. Langfuse and similar tools are primarily ingestion and analysis layers: your application owns the prompt, and the platform observes what you send it. Coze Loop's model assumes the platform is closer to the authoring surface. That is a real trade-off in both directions. You get a tighter loop between editing a prompt and seeing its evaluation score, and you take on a dependency where prompt state lives in Coze Loop's database rather than in your repository. Teams that treat prompts as code under review will find the second arrangement more natural. Teams that want a prompt playground with scoring attached will prefer Coze Loop's framing.

Licence, releases and the cost of keeping this running

Coze Loop is Apache-2.0, which permits commercial use and modification and includes an explicit patent grant. It also means there is no copyleft obligation on your modifications, so the customization the README encourages does not force you to publish changes. That is the licence position; whether your organization's policies on model-provider data handling interact with it is a question for your own review, not something the repository answers. On maintenance: the release cadence shows v1.5.1 and v1.5.0 one day apart in January 2026, with v1.4.1 roughly three months earlier in October 2025. That pattern suggests active development with occasional larger gaps, and it means upgrade notes matter. An upgrade from one minor version to the next should be treated as unverified. Budget for reading release notes and testing an upgrade on a copy of your trace data before applying it in place.

Editorial conclusion

Adopt Coze Loop if you already hold a Volcengine Ark or compatible model endpoint and want prompt versioning, evaluation sets and trace capture behind your own firewall rather than in a vendor console. Skip it if you need a hosted service with no operational surface, or if your tracing instrumentation is already written against OpenTelemetry and you are unwilling to add a second SDK. Before committing, verify three things: that the model you intend to configure is reachable through the api_key and model fields in model_config.yaml, that the open-source edition exposes the evaluator types your scoring rubric needs, and that the compose stack's ports do not collide with services already bound on the host.

Frequently asked questions

Official sources

1. coze-dev/coze-loop on GitHub
2. Issues
3. License: Apache-2.0
4. README
5. Releases