# KMinion reads Kafka through the protocol instead of scraping JMX

> Redpanda's KMinion is a Prometheus exporter that talks to the Kafka protocol directly, which is how it produces consumer group lag, log directory sizes and broker information without needing a JMX agent anywhere in the cluster.

**redpanda-data/kminion** — KMinion is a feature-rich Prometheus exporter for Apache Kafka written in Go. It is lightweight and highly configurable so that it will meet your requirements.

- Repository: https://github.com/redpanda-data/kminion
- Stars: 709 · Forks: 140
- Language: Go
- License: MIT
- Published: 2026-10-21 · Updated: 2026-10-21 · Language: en
- Canonical page: https://hysenlabs.com/projects/redpanda-data-kminion

## The premise is that the protocol exposes enough

KMinion is described as a feature-rich Prometheus exporter to monitor your Redpanda or Kafka cluster, and the statement that frames the whole project is that all valuable information that is accessible via the Kafka protocol are supposed to be accessible using KMinion. That is a deliberate constraint and it determines the design. The exporter is an ordinary client of the cluster, using the same protocol any consumer uses, which means it can be deployed as a single binary anywhere with network access, with no agent installed on the brokers and no JMX port to open. The corollary is that it can only report what the protocol exposes. Consumer group lag, log directory sizes, broker addresses, broker ids, controller status and rack ids are all protocol-level facts, and they are the metrics the feature list is built around. The primary language is Go, the licence is MIT, and the repository topics name the consumer group, the exporter and the lag as the subject.

## The feature list is nine things you would otherwise assemble yourself

The features cover a lot of ground for a component that is only a client. Kafka versions from 0.11 upwards are supported, which matters because the protocol changed underneath those versions. The SASL mechanisms include plain, scram-sha-256 and 512, and gssapi with Kerberos, so it can authenticate against a hardened cluster rather than only an open one. TLS is supported regardless of whether you need mutual TLS, a custom certificate authority, encrypted keys or just the trusted root certificates. Consumer group lag is the headline metric, the number of messages a consumer group is lagging behind the latest offset. Log directory sizes are reported as a metric either grouped by broker or by topic. Broker info gives a metric for each broker with its address, broker id, controller flag and rack id. Configurable granularity lets you export metrics per partition or per topic, which helps to reduce the number of exported metric series. Configurable targets let you pick which topics or groups to export using regular expressions. And the configuration can be written in YAML, in environment variables, or in a mix of the two.

## End-to-end monitoring measures the round trip, not the internals

One feature deserves its own section because it is the one that is not a Kafka API in disguise. End to end monitoring sends messages to its own topic and consumes them, measuring the message's real-world roundtrip latency, and it also provides ack-latency and offset-commit-latency, with more detail in the end-to-end document in the docs directory. This is the closest thing in the feature list to an actual health check rather than an inspection of recorded state. The Kafka APIs will tell you how far behind a consumer group is, which is a fact about the past. They will not tell you whether a message written now comes back in two milliseconds or two hundred. The exporter does that, by publishing and consuming on its own topic, and the two extra latencies matter because they isolate different parts of the path, since the acknowledgement latency measures how long the broker takes to confirm and the offset-commit latency measures how long the consumer takes to record its position. It is also the one feature with a real side effect on the cluster, since it creates traffic and a topic of its own.

## Configuration can come from a file, the environment, or both

All options in KMinion can be configured via YAML or environment variables, and configuring some options via YAML and some via environment variables is also possible, with environment variables taking precedence in that case. The reference config with additional documentation lives at docs/reference-config.yaml, which is the file to read before writing your own. If you want to use a YAML config file, you specify its path with the CONFIG_FILEPATH environment variable, and that is the whole mechanism, so there is no separate config subcommand and no flag to remember. The mix is worth understanding because it is how the Helm chart works in practice, with the chart supplying the base configuration and the deployment supplying the connection details, and the precedence rule means a value in the environment silently wins over the same value in the file. The repository layout shows where the pieces live, with config.go at the root, kafka/ for the protocol client, minion/ for the collection logic, prometheus/ for the metric definitions, logging/ for the log setup, e2e/ for the end-to-end tests, docs/ for the documentation, a Dockerfile, a .goreleaser.yml and a docker-compose.yml.

## Running it in Docker or through the Helm chart

Images are built on each push to master and for every new release, and the available tags are listed in the Docker Hub repository.

```shell
docker pull redpandadata/kminion:latest
```

A Helm chart is maintained as part of Redpanda's helm-charts repository, which is the usual route for a Kubernetes deployment. The Docker image is a two-stage build that ends up very small. The builder stage is golang 1.26.9 on alpine, it installs git and the certificate authorities, copies the module files and downloads the dependencies before copying the source, and builds with CGO disabled, stripping the binary with the -w and -s linker flags and stamping the version, the commit and the build time into it. The runtime stage is alpine 3 and it copies only the certificate bundle and the binary, creates a redpanda group and user, and runs as that user rather than as root.

## Testing it locally means a Kafka and ZooKeeper cluster

The repository contains a docker-compose file you can run on your machine, and it spins up a Kafka and ZooKeeper cluster with KMinion on port 8080 exposed to the host.

```shell
# 1. Clone this repo
# 2. Browse to the repo's root directory and run:
docker-compose up
```

The compose file is a conventional Kafka development stack. ZooKeeper is the confluentinc image with the client port on 2181 and a tick time of 2000. Kafka is the confluentinc image depending on ZooKeeper, with a broker id of 1, plaintext listeners on the internal port and the host port, a single replication factor for the offsets topic and for the transaction state log, and a minimum in-sync replica count of one, which are the settings you need for a single broker that will not tolerate a real cluster configuration. The KMinion service is built from the local Dockerfile rather than pulled, depends on both other services, exposes 8080, and is given the broker address as KAFKA_BROKERS in the environment.

## Dashboards are supplied, with a warning attached

The readme points at three Grafana dashboards on grafana.com, one for the cluster, one for consumer groups and one for topics, with the screenshot gallery showing the three of them side by side. The caveat is stated plainly and it is the right caveat: the dashboards might not immediately work for you due to different labelling in your Prometheus config. That is a common fate for shared dashboards, because a dashboard is a query against a specific set of metric names and labels, and the names are chosen by whoever wrote the exporter while the labels are chosen by whoever wrote the scrape configuration in Prometheus. If the exporter offers a configurable granularity, which it does, per partition or per topic, then the labels in your Prometheus configuration decide which of those two shapes the dashboard queries expect. The metrics themselves are listed in docs/metrics.md, which is the reference for the exporter's actual output, and the project communicates through Slack with the Redpanda workspace being the place for interactive discussion and support.

## Where this is the wrong choice

Three limits are worth naming. The first is the one the design implies, which is that anything only the broker process knows is out of reach, so if you need JVM heap, garbage collection or internal thread metrics you need a JMX exporter as well, and KMinion does not replace that, it complements it. The second is the granularity trade-off, since per-partition metrics on a busy cluster multiply the series count and the exporter gives you the knob to reduce it, which means the default is a choice about cost that you will want to review once the cluster is busy rather than at install time. The third is the permission model, because an exporter that reads consumer group offsets and cluster metadata needs a client account with enough rights to do both, and getting that right means granting more than a monitoring-only account usually has. The regular release cadence helps here, with 2.3.7 released on the same day as the last push and 2.3.6 and 2.3.5 in the two months before it, so the project is actively maintained rather than a stable-and-abandoned tool.

## Conclusion

KMinion suits an operator running Kafka or Redpanda who wants consumer group lag and broker metrics without a JMX agent, and it suits anyone who needs the exporter to be one binary with no sidecar on every broker. It is the wrong tool if you need the internal metrics only the broker process knows about, because the exporter deliberately works from the protocol rather than from inside the broker. It is also the wrong choice if you cannot run a privileged consumer account, since it needs list and describe permissions across the cluster. Before deploying it, read the metrics documentation and the reference config, and check the end-to-end monitoring notes, because that feature sends messages to its own topic and consumes them, which is a side effect your topic count should expect. The last push was on 2026-10-19 and the current release is v2.3.7 from the same day.

## FAQ

### How do I install and run KMinion?

Pull the image with docker pull redpandadata/kminion:latest, or install the Helm chart from Redpanda's helm-charts repository. All options are set through YAML or environment variables, and if you use a YAML file you point at it with the CONFIG_FILEPATH environment variable. Environment variables take precedence over the file when you mix them.

### What metrics does KMinion export?

Consumer group lag as the number of messages a group is behind the latest offset, log directory sizes grouped by broker or by topic, broker information including address, broker id, controller status and rack id, and end to end roundtrip latency along with ack-latency and offset-commit-latency. The full list is in the docs/metrics.md file, and topics or groups can be selected with regular expressions.

### How do I test KMinion on my own machine?

Clone the repository and run docker-compose up from its root directory. The compose file starts a ZooKeeper and a Kafka cluster and runs KMinion on port 8080, exposed to the host, so you can scrape it directly. The stack uses the confluentinc images with single broker settings for the offsets topic and the transaction state log.

## Sources

- [Issues](https://github.com/redpanda-data/kminion/issues)
- [License: MIT](https://github.com/redpanda-data/kminion/blob/master/LICENSE)
- [README](https://github.com/redpanda-data/kminion/blob/master/README.md)
- [redpanda-data/kminion on GitHub](https://github.com/redpanda-data/kminion)
- [Releases](https://github.com/redpanda-data/kminion/releases)

---

Hysen Labs editorial analysis, written from the project's own repository and release notes. Cite the canonical page: https://hysenlabs.com/projects/redpanda-data-kminion