kafka_exporter: Prometheus metrics for Kafka brokers, topics and consumer groups

What kafka_exporter is for, and who actually needs it

Kafka exposes almost nothing an ordinary Prometheus scrape can read. Its own metrics live in JMX, which means a Java agent, an exporter that speaks JMX, and a JVM on every broker you want to observe. kafka_exporter takes the other route. It is a Go process that connects to Kafka as a client, using the Sarama library, and translates what it learns into Prometheus metrics. The README states support for Apache Kafka version 0.10.1.0 and later.

The audience is narrow and specific. Platform teams running Kafka on Kubernetes or on plain VMs, who already scrape Prometheus and do not want to add a JMX exporter sidecar to every broker. If your monitoring stack is not Prometheus-shaped, this project has nothing to offer you. If it is, the appeal is operational: one binary, one port, no JVM.

How kafka_exporter talks to Kafka and what it exposes

The binary is a single Go program built from kafka_exporter.go, with consumer group collection split into consumer_groups.go and SCRAM authentication in scram_client.go. It depends on github.com/IBM/sarama as the Kafka protocol client, github.com/prometheus/client_golang to serve the metrics endpoint, and github.com/krallistic/kazoo-go, which is the ZooKeeper client used for consumer group discovery. That last dependency is worth noting: consumer group offsets come in part from ZooKeeper, not only from the broker, which is why the project lists kazoo-go alongside Sarama.

The README groups the exposed metrics into three families: Brokers, Topics and Consumer Groups. The repository also ships a Grafana dashboard as kafka_exporter_overview.json with a preview image, kafka_exporter_overview.png. The metrics are served on port 9308, which the Dockerfile declares with EXPOSE 9308 and which the docker run example maps as 9308:9308.

One design consequence is that the exporter is a Kafka client, not a broker-side plugin. It polls. Every scrape interval it opens connections, asks the cluster for metadata, topic offsets and consumer group state, and returns a snapshot. That is cheap enough at normal intervals and expensive if you scrape every few seconds against a large cluster, because the request count scales with the number of topics and groups, not with the size of the response.

Installing kafka_exporter and running a first scrape

The README points to two distribution paths: prebuilt binaries on the Releases page, and the Docker Hub image danielqsj/kafka-exporter. The image is the shorter route and does not require a Go toolchain.

Pull the image and run it against one broker. The container listens on 9308 and the command takes one or more --kafka.server flags.

docker pull danielqsj/kafka-exporter:latest
docker run -ti --rm -p 9308:9308 danielqsj/kafka-exporter --kafka.server=kafka:9092

After the container starts, the metrics endpoint answers on http://localhost:9308/metrics. If you are running it next to Kafka with Compose, the README gives this file, which sets the same flag and the same port mapping.

services:
  kafka-exporter:
    image: danielqsj/kafka-exporter
    command: ["--kafka.server=kafka:9092", "[--kafka.server=another-server ...]"]
    ports:
      - 9308:9308

Run docker-compose up -d and the exporter joins the same network as the broker. To build from source instead, the Makefile provides make for the binary and make docker for the image; the Dockerfile copies the compiled binary from .build/linux-${TARGETARCH}/ into a quay.io/prometheus/busybox base and runs it as the nobody user. The default kafka.version flag value is 2.0.0, so set it explicitly if your brokers are older or newer than that.

The flags you will actually have to set: SASL, TLS and OAuthBearer

A default Kafka with no authentication is the easy case. The hard case is a managed cluster, and the flag table is where kafka_exporter earns or loses its place. SASL/PLAIN is enabled with sasl.enabled, sasl.username and sasl.password. The sasl.mechanism flag accepts sha256 or sha512 for SCRAM, plus gssapi, awsiam and oauthbearer. Kerberos needs sasl.service-name, sasl.kerberos-config-path, sasl.realm, sasl.keytab-path and sasl.kerberos-auth-type, which is either keytabAuth or userAuth. AWS MSK IAM authentication uses sasl.aws-region, falling back to the AWS_REGION environment variable.

OAuthBearer has two token sources, and the precedence is documented: sasl.oauthbearer-token-file takes precedence over sasl.oauthbearer-token-url, and the file is re-read on each SASL handshake. That matters for Kubernetes projected service account tokens, which rotate. A token URL would need the exporter to fetch and cache; the file path avoids that. TLS is a separate switch, tls.enabled, with tls.server-name used to verify the hostname on the returned certificate.

The practical constraint is that every one of these is a static flag passed at process start. There is no config file documented in the README, no hot reload, and no way to rotate a SASL password without restarting the exporter. On a Kerberos or OAuthBearer deployment where credentials rotate on a schedule, the token file path is the only mechanism the documentation offers for staying current without a restart.

Where kafka_exporter is the wrong tool

The README says it plainly: for other metrics from Kafka, look at the JMX exporter. That is not marketing modesty, it is a boundary. Anything that lives inside the broker JVM, including garbage collection behaviour, heap usage, request handler thread pools and the deep broker-side request timing metrics, is not available here. If your incident response depends on correlating GC pauses with consumer lag, kafka_exporter will not carry half of that correlation.

The second boundary is version. Support starts at Apache Kafka 0.10.1.0. A cluster older than that is out of scope. The default kafka.version flag is 2.0.0, and the flag exists because protocol behaviour differs across broker versions; pointing the exporter at a broker whose version does not match the flag is a configuration error the README does not promise to catch for you.

The third is scale-related and follows from the polling design. Consumer group collection uses ZooKeeper through kazoo-go. Clusters that have moved consumer group storage entirely off ZooKeeper, or clusters where ZooKeeper is not reachable from where the exporter runs, are a configuration the README does not walk through. And a cluster with thousands of consumer groups will make each scrape a large fan-out of metadata requests. The README does not document a scrape interval or a timeout tuning section, so the default of 30s in a Prometheus scrape config is the only lever most operators will have.

kafka_exporter compared with the JMX exporter

The alternative the project itself names is the JMX exporter. The difference is architectural, not cosmetic. The JMX exporter runs as a Java agent attached to the broker JVM or as a separate process connecting to the JMX port, and it translates MBeans into Prometheus metrics. It sees everything the broker exposes internally, at the cost of a JVM dependency and a configuration file that maps MBean patterns to metric names. That mapping file is powerful and tedious.

kafka_exporter inverts the trade. No JVM, no MBean rules, a flag list instead of a YAML mapping, and a fixed metric set covering brokers, topics and consumer groups. You get less, and you get it with less to maintain. For a team whose questions are consumer lag, topic throughput and broker reachability, the fixed set is enough. For a team debugging broker-side latency, it is not, and no amount of exporter configuration will close that gap. The README's own pointer to the JMX exporter is the honest version of this comparison.

Maintenance, release cadence and the Apache-2.0 licence

The last push to the repository was on 2026-09-24, and the most recent release is v1.10.0 from 2026-09-08. Before that, v1.9.0 landed on 2025-02-17 and v1.8.0 on 2024-08-20. The gap between v1.9.0 and v1.10.0 is roughly seven months, and the gap before it roughly six. That is a slow but real cadence, and the repository is not archived.

The upgrade cost is mostly in the flag surface. A release that adds SASL mechanisms or token sources, as v1.10.0 appears to have done given the oauthbearer-token-file flag with its documented re-read behaviour, changes what you can configure rather than what you must. The Go module requires go 1.27.0 with toolchain go1.27.1, so building from source pins you to a recent Go release; the Makefile's crossbuild target passes --go=1.27 to promu. If you consume the Docker Hub image instead, that constraint disappears.

Licensing is Apache-2.0, which the repository states in LICENSE and the README badge repeats. Apache-2.0 includes an explicit patent grant and requires that you preserve notices and state changes. It is permissive and compatible with closed-source internal deployment. This is a description of the licence text, not legal advice; your own review decides whether your distribution model fits.

Editorial conclusion

Adopt kafka_exporter when you want Kafka broker, topic and consumer group metrics in Prometheus without running a JVM agent, and when your brokers are Apache Kafka 0.10.1.0 or later with SASL or TLS that the flags cover. Do not adopt it if you need JVM internals, per-partition lag, or a broker version the compatibility line does not include. Before rollout, verify the scrape interval against the documented default of 30s, confirm the kafka.version flag matches your brokers, and check the chart values in charts/ for the version you intend to run.

Frequently asked questions

Official sources

1. danielqsj/kafka_exporter on GitHub
2. Issues
3. License: Apache-2.0
4. README
5. Releases