kafka-stack-docker-compose: a local Kafka cluster in a Compose file

The networking problem this repository exists to solve

Kafka clients do not connect to the address you dial. The broker returns its advertised listener, and the client reconnects there. Inside Docker that breaks quickly: a container reaches the broker as kafka1:19092, the host reaches it as 127.0.0.1:9092, and a client on the host that was handed the container hostname simply hangs. The repository's stated goal is to replicate real deployment configurations, with Zookeeper and Kafka servers distinct from each other, while removing those networking hurdles across platforms.

The audience is narrow and specific. It is for developers who need a working broker on their own machine, for people preparing a connector or Streams job against a realistic multi-broker cluster, and for anyone who wants to watch replication and leader election happen without renting three servers. The README points at Conduktor's free UI as the way to inspect the result, and the repository carries several Compose files rather than one, so the topology is a choice you make before you start.

Six Compose files, three variables and one advertised listener

The repository root holds the whole product: conduktor-kafka-single.yml, conduktor.yml, full-stack.yml, zk-single-kafka-single.yml, zk-single-kafka-multiple.yml, zk-multiple-kafka-single.yml, zk-multiple-kafka-multiple.yml and zk-multiple-kafka-multiple-schema-registry.yml. Each is a standalone topology. Nothing is generated, and nothing is templated beyond Compose's own variable substitution.

The mechanism that makes cross-platform access work is the advertised listener string, which lists three named listeners at once. The README shows the shape used for kafka1: INTERNAL on kafka1:19092, EXTERNAL on the host IP at 9092, and DOCKER on host.docker.internal:29092. A process inside the Compose network resolves the broker through INTERNAL, a process on the host uses EXTERNAL, and a container outside the network on Mac or Windows uses DOCKER. The DOCKER_HOST_IP variable feeds the external entry, defaulting to 127.0.0.1, and the README states Kafka is exposed there by default.

The full stack file is the widest one. According to the README it brings up Conduktor Platform on 8080, a single Zookeeper on 2181, a single Kafka on 9092, Schema Registry on 8081, REST Proxy on 8082, Kafka Connect on 8083, ksqlDB Server on 8088 and an experimental JMX port on 9001. The KRaft variant, conduktor-kafka-single.yml, drops Zookeeper entirely and keeps Kafka on 9092 with Conduktor on 8080, which the README describes as fitting most development requirements. The multi-broker files spread Kafka across 9092, 9093 and 9094, and the multi-Zookeeper files spread Zookeeper across 2181, 2182 and 2183. Ports are the contract here, and they are hard-coded per file rather than parameterised.

Installing it and producing your first topic

There is no package to install. The repository is consumed by cloning it and running Docker Compose against one of the files in the root, so the only prerequisites are a Docker Engine with the Compose v2 plugin and a free port range. The README gives the up and down commands for each topology in the same form.

For a single broker without Zookeeper, the KRaft file is the shortest path and the one the README calls sufficient for most development work:

docker compose -f conduktor-kafka-single.yml up
docker compose -f conduktor-kafka-single.yml down

After the containers report healthy, Kafka answers on 127.0.0.1:9092 and the Conduktor UI on 8080. To confirm the broker is reachable, the README's own port listing is what you check against; the repository does not document a topic creation command. If you need to exercise replication instead, start the three-broker file:

docker compose -f zk-single-kafka-multiple.yml up

That file exposes Kafka on 9092, 9093 and 9094 on the host. One caveat worth knowing before you start: the README notes an issue on Apple M4 machines with certain Java-based Docker images and says to uncomment the CONSOLE_JAVA_OPTS environment variable set to "-XX:UseSVE=0" in conduktor.yml.

Disk growth, pinned images and the Apple M4 workaround

The most predictable failure is disk. Kafka keeps segments until retention says otherwise, and a test cluster that receives generated load will fill a laptop drive. The README addresses this directly, and it does so with a warning attached: reducing KAFKA_LOG_SEGMENT_BYTES to 16MB and KAFKA_LOG_RETENTION_BYTES to 128MB is described as being for testing only. Those are not values to carry into anything that matters.

The second limitation is version drift. The newest release listed is v7.2.0, pairing Kafka 3.2.0 with Confluent 7.2.0, and the README's own port and listener examples reference the confluentinc/cp-kafka:7.2.1 image. Anyone who needs a current Kafka feature set will be editing image tags themselves, and the repository does not document what else changes when they do. The last push to the repository was on 2026-09-22, so the files are being touched, but the release tags have not moved since 2022.

Third, the M4 note is a real constraint rather than a footnote. If you are on that hardware, the Conduktor container needs the JVM option above before it will behave, and the fix lives in a specific file rather than in a general troubleshooting section.

Finally, the data reset story is blunt. The README says data is persisted from within the docker compose folder and that a fresh start means running the down command for that file. There is no documented volume cleanup step beyond that, and the README does not document rollback or backup.

When a plain broker container is the better choice

The alternative most teams reach for is a single broker container started directly, without a Compose file describing three Zookeeper nodes or a Connect worker. The difference is not cosmetic. This repository spends its complexity on topology: separate Zookeeper hosts, multiple advertised listeners, a fixed port map per service. If your task is producing and consuming messages from an application, that complexity buys nothing and costs you startup time, memory and a port collision surface.

Where the repository wins is precisely where a single container loses. Testing a consumer group rebalance across three brokers, watching a topic's partitions move when a broker stops, or wiring a connector into a stack that already has Schema Registry and REST Proxy running are all things the multi-file layouts make routine. The zk-multiple-kafka-multiple-schema-registry.yml file is the clearest example: it exists for people who want the registry present alongside a three-by-three cluster, which is a combination a minimal container cannot offer without manual assembly.

Who should adopt it, and what to check first

Adopt it if you are building a Kafka client, a connector or a stream processing job and you need a broker topology that behaves like a real one on a single machine. Adopt it if you want to demonstrate replication or Zookeeper fault tolerance to someone without provisioning infrastructure. The Compose files are readable, the port assignments are documented in the README, and the KRaft file covers the common case with two containers.

Do not adopt it as a production reference. The README's own framing of the larger layouts is experimental, and the disk settings it recommends are marked for testing. Do not adopt it if you need a Kafka version newer than what the pinned Confluent images provide, unless you are prepared to change image tags across several files without guidance.

Before you commit to a file, verify two things. First, confirm which listener your client will be handed, by reading the KAFKA_ADVERTISED_LISTENERS line in the file you chose and matching it against where your code runs. Second, confirm the ports are free on your host, because the files hard-code 8080, 8081, 8082, 8083, 8088, 9001, 9092, 9093, 9094, 2181, 2182 and 2183 across the set. The README shows how to move Zookeeper and Kafka ports, but it does not walk through the full set, so a collision on 8080 or 8081 is yours to resolve.

Licence and the cost of keeping up

The repository is licensed Apache-2.0, which permits commercial use, modification and redistribution provided the licence and notices are preserved. That covers the Compose files and the shell script in the root. It does not cover the container images the files pull: Confluent Platform images, Conduktor Platform and the other services carry their own terms, and the README's stack list includes Conduktor Platform, Zoonavigator and ksqlDB Server alongside the Apache-licensed Kafka and Zookeeper. Anyone planning to redistribute a modified stack should check each image's licence separately; this is a description of the repository's terms, not legal advice.

Upgrade cost is the practical concern. Because the topology is expressed as literal YAML with image tags inline, a version bump means editing every file that references the old tag, plus any listener or environment variable that changed between Confluent releases. The repository provides no migration notes and no changelog beyond the release titles. A team that pins to v7.2.0 and stays there pays nothing; a team that wants to track upstream pays in repeated manual edits across six to eight files.

Editorial conclusion

Adopt it if you need a multi-broker Kafka topology on a laptop for client development, connector experiments or replication behaviour, and if you can live with images pinned to Confluent 7.2.x. Do not adopt it as a production blueprint: the README itself frames the stack sizes as experiments, and the log segment and retention overrides it suggests are labelled for testing only. Before relying on it, check that the Compose file you picked matches the topology you actually need, and read the LISTENER_DOCKER_EXTERNAL mapping in that file, because that single value decides whether clients inside containers and clients on the host can both reach the broker.

Frequently asked questions

Official sources

1. conduktor/kafka-stack-docker-compose on GitHub
2. Issues
3. License: Apache-2.0
4. README
5. Releases