topicctl applies Kafka topic configs the way kubectl applies manifests

The gap topicctl was built to close

The motivation section names the problem precisely. Managing Kafka topics with the standard tooling can be tedious and error-prone, there is no standard declarative way to define topics such as YAML files that can be checked into git, and understanding the state of a cluster at a given moment requires knowing and using multiple commands with different interfaces.

That last point is the one that bites hardest in practice. A question like how far behind a consumer group is, or which brokers exist and what configuration a topic carries, is spread across several tools with different output formats. topicctl was created to make topic management more transparent and user friendly, and the project says it was inspired by kubectl and other tools used in non-Kafka contexts.

That inspiration explains the whole shape of the tool. A YAML file per topic, an apply command with a diff and a confirmation step, a check command that validates without changing anything, and a separate read only interface for exploring. It is a deliberate move to make Kafka topics look like the Kubernetes resources most platform engineers already manage.

The repository is from Segment, and the readme links a Segment blog post about easier management of Kafka topics for the longer explanation.

apply reconciles the desired state against the cluster

The core subcommand is apply, and its contract is simple:

topicctl apply [path(s) to topic config(s)]

The apply subcommand ensures that the actual state of a topic in the cluster matches the desired state in its config. If the topic does not exist, the tool creates it. If it exists but its cluster state is out of sync, the tool initiates the necessary changes to bring it into compliance.

That is a reconciler, not a create script. The difference matters on the second run. A creation script fails when the topic already exists, while an applier computes the gap and does nothing when there is none, which is what makes the command safe to put in a pipeline or run on a schedule.

The configuration files are YAML, one per topic, and the quick tour shows the workflow with a wildcard over a directory:

topicctl apply --skip-confirm examples/local-cluster/topics/*yaml

The skip-confirm flag is the other half of that command. By default the tool asks before changing the cluster, and that is the right default for a command that can increase partitions or change replication on a production topic.

The scaling example in the tour shows the intended workflow. You edit the partitions value in the topic YAML file, then re-apply the file and the cluster is brought in line.

The repl is where you look at what the cluster actually holds

The second half of topicctl is a repl for interactive exploration of brokers, topics, consumer groups, messages and more. It takes a cluster config and gives you a prompt:

topicctl repl --cluster-config=examples/local-cluster/cluster.yaml

The commands inside it are the ones the motivation section complained about being scattered across separate tools:

get brokers
get topics
get partitions
get partitions topic-default
get offsets topic-default
tail topic-default

So the cluster state is one place rather than several. The get subcommand itself is a family of operations listed in the readme, covering brokers, consumer groups, members of a group, lag for each topic partition for a consumer group, config key and value pairs for a broker or topic, partitions, offsets with message counts and start and end times, and balance, which is the number of replicas per broker position for a topic or for the cluster as a whole.

Those operations are also available as ordinary subcommands, not only inside the repl. The readme describes a separate tester subcommand that sends test messages to a topic, which is what the quick tour uses to produce something to tail while the repl stays open.

bootstrap imports topics you did not create

The first step with any existing cluster is to bring the existing topics under management, and the bootstrap subcommand exists for that:

topicctl [flags] bootstrap

It creates apply topic configs from the existing topics in a cluster, which the readme says can be used to import topics not created or previously managed by topicctl. The output goes either to a directory when the output flag is set, or to standard output.

Two details matter for correctness. The placement strategy for the bootstrapped configs defaults to cross-rack unless a different one is set with the placement-strategy flag, so the imported files describe the replication layout the tool will maintain rather than an arbitrary one. And by default internal topics such as the consumer offsets topic are excluded, with the allow-internal-topics flag available if you want them included.

Getting this right matters because of how apply behaves. If a topic exists in the cluster but has no config file, apply will not know about it, and the first time someone writes a config for a similarly named topic, the reconciler may treat the existing topic as the thing it should be reconciling into compliance. Bootstrap gives you the baseline first.

check validates without touching the cluster

The check subcommand is the read only half of the workflow:

topicctl check [path(s) to topic config(s)]

It validates that each topic config has the correct fields set and is consistent with the associated cluster config. Unless the validate-only flag is set, it then checks the topic config against the state of the topic in the corresponding cluster.

The distinction between the two modes is the useful part. With validate-only you are checking the files against each other, so you can run it in a pull request or a lint step without credentials for the cluster. Without it, the command also compares against live state, which answers the question that matters in production, whether the files still match reality.

Two other subcommands cover the resources that apply does not touch. The create command creates resources in the cluster from a configuration file, and currently only ACLs are supported. It is deliberately separate from apply because it is intended for immutable resources managed by topicctl. The delete subcommand deletes a particular resource type, currently either ACLs matching the provided flags, or a single topic.

That split between reconciling and one shot operations is a sensible design, since a topic configuration can be reconciled towards, but an ACL is created once and then exists.

ZooKeeper-less access, TLS and SASL changed the config format

The revamp that produced v1 is the reason to read the upgrading notes before using this tool. The readme says the project was revamped to support ZooKeeper-less cluster access as well as additional security options, TLS and SSL and SASL, with a blog post explaining why and how.

The compatibility statement is reassuring but qualified. All changes should be backwards compatible, but you need to update your cluster configs if you want to take advantage of the new features, and the readme points at the clusters section for the latest config format. The old code is preserved in the v0 branch, so a revert is possible if the new format causes problems.

The dependency list shows what that support required. The AWS SDK for Go v2 appears with the EC2 and Secrets Manager services, which points at fetching credentials from AWS Secrets Manager and metadata for managed Kafka deployments, and there is a dedicated SASL package for AWS IAM authentication. The Kafka client used is segmentio's own kafka-go, and there is a ZooKeeper client, go-zookeeper, which remains in the tree for compatibility with the older path rather than being required for the new one.

A six node local cluster for testing the workflow

The quick tour starts by bringing up a local Kafka cluster with Docker Compose:

docker-compose up -d

The compose file in the repository describes a ZooKeeper service and six brokers spread over two racks, with broker ids 1 to 3 in zone1 and the rest in zone2, and each broker advertising a distinct address. The image tag is controlled by the KAFKA_IMAGE_TAG environment variable, which the file explains defaults to Kafka 2.7.0, and the images come from the legacy Bitnami repository. There is a separate compose file, docker-compose-auth.yml, for a secured setup, and the examples/ directory carries local-cluster, msk and auth configurations.

Because the brokers advertise addresses in the 169.254.123.123 range rather than localhost, there is an extra step. The tour asks you to run the network alias script so the broker addresses resolve locally:

./scripts/set_up_net_alias.sh

That is a detail worth understanding rather than copying. The advertised listener is what clients connect to, so in a local setup you either remap those addresses or configure the cluster file with the addresses the brokers actually advertise. The whole tour is then apply some configs, send test messages with the tester subcommand, explore in the repl, scale a topic by editing its YAML and re-applying, and finally tear the cluster down with docker-compose down.

Go 1.24, a scratch based image and an MIT licence

The implementation is Go on version 1.24.4, with the module at github.com/segmentio/topicctl. The command line is built with cobra, the interactive repl uses go-prompt with a spinner and coloured output, and the YAML parsing goes through ghodss/yaml so that JSON style YAML keys work. logrus with a prefixed formatter handles the logging, tablewriter renders the tabular output that the get commands produce, and hashicorp's go-multierror aggregates failures.

The Makefile is small. The topicctl target builds into build/topicctl with the version stamped in through linker flags, install runs go install, vet runs go vet, and the test target runs vet and then the test suite serially with a count of one and parallelism of one, which is the right shape for tests that talk to a broker. There is a separate test-v2 target that sets a broker admin environment variable before running the same suite, which is how the readme distinguishes the two Kafka protocol paths.

The Dockerfile builds with golang:1.24.4 with CGO disabled, then copies a single binary into a scratch image with an entrypoint of /bin/topicctl. A scratch base means there is no shell in the image, so entrypoint overrides have to be done differently than in a normal container, and the binary is static as a result.

The licence is MIT, with the LICENSE file at the top of the repository and a NOTICE file alongside it. The releases are recent: v2.0.2 on 2026-03-18, v2.0.1 on 2026-01-15 and v2.0.0 on 2026-01-07, and the last push was on 2026-06-25 with the repository not archived.

Editorial conclusion

Adopt topicctl if you run Kafka and want topic definitions kept in YAML in a repository, because apply reconciles those files against the cluster and check verifies them without changing anything. Skip it if your Kafka estate is small and the standard command line tools cover it, since topicctl adds a configuration format, a cluster config file and a new dependency surface to maintain. Before you rely on it, read the clusters section of the readme because v1 changed the config format for ZooKeeper-less access and TLS, confirm your brokers expose the endpoints the cluster config expects, and bootstrap your existing topics first so the tool has a baseline rather than creating duplicates.

Frequently asked questions

Official sources

1. Issues
2. License: MIT
3. README
4. Releases
5. segmentio/topicctl on GitHub