terraform-provider-kafka: topics and ACLs as Terraform resources

A provider that talks to the broker, not to a management API

terraform-provider-kafka is a Terraform plugin for managing Apache Kafka, and the resource list is short: kafka_topic, kafka_acl and kafka_quota.

What makes it more than a topic name generator is that it talks to the cluster over the Kafka protocol itself. The dependency on the Sarama client, github.com/IBM/sarama at 1.46.3, is the giveaway. There is no admin REST layer in the way, and the provider negotiates with the brokers the same way any other Kafka client does.

That has a practical consequence for how you use it. The provider needs credentials, and it needs them at plan time as well as apply time, because the plan has to read the current state of a topic to diff it against the configuration. A pipeline that runs plan on every pull request therefore needs a Kafka identity that can read, and a pipeline that applies needs one that can write.

The plugin is available on the Terraform registry, and installation is the required_providers block followed by terraform init:

terraform {
  required_providers {
    kafka = {
      source = "Mongey/kafka"
    }
  }
}

provider "kafka" {
  bootstrap_servers = ["localhost:9092"]
  ca_cert           = file("../secrets/ca.crt")
  client_cert       = file("../secrets/terraform-cert.pem")
  client_key        = file("../secrets/terraform.pem")
  tls_enabled       = true
}

There is a second install path for people who cannot use the registry, downloading and extracting the latest release into the Terraform plugin directory, which is typically ~/.terraform.d/plugins/.

The repository is Go, and the module is github.com/Mongey/terraform-provider-kafka, built on the Terraform plugin SDK v2 at 2.38.2 with the plugin docs generator at 0.24.0. It targets Go 1.25.8 in go.mod, and the licence is MIT.

TLS is on by default, and the SASL mechanism is the switch that matters

The provider configuration is where most of the decisions live, and the defaults lean towards security.

The bootstrap_servers property is a list of host and port addresses used to discover the full set of alive brokers, and it is the only required setting. tls_enabled defaults to true, so communication with the cluster is over TLS unless you turn it off. skip_tls_verify defaults to false.

For certificate-based authentication there are ca_cert, client_cert, client_key and client_key_passphrase, each of which takes either the certificate content itself or a path to a file in PEM format. The client_cert description has a detail that matters in practice: if you have intermediate CA certificates, append them to client_cert.

The SASL settings decide how the provider authenticates once the transport is up. sasl_username and sasl_password default to empty strings, and sasl_mechanism defaults to plain, with the allowed values listed as plain, aws-iam, scram-sha256, scram-sha512 and oauthbearer.

The plain and scram mechanisms are the familiar ones. The aws-iam mode is the interesting one, and it has four documented variants, which is a sign that the author has run this against real MSK and IAM setups rather than only against a local broker.

With an assumed role:

provider "kafka" {
  bootstrap_servers = ["localhost:9098"]
  tls_enabled       = true
  sasl_mechanism    = "aws-iam"
  sasl_aws_region   = "us-east-1"
  sasl_aws_role_arn = "arn:aws:iam::account:role/role-name"
}

With an AWS profile, and the same block with sasl_aws_profile set to dev instead of a role ARN. With a profile in a non-default shared config file, where sasl_aws_shared_config_files points at the path. With static credentials taken from a Vault data source, where sasl_aws_access_key, sasl_aws_secret_key and sasl_aws_token are filled in from data.vault_aws_access_credentials.creds. And with static or container credentials from the environment, where the README asks you to export AWS_ACCESS_KEY_ID, AWS_SECRET_ACCESS_KEY and optionally AWS_SESSION_TOKEN, or AWS_CONTAINER_AUTHORIZATION_TOKEN_FILE and AWS_CONTAINER_CREDENTIALS_FULL_URI.

The SASL dependencies in go.mod line up with that list: the AWS MSK IAM SASL signer, the AWS SDK v2 credentials and STS packages, the xdg/scram library for the scram mechanisms, and golang.org/x/oauth2 for the oauthbearer one.

Redpanda works, but only if you pin the protocol version

There is a compatibility note for Redpanda that is worth understanding rather than just copying.

provider "kafka" {
  bootstrap_servers = ["localhost:9092"]
  kafka_version = "2.1.0"
}

The reason given is that Redpanda does not implement some Metadata APIs, so the Kafka version has to be forced when creating the provider. The kafka_version property takes the Kafka protocol version in $MAJOR.$MINOR.$PATCH format, and the description notes that some features may not be available on older versions.

That is a workaround, and it is a reasonable one, but it tells you something about how the provider discovers the cluster. It asks the broker for metadata, and a broker that answers metadata differently or does not implement part of the API confuses the client. Pinning the version makes the client ask for a shape of metadata the broker does understand.

The general lesson for anyone running this against a non-Apache broker is that the provider is a Kafka client first and a Terraform plugin second, so any incompatibility in the protocol layer shows up as a version issue rather than as a Terraform issue.

The test environment in the repository reflects this. The docker-compose.yaml runs apache/kafka-native:3.8.0 with SSL listeners on an internal port and an external one, the StandardAuthorizer as the authorizer class, and a KAFKA_ALLOW_EVERYONE_IF_NO_ACL_FOUND setting to true, which is the sort of thing you need in a test cluster where the ACL tests would otherwise be locked out.

Building it, and the TLS cluster the acceptance tests need

The development instructions in the README are a numbered list that doubles as a description of how the project is tested.

The first step is to install Go, then clone the repository to a path under $GOPATH:

``` bash mkdir -p $GOPATH/src/github.com/Mongey/terraform-provider-kafka; cd $GOPATH/src/github.com/Mongey/ git clone https://github.com/Mongey/terraform-provider-kafka.git cd terraform-provider-kafka ```

Then the build and test commands are make build and make test, followed by starting a TLS enabled Kafka cluster with docker-compose up, and finally running the acceptance tests with make testacc.

The fact that the acceptance tests need a real TLS enabled cluster is a design decision with consequences. A provider for topics and ACLs cannot be meaningfully tested against a mock, because the behaviour under test is precisely the broker's response to a topic configuration and to an ACL grant. That means the test suite is slower and needs Docker, but it also means the acceptance tests exercise the same code path as production.

The repository layout supports this. There is a kafka/ directory holding the provider code, a main.go entry point, a GNUmakefile for the make targets, a Dockerfile, a .goreleaser.yml for the release binaries, a docker-compose.testacc.yaml separate from the general docker-compose.yaml, a secrets/ directory holding the TLS certificates and keys, a templates/ directory, a tls-debug/ directory, an examples/ directory with an examples/main.tf, a docs/ directory for the generated documentation, a bin/ directory, and a tools/ directory.

The CircleCI badge in the README header is the sign that the whole sequence above runs on every push.

Topics, ACLs and quotas, and the release cadence

The three resources are the reason to use this at all, and the resource names map to Kafka's own vocabulary.

A topic resource manages the topic itself, which means partitions, replication factor and configuration overrides such as retention. An ACL resource manages an access control entry, which in Kafka is a principal, a host, an operation and a resource pattern. A quota resource manages client quotas, which is what the compose file's KAFKA_LOG4J_LOGGERS setting of kafka.server.ClientQuotaManager=WARN is about, since that logger is where the quota manager reports.

The pattern of a provider like this is worth naming. The topic is infrastructure that is expensive to change once data is in it, so it belongs in code. The application is what produces and consumes it, and that stays in application code. The provider sits in between and owns the middle ground, the topic configuration and the access rules, where a drift between what is documented and what is running is most likely to bite.

On releases, the tags are v0.12.1 on 2025-07-28, v0.13.0 on 2025-08-12 and v0.13.1 on 2025-09-22, and the last push to the repository was on 2026-08-27. The zero major version means the provider's own API, the resource schemas, is still allowed to break between minor versions, which is the usual Terraform provider convention and is worth remembering if you pin the version in your configuration and upgrade Terraform at the same time.

The topics on the repository are kafka, kafka-acl, kafka-topic, terraform, terraform-provider, and the licence is MIT with a LICENSE file at the root.

Editorial conclusion

terraform-provider-kafka suits a platform team that creates Kafka topics from code and needs the topic configuration, its partitions and its ACLs to live in the same state file as the rest of the infrastructure. It does not suit you if your topics are created by the applications that use them, since a topic in Terraform state and a topic in application code will fight. Before you adopt it, decide which authentication mode matches your cluster, because the SASL mechanism is a provider level setting, and remember that a provider that talks to the live broker means a plan and an apply both need credentials, so a read-only plan in CI needs a read-only identity.

Frequently asked questions

Official sources

1. Issues
2. License: MIT
3. Mongey/terraform-provider-kafka on GitHub
4. README
5. Releases