KafkaEx is an Elixir Kafka client with the gaps clearly marked

A BEAM client that negotiates the protocol version for you

KafkaEx is an Elixir client for Apache Kafka, and the requirement line is short: Elixir 1.14 and newer, Erlang OTP 24 and newer, and supported Kafka versions 0.11.0 and newer.

The implementation detail that matters is in the Features section. KafkaEx v1.0 uses Kayrock for Kafka protocol serialisation with automatic API version negotiation, and the README is emphatic that no manual version configuration is needed. That is a real advantage for a client library, because the Kafka protocol has grown a new request and response type in most releases, and a client that hardcodes a version either breaks against a newer broker or refuses to talk to an older one. The project status list also mentions automatic API version negotiation with per-request and per-app overrides, so the automatic path can be corrected without replacing the client.

The scope list is a checked inventory. Producer, consumer groups and admin APIs are in. SASL is in, with PLAIN, SCRAM-256 and SCRAM-512, OAUTHBEARER and AWS MSK IAM. Compression is in, with gzip, snappy, lz4 and zstd. First-class telemetry with more than 27 events is in.

The project is MIT licensed, published on Hex.pm as `kafka_ex`, with the API documentation on HexDocs, and the last push was on 2026-09-23 with the most recent tags v1.1.2 and v1.0.3 from 2026-09-09.

The status page lists what is not there yet

A client library README that enumerates its own gaps is worth reading carefully, because those gaps decide whether you can use it.

Three features are marked as not implemented. The idempotent and transactional producer is planned for after 1.0. The KIP-848 new rebalance protocol is planned for after 1.0. And KIP-368 proactive SASL re-authentication on token expiry is not there, with a pointer to the token expiry behaviour section of AUTH.md for the current reconnect-driven behaviour, which is the important detail, because it means that when a SASL token expires, the client recovers by reconnecting rather than by re-authenticating in place.

The fourth item is a warning rather than a gap. Kafka 4.0 compatibility is tracked in issue 497, and the same note appears in the supported versions section, which says Kafka 4.0 and later have partial compatibility.

The practical reading is straightforward. If your system needs transactions, or it runs a Kafka cluster on the new rebalance protocol, or your cluster uses short lived SASL credentials, this client is not there yet. If it produces and consumes with SASL, TLS and consumer groups, it is.

Kafka 0.11 is the floor, 3.8.x is the tested ceiling

The supported versions section gives a minimum and a tested range, and the reason for the minimum is spelled out. Kafka 0.11.0 and newer is required for the RecordBatch format, headers and timestamps, which is to say the client only speaks the modern record format and will not talk to a broker old enough to use the legacy one.

The tested range is Kafka 2.1.0 through 3.8.x. Beyond that, Kafka 4.0 and later is partial compatibility, tracked in issue 497.

That gap between 3.8.x and 4.0 is the thing to check first. A broker at 4.0 falls into the partial bucket, which means some operations behave as expected and others do not, and the README does not say which. If your cluster is on 4.0, the sensible first step is to read that issue rather than to read the feature list.

The client also reports itself as KafkaEx v1.0, described as the Kayrock-based release, and there is a migration document in the repository root, `UPGRADING.md`, for anyone coming from an earlier line.

Quick start: mix.exs, deps.get, three API calls

Adding the dependency is a normal Mix entry:

def deps do
  [
    {:kafka_ex, "~> 1.0"}
  ]
end

and then the usual fetch:

mix deps.get

The simple producer and consumer example shows the shape of the API, and it is small enough to take in at a glance:

# Start a client
{:ok, client} = KafkaEx.API.start_client(brokers: [{"localhost", 9092}])

# Produce a message
{:ok, _metadata} = KafkaEx.API.produce_one(client, "my-topic", 0, "hello")

# Fetch messages from offset 0
{:ok, result} = KafkaEx.API.fetch(client, "my-topic", 0, 0)

Three things to notice. The client is started once and passed explicitly, the topic and partition are positional, and the fetch takes an offset. The README also shows a batch production API and a consumer that keeps an offset, and the features list covers message fetching with offset management, consumer groups with automatic partition assignment, and the metadata and offset management APIs for discovering brokers, topics and partitions and for committing, fetching and resetting offsets.

Topic management is included as well, with create and delete available programmatically.

Using the API as a behaviour in your supervision tree

For production applications the README recommends defining a module with the KafkaEx.API behaviour, and the difference from the previous example is that the client is no longer passed to every call.

defmodule MyApp.Kafka do
  use KafkaEx.API, client: MyApp.KafkaClient
end

# In your application.ex supervision tree:
children = [
  {KafkaEx.API, name: MyApp.KafkaClient, brokers: [{"localhost", 9092}]}
]

# Now call without passing client:
MyApp.Kafka.produce_one("my-topic", 0, "hello")
{:ok, messages} = MyApp.Kafka.fetch("my-topic", 0, 0)

The supervision tree is the important part. A `{KafkaEx.API, name: ..., brokers: ...}` child is started under your application's supervisor, so the connection is owned by the supervision tree and restarted with it, rather than being created inside a request handler.

That is the shape an Elixir team expects, and it is why the library is a behaviour and not just a module of functions. The complete API reference is on HexDocs, and the README's table of contents lists separate sections for consumer groups, metadata and offsets, topic management, compression, SASL, telemetry, error handling and resilience, and testing.

Configuration keys, and the aliases being retired

KafkaEx can be configured through `config.exs`, by passing options directly to `KafkaEx.API.start_client/1`, or both, and options passed to `start_client/1` override the matching `config.exs` defaults.

The basic configuration block is:

# config/config.exs
config :kafka_ex,
  # List of Kafka brokers
  brokers: [{"localhost", 9092}, {"localhost", 9093}],

  # Client identifier
  client_id: "my-app",

  # Default consumer group
  default_consumer_group: "my-consumer-group",

  # Per-attempt request timeout, in milliseconds (default 15_000).
  # Was `:sync_timeout` (now a deprecated alias, removed in 2.0).
  request_timeout: 10_000

Two deprecations are visible there. The broker option key is `:brokers`, and the legacy `:uris` is still accepted but deprecated. And `request_timeout`, which replaces `:sync_timeout`, is itself a deprecated alias that will be removed in 2.0, with a default of 15,000 milliseconds per attempt.

TLS is configured with a `use_ssl` flag and an `ssl_options` list carrying `cacertfile`, `certfile`, `keyfile` and a `verify` setting, and the consumer group settings include auto-commit behaviour with `commit_interval` at 5,000 and `commit_threshold` at 100, so offsets are committed on a timer or on a count. The offset reset policy when no committed offset exists is one of `:latest`, which is the library default and matches the Kafka and Java client default, `:earliest`, which suits new consumer groups that want to read existing data, or `:none`, which raises instead of guessing.

The test environment runs four brokers and four SASL methods

The repository ships a Docker Compose environment for testing, and it is worth reading because it shows exactly which authentication paths are exercised.

There is a Zookeeper service on `confluentinc/cp-zookeeper:7.0.4` with the client port mapped to 32181 and a healthcheck, and two Kafka brokers, `kafka-1` and `kafka-2`, on `confluentinc/cp-kafka:7.0.4`, each with a broker id and its own listener set. The first broker publishes 9092 for NOAUTH, 9192 for PLAIN, 9292 for SCRAM and 9392 for OAUTH, plus an internal listener on 29092, and the second mirrors that on 9093, 9193, 9293 and 9393 with an internal 29093. The `KAFKA_LISTENERS` and `KAFKA_ADVERTISED_LISTENERS` entries set those up, and there is a healthcheck that asks the broker for its API versions.

The SSL files are mounted into the containers from an `ssl/` directory in the repository, with the server keystore and truststore as p12 files, and there is a `docker-compose-arm.yml` alongside the main file for the ARM case.

Around that, the repository has the usual Elixir project furniture: `mix.exs` and `mix.lock`, `.formatter.exs`, `.credo.exs` for the static analysis rules, `config/`, `lib/`, `test/`, `scripts/`, `coveralls.json`, and separate CI workflows for unit tests, integration tests and static checks. There are also `AUTH.md`, `UPGRADING.md`, `usage-rules.md`, `kayrock.md`, `AGENTS.md` and `CLAUDE.md`.

The alternative approach, for teams that need the features this client does not have, is to drive the official Kafka client from Elixir over a port or a native bridge. You gain the whole protocol surface, including transactions and the new rebalance protocol, at the cost of a foreign process in your supervision tree and a serialisation boundary on every message. KafkaEx is the opposite trade, and its status page lets you make the decision with facts rather than guesses.

Editorial conclusion

Adopt KafkaEx when your Elixir system needs to produce to and consume from Kafka without leaving the BEAM, and when the missing protocol features do not apply to you. Do not adopt it if you need exactly once delivery through the idempotent producer, the KIP-848 rebalance protocol, or proactive SASL re-authentication, since all three are listed as post-1.0 work. Verify first that your brokers are in the tested range, Kafka 2.1.0 through 3.8.x, that your Elixir is 1.14 or newer with OTP 24 or newer, that the SASL mechanism your cluster uses is one of PLAIN, SCRAM-256 or 512, OAUTHBEARER or AWS MSK IAM, and that Kafka 4.0 is acceptable for you as a partial compatibility case tracked in issue 497.

Frequently asked questions

Official sources

1. Issues
2. kafkaex/kafka_ex on GitHub
3. License: MIT
4. README
5. Releases