kafka4beam/brod: an Erlang implementation of the Kafka protocol

Brod speaks the protocol itself instead of wrapping another client

The README describes Brod as an Erlang implementation of the Apache Kafka protocol with support for both producers and consumers, and the feature list opens with support for Apache Kafka v0.8 and later. Nothing in that description suggests a thin layer over an existing client, which is the point: an application written in Erlang or Elixir can produce and consume without a JVM process, a Python sidecar or a broker-side proxy.

The library also exposes what the README calls direct APIs for message send, fetch and cluster inspection and management, without having to start clients, producers or consumers. That is a second entry point into the same protocol implementation, useful for one-off sends or tooling that does not want a long-lived connection. The project is published to Hex as `brod` and the repository carries a NOTICE file crediting Klarna Bank AB, which is where the code originally came from.

One TCP connection per brod_client, with a two-level supervision tree

The client is the piece that owns the network. The README states that a `brod_client` in Brod is a `gen_server` responsible for establishing and maintaining tcp sockets connecting to Kafka brokers, and that it also manages per-topic-partition producer and consumer processes under two-level supervision trees. Producers and consumers therefore need a client to be running first, since the client is the parent in that tree.

The connection rule is unusually strict for a client library. The feature list states that Brod opens a maximum of one tcp connection to a broker per `brod_client`, and that you can create more clients if needed. In other words, horizontal connection capacity is explicit and manual: one client means one socket per broker, and scaling to many partitions on one broker means running several clients. The alternative is provided by the API surface rather than hidden by configuration, since `brod:start_client/1,2,3` adds a new client to the Brod supervision tree on demand, and socket options can be passed through as `{extra_sock_opts, ExtraSockOpts}`.

The producer batches when in-flight requests pile up

Producer behaviour is described in terms of counters you can reason about. The producer starts to batch automatically when the number of unacknowledged, in-flight requests exceeds a configurable maximum, and the producer supports in-flight requests with asynchronous acknowledgements. The callback form, `brod:produce_cb/6`, hands the base offset back through a function of partition and base offset once the broker acknowledges.

Cluster disturbances are handled rather than surfaced as errors. The feature list says that both the consumer and the producer handle leader re-election and other cluster disturbances internally, and that the producer tries to re-send buffered messages on common errors such as "Not a leader for partition", with those errors resolved automatically by refreshing metadata. Buffered messages, not sent messages, are what survive, so a leader change costs a resend rather than a dropped batch.

The consumer side mirrors the same counters. The simple poller has a configurable prefetch count and keeps sending fetch requests as long as the total number of unprocessed messages is below that count. Two subscription styles exist beyond the poller: a group subscriber with Kafka or custom offset storage, and a topic subscriber that reads from all or selected partitions without using consumer groups.

Building Brod needs OTP 24, CMake and a Kafka it can talk to

The build section lists the requirements: a minimum Erlang/OTP version of 24, and CMake 4, which is required to build the NIF for `crc32cer`. The two commands given for the project are:

make compile
make test-env t # requires docker-compose in place

The second target stands up a test environment and then runs the unit and common test suites, and it needs docker-compose in place. The Makefile shows what the test environment involves: the `test-env` target runs a setup script, creates an SSL directory under test/data/ssl, and copies the CA, client key and client certificate out of a Kafka container, so the integration suites can exercise TLS. The `KAFKA_VERSION` variable is exported from the Makefile and defaults to 4.0, and the unit and common test targets tag their coverage output with that version.

For a first look at the protocol layer, the README suggests a shell against a broker already running on localhost:9092 with a topic named `test-topic`.

Brokers older than 0.10 need query_api_versions switched off

There is one compatibility setting that bites people on older clusters, and the README gives it its own section. For Kafka 0.9.x or earlier, the client config must contain `{query_api_versions, false}`, because `ApiVersionRequest` was introduced in Kafka 0.10 and sending that request to an older broker causes a connection failure.

In `sys.config` the setting sits alongside the endpoint list:

[{brod,
   [ { clients
     , [ { brod_client_1 %% registered name
         , [ { endpoints, [{"localhost", 9092}]}
           , { query_api_versions, false} %% <---------- here
           ]}]}]}]

The same client can also be started on demand with a named config:

ClientConfig = [{reconnect_cool_down_seconds, 10}],
ok = brod:start_client([{"localhost", 9092}], brod_client_1, ClientConfig).

The Elixir equivalent of the sys.config form is:

config :brod,
  clients: [
    # :brod_client_1 is the registered name of the client
    brod_client_1: [
      endpoints: [{"localhost", 9092}],
      reconnect_cool_down_seconds: 10
    ]
  ]

No compression codec is enabled until you add one

Compression is opt-in, and the README is explicit that Brod does not depend on any compression or decompression implementation by default. Both producers and consumers ship with no compression. To turn it on you add the compression application as a dependency in your project's rebar.config, and the example given is snappyer:

{deps, [
    {snappyer, "1.2.9"}
]}.

For the list of further compression options, the README points at the compression section of the kafka_protocol repository, which is where the codec implementations and their support matrix live. This is the practical trap in adopting Brod for an existing topic: a producer that writes uncompressed records to a topic that consumers expect to be compressed, or the reverse, will not interoperate with clients that assume a codec. The default being off is a reasonable library choice, but it puts the topic configuration on the adopter.

The working tree runs ahead of the published releases

The repository is not archived and the last push was on 2026-09-25, so the code is being worked on. The release history tells a different story. The most recent tagged releases are 3.16.2 from 2022-03-26, 3.15.0 from 2020-10-05 and 3.9.4 from 2019-11-29, and the repository currently has no newer GitHub release.

That gap is the important detail for an adopter. The Makefile already defaults `KAFKA_VERSION` to 4.0, while the newest published version on Hex is 3.16.2 from 2022-03-26, so the code in the tree is being tested against a much newer broker than the last tagged artifact. Anyone pinning `brod` from Hex gets a package whose last release predates the current state of the source, and the difference between the two is not something the README spells out. The licence is Apache-2.0, with the LICENSE file at the repository root and a NOTICE file crediting Klarna Bank AB.

The direct APIs skip the client, the producers and the subscribers

There is a real alternative approach inside Brod itself, and it is worth weighing before you build a supervision tree. The client-based path is the full one: start a `brod_client`, start a producer or a subscriber, register callbacks, and let the OTP hierarchy manage restart and shutdown. The direct path calls the protocol functions without any of that, which is what the README means by direct APIs for message send, fetch and cluster inspection and management without having to start clients, producers or consumers.

The README's quick demo shows both in one session. First the client and producer, then a subscriber and a callback producer:

rr(brod),
{ok, _} = application:ensure_all_started(brod),
KafkaBootstrapEndpoints = [{"localhost", 9092}],
Topic = <<"test-topic">>,
Partition = 0,
ok = brod:start_client(KafkaBootstrapEndpoints, client1),
ok = brod:start_producer(client1, Topic, _ProducerConfig = []),
{ok, FirstOffset} = brod:produce_sync_offset(client1, Topic, Partition, <<"key1">>, <<"value1">>),
ok = brod:produce_sync(client1, Topic, Partition, <<"key2">>, <<"value2">>),
SubscriberCallbackFun = fun(Partition, Msg, ShellPid = CallbackState) -> ShellPid ! Msg, {ok, ack, CallbackState} end,
Receive = fun() -> receive Msg -> Msg after 1000 -> timeout end end,

brod_topic_subscriber:start_link(client1, Topic, Partitions=[Partition],
                                 _ConsumerConfig=[{begin_offset, FirstOffset}],
                                 _CommittedOffsets=[], message, SubscriberCallbackFun,
                                 _CallbackState=self()),
AckCb = fun(Partition, BaseOffset) -> io:format(user, "\nProduced to partition ~p at base-offset ~p\n", [Partition, BaseOffset]) end,
ok = brod:produce_cb(client1, Topic, Partition, <<>>, [{<<"key3">>, <<"value3">>}], AckCb).
Receive().
Receive().
{ok, {_, [Msg]}} = brod:fetch(KafkaBootstrapEndpoints, Topic, Partition, FirstOffset + 2), Msg.

For a one-shot send or an operational check, the direct functions are the smaller commitment. For a service that must survive broker restarts, the supervision tree is the one that recovers. The library also ships an escriptized command line tool for message send and fetch and for cluster inspection, which is a third option for operators who do not want to write Erlang at all.

Editorial conclusion

Brod fits teams that already run Erlang or Elixir and want Kafka traffic inside an OTP supervision tree without a Java client or a sidecar service. It does not fit a team that wants a maintained tagged release to pin, since the newest tagged release is 3.16.2 from 2022-03-26 while the last push was on 2026-09-25 and the Makefile already defaults to KAFKA_VERSION 4.0. Before adopting, check that your brokers expose a protocol version Brod will negotiate, decide whether to run compression, since no codec is enabled by default, and confirm that one TCP connection per brod_client is enough for your topic count.

Frequently asked questions

Official sources

1. Issues
2. kafka4beam/brod on GitHub
3. License: Apache-2.0
4. README
5. Releases