rust-rdkafka: a Rust wrapper around librdkafka with async consumers, EOS and wide Kafka compatibility

librdkafka wrapped in Rust safety, not a native Rust Kafka implementation

rust-rdkafka connects Rust services to Kafka by wrapping librdkafka, the C library that Confluent maintains. The Rust layer provides type-safe abstractions and bridges the C library to async runtimes, while librdkafka handles protocol framing, broker connections, compression, partition assignment and offset management.

The alternative approach is a native Rust Kafka implementation with no C dependency. rust-rdkafka is not that, and the trade-off is explicit: the project accepts the complexity of a C build dependency in exchange for librdkafka's tested internals and wide protocol coverage. A wide range of Kafka brokers is supported going back to early protocol iterations, with compatibility details documented in the librdkafka project.

Beyond consuming and producing messages, the feature set covers administration and metadata operations: creating and deleting topics, adding partitions, altering broker and topic configurations, and reading cluster metadata (topic-partitions, replicas, active brokers) along with group metadata (groups, members, hostnames). Access to producer and consumer metrics, errors and callbacks is also part of the API.

Published under the MIT licence, master was last pushed on 2026-07-15 and the repository has no GitHub releases; change history is in changelog.md at the root. The crate works with Tokio, async-std or smol, or with no external async runtime via the `naive-runtime` feature.

Adding rdkafka to Cargo.toml and the outdated install example in the README

The crate is published on crates.io as `rdkafka`. The README's install section gives this Cargo.toml entry:

[dependencies]
rdkafka = { version = "0.25", features = ["cmake-build"] }

The version shown is 0.25, while Cargo.toml in the repository records the current crate at 0.39.0; copying the example directly installs code that is 14 minor steps behind the current state. The crates.io badge at the top of the README links to the actual published crate, and that badge is the right reference for a new Cargo.toml entry.

Adding the `cmake-build` feature instructs the build to compile librdkafka using CMake and link it statically into the final binary. This is the recommended path when CMake is available. Without the feature, librdkafka still compiles via a different toolchain path. The `dynamic-linking` feature links against a system-installed librdkafka instead of building from source.

Default features are `libz` and `tokio`. Optional features include `ssl`, `gssapi` and `zstd-pkg-config`, each requiring a corresponding system library. Cargo.toml lists both `cmake-build` and `cmake_build` as features; the hyphenated form is the canonical one and the other is an alias. The `rdkafka-sys` crate, a workspace member listed under rdkafka-sys/ in Cargo.toml, provides the raw C bindings.

BaseConsumer requires manual poll calls; StreamConsumer delegates polling to the runtime

Three consumer types cover different polling requirements and integration styles.

BaseConsumer is the lowest-level option, a thin wrapper around librdkafka's consumer. Nothing happens automatically. The application must call `poll()` periodically to process rebalances, fire callbacks and receive messages, and a long gap between poll calls delays all consumer-side events by the same amount.

StreamConsumer implements the Stream trait and handles polling internally. Messages arrive as items in an async stream, and the consumer integrates with Tokio tasks without requiring explicit poll management. The examples folder contains asynchronous_processing.rs for Tokio, runtime_async_std.rs for async-std and runtime_smol.rs for smol, all using the same StreamConsumer interface with different async runtimes. This shows that the consumer is not tied to a single executor.

Automatic consumer rebalancing is built in, and the library provides customizable callbacks for both pre- and post-rebalance events. Offset commits are also customizable. Committing an offset before processing is complete risks message loss on failure, because upon recovery the consumer resumes from after the committed position, skipping the message where the failure occurred.

The at_least_once.rs example in the examples folder demonstrates the commit ordering that prevents this. For projects that want rust-rdkafka without any async library, the `naive-runtime` feature provides a minimal built-in executor.

BaseProducer, ThreadedProducer and FutureProducer offer three delivery confirmation models

Three producer types address different requirements for handling delivery confirmation.

BaseProducer is the low-level wrapper and, like BaseConsumer, requires periodic `poll()` calls to fire delivery callbacks. Without polling, confirmation that a message was delivered or rejected never reaches the application, though production queuing continues unaffected until the next poll is called.

ThreadedProducer wraps BaseProducer with a background thread that calls poll automatically. Messages are sent in the normal way, and the thread handles delivery callbacks without requiring explicit poll calls from the application. This suits blocking or callback-driven production where async overhead is not appropriate.

FutureProducer returns a Rust Future for each message. When the broker acknowledges or rejects the message, the future resolves, and applications can await delivery confirmation before proceeding. This model fits the Tokio task model directly. The simple_producer.rs example in the examples folder demonstrates basic usage.

Synchronous and asynchronous production are both available across the three types. BaseProducer and ThreadedProducer use callbacks for confirmation, and FutureProducer uses futures. For exactly-once semantics, a transactional producer sits above these three types and groups produced records together with consumer offset commits into atomic transactions, committing or aborting both together.

Exactly-once semantics require transactional producers and isolation.level read_committed

Exactly-once semantics (EOS) in rust-rdkafka work through transactional producers. A transaction groups produced records and consumer offset commits into a single atomic operation: all records commit and all offsets advance, or neither does. No partial state exists between those two outcomes.

On the consumer side, `isolation.level` must be set to `read_committed`. A consumer with that setting skips messages from uncommitted or aborted transactions. Without it, the consumer sees all messages regardless of transaction state, including messages from transactions that were later rolled back.

The documented use case for EOS is read-process-write pipelines where each message must be processed exactly once. The README points to the Transactions section of the producer documentation for the complete API. No throughput overhead figures for transactional operations appear in the README or Cargo.toml.

At-least-once delivery is a simpler and less restrictive guarantee: every message is processed at least once and no message is lost. It requires only that offset commits happen after processing is complete. If a failure occurs between processing and the commit, the message is reprocessed upon recovery. The at_least_once.rs example in the examples folder shows the commit ordering that prevents the loss scenario.

Compiling librdkafka from source requires the GNU toolchain and several optional system libraries

librdkafka compiles from source during cargo build by default and links statically into the Rust binary. Building it requires the GNU toolchain, GNU make and pthreads as non-optional dependencies. zlib is optional but included by default as the `libz` feature.

Four optional cargo features each need a corresponding system library on the build machine. The ssl feature needs the ssl development headers (on Ubuntu packaged as `libssl-dev`). Adding SASL and Kerberos authentication via gssapi needs `libsasl2-dev`. Zstandard compression via zstd-pkg-config needs `libzstd-dev`. The cmake-build feature needs cmake, and the README notes that using CMake for the build is the encouraged approach when cmake is available.

The README includes benchmark figures collected on a Dell laptop with an Intel Core i7-4712HQ at 2.30GHz, using a local Kafka broker with 3 partitions and default configuration. Producing 5 million 10-byte messages averaged 1,045,413 messages/s and 9.970 MB/s over 5 runs. Producing 100,000 10KB messages averaged 24,623 messages/s and 234.826 MB/s over 5 runs. These are from a specific hardware setup and the README links to the kafka-benchmark project for broader measurements.

The static link means deployment machines do not need librdkafka installed separately. The `dynamic-linking` feature serves environments where a shared library is preferred.

The README warns APIs may change and gives no explanation for the outdated install example

Two points in the repository are worth reading before integrating this crate into a production service.

The Features section of the README carries a warning: the library is under active development and the APIs may change. That statement has no qualifier and no date. Whether it reflects the interface at the current state of the codebase or dates from an earlier development period is not addressed anywhere in the repository. Engineers who need long-term stability should read the project's change log before updating across minor updates.

The install example in the README is outdated, with no note explaining why, and the crates.io badge at the top of the file is the right source for the actual current crate identifier when writing a new Cargo.toml entry. Copying the example without checking that badge installs outdated code.

The project lists several downstream users in the README: kafka-view, kafka-benchmark, callysto, bytewax and kafka-mock-gen. These are presented as examples of projects using rust-rdkafka. No claims about their production status or configuration appear in the README.

Editorial conclusion

rust-rdkafka is the right choice for Rust services that need a full Kafka client and can tolerate a C build dependency. The static librdkafka link removes runtime library surprises, but every developer machine and CI environment must supply the GNU toolchain, CMake and pthreads before cargo build succeeds. Skip it when the project cannot depend on a C toolchain, or when API stability is a hard requirement given the README's own warning that the APIs may change. Before adding the dependency, check crates.io rather than copying the README's install example directly, since the example does not reflect the actual published crate.

Frequently asked questions

Official sources

1. fede1024/rust-rdkafka on GitHub
2. Issues
3. License: MIT
4. README