confluent-kafka-python: the Kafka client that wraps librdkafka in a C extension

Why the client is a C extension and not pure Python

The design decision that explains everything about this package is in setup.py, and it is worth reading before anything else.

The package builds a single extension module called confluent_kafka.cimpl from a set of C sources.

module = Extension(
    'confluent_kafka.cimpl',
    libraries=[librdkafka_libname],
    sources=[
        os.path.join(ext_dir, 'confluent_kafka.c'),
        os.path.join(ext_dir, 'Producer.c'),
        os.path.join(ext_dir, 'Consumer.c'),
        os.path.join(ext_dir, 'ShareConsumer.c'),
        os.path.join(ext_dir, 'Metadata.c'),
        os.path.join(ext_dir, 'AdminTypes.c'),
        os.path.join(ext_dir, 'Admin.c'),
    ],
)

The library name differs by platform. On Unix the extension links as -lrdkafka, while on Windows it needs the full librdkafka name, and the setup script picks between them by checking platform.system().

So this is a binding over librdkafka, the C client for Kafka, rather than a reimplementation of the protocol in Python. The readme is explicit about the consequence: the Apache Kafka Python client, kafka-python, is a pure Python implementation which works but has significant performance limitations for high throughput production use, while this client uses the same high performance C library used by Confluent's other clients.

The performance note in the readme is the argument the whole project rests on, and it is worth naming plainly rather than repeating the marketing around it. The claim is comparative and structural: a C library doing the network and protocol work will beat a pure Python implementation doing the same, and this client gets that C library for free because Confluent maintains it.

The cost is that installing this package is not the same as installing a pure Python wheel. It needs a compiler, and it needs librdkafka available. The repository root reflects that: an INSTALL.md, a DEVELOPER.md, a requirements/ directory, a Makefile and a service.yml for continuous integration.

The API surface is Producer, Consumer and AdminClient

The readme describes the client as providing a high-level Producer, Consumer and AdminClient, compatible with all Apache Kafka brokers at or above version 0.8, with Confluent Cloud and with Confluent Platform.

That broker floor of 0.8 is old enough to be worth noting. The client is not restricted to a recent broker, which means an existing cluster on an old version is not a blocker, and also that the newest protocol features are only available against brokers that implement them.

The feature list adds up to a fairly conventional Kafka client surface with some additions. There is full support for the Kafka protocol, transactions and the administration APIs. There is direct Schema Registry integration with synchronous and asynchronous clients, handling Avro, Protobuf and JSON Schema with automatic schema evolution handling. Error handling is described as detailed and context-aware, which for a client means the exception carries the Kafka error code and a message rather than a bare failure.

Two additions are Cloud-specific. Producers can automatically connect to brokers in the same availability zone, which the readme says reduces latency and data transfer costs without manual configuration. And there are pre-defined configuration profiles optimised for common cases such as high throughput or low latency, which is a convenience for people who do not want to hand tune librdkafka settings.

The one genuinely new protocol feature is Queues for Kafka, exposed as a ShareConsumer following KIP-932, which gives queue-like, cooperative consumption with per-record acknowledgement. The readme is clear about its status: Preview, and not recommended for production, with a guide in docs/kip-932-share-consumer.md.

That last detail is the clearest example of the project treating a Kafka proposal as a first class citizen, and it is also a reminder that client features often arrive before the protocol is settled.

The asyncio producer, and the one thing it cannot do

The AsyncIO Producer is the piece of the API that most directly changes how you write an application, so the readme's example is worth reading closely.

import asyncio
from confluent_kafka.aio import AIOProducer

async def main():
    p = AIOProducer({"bootstrap.servers": "mybroker"})
    try:
        # produce() returns a Future; first await the coroutine to get the Future,
        # then await the Future to get the delivered Message.
        delivery_future = await p.produce("mytopic", value=b"hello")
        delivered_msg = await delivery_future
        # Optionally flush any remaining buffered messages before shutdown
        await p.flush()
    finally:
        await p.close()

asyncio.run(main())

Two awaits are needed and the comment explains why. The first await on the produce coroutine yields the Future, and the second await on that Future yields the delivered Message. It is a slightly unusual shape, and it comes from the fact that the produce call both enqueues and returns something awaitable.

The notes then give the important limitation. Batched async produce buffers messages, and delivery callbacks, stats, errors and the logger all run on the event loop. Per-message headers are not supported in the batched async path, and if headers are required you should use the synchronous Producer.produce and offload to a thread in an async application.

That header restriction is the kind of detail that decides whether the async path is usable for you. Kafka headers are how you carry metadata alongside a record, and if your producer sets headers on every message, the async path is off the table and you are back to threads.

The readme also points at a more complete example, examples/asyncio_example.py, which includes both an async producer and a consumer, and at a separate architecture document, aio_producer_simple_diagram.md, for the implementation details. The existence of that diagram file suggests the async implementation is not a thin wrapper over the synchronous one, which is consistent with the batching and callback behaviour described.

The examples directory is the actual documentation

For a client library, the examples are the documentation that matters, and this repository has a substantial set.

The top level examples directory covers the ground a user actually needs. There is adminapi.py and adminapi_logger.py for the administrative API, consumer.py and multithreaded_consumer.py and multithreaded_producer.py for the concurrency patterns, context_manager_example.py, get_watermark_offsets.py for reading partition offsets, eos_transactions.py for exactly once processing through the transactional API, confluent_cloud.py for the Cloud configuration, and docker/ for running the examples against a broker in a container.

Then there is a whole set per serialization format. The avro/ subdirectory plus avro_consumer.py, avro_producer.py, avro_association.py, avro_consumer_encryption.py and avro_producer_encryption.py cover Avro with Schema Registry, and the parallel json_consumer.py, json_producer.py, json_consumer_encryption.py and json_producer_encryption.py do the same for JSON Schema.

The encryption variants are the interesting ones. They show Schema Registry encryption in use, which is a feature you would otherwise have to read the API reference to work out how to combine.

The readme also points to the confluentinc/examples repository for the Confluent Cloud versions of these demonstrations, covering exactly once data processing through the transactional API, asyncio integration, deserializing and serializing Protobuf, JSON and Avro with Schema Registry, and Cloud configuration. And it says the tests directory is useful as a reference for example usage, which is a real hint: a test suite shows the call sequences that actually work, including the error paths that the documentation does not mention.

There is also a fips/ directory, which suggests a FIPS mode build, and an examples/Makefile and README for running them.

Where confluent-kafka-python is the wrong tool

There are four cases where this is the wrong choice, and they are all cases where the trade the readme makes does not pay off.

The first is a deployment that cannot build a C extension. The package ships a compiled module, so you need a compiler and librdkafka, and in a locked down or multi architecture environment that is a packaging project of its own. The pure Python kafka-python client exists precisely for that case, and the readme is honest that it is slower rather than pretending otherwise. If your throughput needs are modest, the slower client is the cheaper one.

The second is a deployment where Confluent Cloud is not an option and enterprise support is worthless to you. The readme says the client is optimized for and fully supported with Confluent Cloud and Confluent Platform, and the feature list leads with professional support backed by enterprise SLAs and 24/7 options. Those are the vendor's terms for a commercial product, and if you are running open source Kafka on your own hardware, the support argument does not apply to you. The library still works, and it still uses librdkafka, but you are carrying the maintenance of the C layer yourself.

The third is a team that wants to avoid the vendor. The package is Apache-2.0 by the classifier in the manifest, and the readme points at the LICENSE file, though the licence field on the repository is listed as a custom one that GitHub cannot classify, so read the file rather than relying on the badge. Either way, the repository is hosted under the confluentinc organisation, the Schema Registry integration is Confluent's own service, and the Cloud features are Confluent's product. A team whose policy is to avoid that dependency should know what they are adopting.

The fourth is a workload that needs the share consumer. It is marked Preview and not recommended for production, and queue semantics per record acknowledgement are the kind of feature you do not want to bet a pipeline on while it is settling.

On the alternative, the difference in approach against kafka-python is not a better API. It is where the work happens. A pure Python client does the protocol work in Python, which means you can debug it, patch it and read it, at the cost of throughput. This client hands the protocol work to librdkafka, which is where the throughput and the low latency come from, and the price is a compiled dependency and the C layer's own release cycle.

For a producer doing a few thousand records a second, the difference may not matter and the pure Python client is the lower friction choice. For a pipeline moving data continuously, the C library is the reason you would pick this one.

On maintenance, the release cadence is steady: v2.16.0 on 2026-10-07, v2.15.1 on 2026-09-10 and v2.15.0 on 2026-06-30, with the last push on 2026-10-18. The package is classified as Development Status :: 5, Production/Stable, requires Python 3.8 or newer, and the repository has a CHANGELOG.md, a DEVELOPER.md, a tox.ini and a service.yml for continuous integration.

Editorial conclusion

confluent-kafka-python suits a Python team that needs high throughput, Schema Registry and transactional semantics and can accept a compiled extension in its build, and it does not suit a project that needs a pure Python install with no compiler or no librdkafka in the pipeline. Before you adopt it, check that your deployment can build a C extension, since the package ships confluent_kafka.cimpl linked against librdkafka, and read the ShareConsumer documentation carefully, because the readme marks it as Preview and not recommended for production.

Frequently asked questions

Official sources

1. confluentinc/confluent-kafka-python on GitHub
2. Issues
3. Project website
4. README
5. Releases