kafka-spark-consumer: a receiver based Kafka connector for Spark Streaming

The receiver model returns to avoid the write-ahead log

Kafka-spark-consumer began as a fork. The README states that the code is taken from the Kafka spout of the Apache Storm project, which was itself created by wurstmeister, and that the original spout was modified to work with Spark Streaming. The goal was better handling of Kafka offsets and of failures. The library implements a custom reliable receiver that uses the Kafka Consumer API to fetch messages and stores every received block in the Spark BlockManager. The distinguishing claim is that a Spark Streaming job using this consumer does not require a write-ahead log for recovery from driver or executor failures, because the processed offset is stored after every batch interval and the consumer can restart from the last processed offset. The README is careful to frame this as the difference from the Spark out of box Kafka consumers.

Zookeeper holds the offsets, and they are committed per batch

Offset storage is the second load-bearing decision. The consumer keeps the offset for each Kafka partition in Zookeeper, which the README presents as the mechanism that lets the job recover in case of a failure, and the consumer can commit the offsets of a processed batch once the Spark Streaming batch is completed. Zookeeper is not only storage here, it is also the feedback channel for the rate controller, as described below. The design has an obvious consequence. The guarantee depends on the interaction between three components that are not in the same process: the batch completing in Spark, the offset write reaching Zookeeper, and the receiver restarting and reading that offset back. A failure between the commit and the next batch therefore replays or skips at the batch boundary rather than at the record boundary, which is a coarser guarantee than per-record exactly-once semantics. The README does not describe the window between the Zookeeper write and the batch completion, so anyone depending on precise duplicate behaviour should read the source.

Block settings decouple RDD partitions from Kafka partitions

The most practically useful idea in the library is that the number of partitions in the generated RDD does not have to equal the number of Kafka partitions. The consumer creates blocks from the Kafka stream and writes them to the Spark BlockManager, and because each block written within a batch interval creates one partition in the underlying RDD, you control the RDD parallelism through the block creation interval and the block size. The README works through an example: a topic with 10 partitions, a block interval of 200 ms and a batch interval of 5 seconds produces 5 blocks per second per partition, so 5 x 10 x 5 equals 250 blocks per batch, and therefore an RDD with 250 partitions. With the default one to one mapping, the same job would have 10 partitions and the parallelism you wanted to add would be missing. If the same Kafka partition is read by several receivers, the consumer detects the partition count and spawns receivers accordingly: with 100 partitions and 20 configured receivers, each receiver takes 5 partitions.

The PID controller throttles polls through Zookeeper feedback

Back pressure is handled with a control loop rather than a fixed rate. The library implements a PID, proportional integral derivative, based rate controller, and the feedback loop is built on Zookeeper. The mechanism is direct: the controller alters the number of messages consumed during each poll, so when the downstream Spark processing falls behind, the consumer reads less from Kafka instead of filling the executor memory. The README points at two separate sections for this, one on how to enable back pressure through the Spark consumer properties and one on how to tune the controller. It also mentions a message interceptor capability that preprocesses Kafka messages before they are written to the Spark BlockManager, which is the natural place to drop, reshape or filter records, and a consumer lag checker in the style of ConsumerOffsetChecker for finding lag. These are the pieces that make the library usable as a long running job rather than a one shot fetch.

Building the jar and adding the dependency

The README gives a manual build, three commands from a clean checkout:

    git clone https://github.com/dibbhatt/kafka-spark-consumer

    cd kafka-spark-consumer

    mvn install

The artifact it produces is then declared in your own Maven build:

        <dependency>
                <groupId>dibbhatt</groupId>
                <artifactId>kafka-spark-consumer</artifactId>
                <version>2.1.0</version>
        </dependency>

For interactive work the README points at the Spark Packages route, where the library is resolved at submission time:

    $SPARK_HOME/bin/spark-shell --packages dibbhatt:kafka-spark-consumer:2.1.0

For sbt builds it shows the spDependencies line and, alternatively, adding the Spark Packages repository to resolvers with the artifact in libraryDependencies. The starter code is in the Spark driver, where receivers are launched by calling ReceiverLauncher.launch, and the README points at the Java or Scala example under examples/ for how to use this low level consumer.

The version matrix stops at Spark 2.x and Kafka 0.11

Compatibility is documented as a short table with three rows. Version 2.1.0 covers Spark 2.x and above with Kafka 0.10 and above and adds multi topic support. Version 2.0.0 covers Spark 2.x and above with Kafka 0.10 and above, adds Kafka security and uses the new Kafka Consumer API. Version 1.0.9 covers Spark before 2.0, meaning 1.6 and prior, all Kafka versions from 0.8.x through 0.11.x, has no Kafka security support and uses the low level SimpleConsumer API. Elsewhere the README states the consumer supports all Kafka versions 0.8, 0.9, 0.10 and 0.11 and works with all Spark versions, and then shows an example that sets spark.version to 2.2.0 and kafka.version to 1.1.0, which does not line up with the 0.11 ceiling in the table. That inconsistency is worth resolving against your own build before you start. The instructions also add a repository hosted at dl.bintray.com for the Spark Packages coordinates, so whether that artifact repository is still reachable from your network is a dependency of the Spark Packages route rather than of the local Maven build.

Apache-2.0 code with no tagged releases and a spring 2026 push

The licence is Apache-2.0, with LICENSE and NOTICE files at the root of the repository, which is a permissive grant that lets you embed the library in an internal product without copyleft obligations on your own code. The repository layout is small: pom.xml, src/, examples/scala/ and the usual metadata files, so the library itself is a single Maven module rather than a multi module build. There are no GitHub releases, which means there is no versioned artefact to fetch from the releases page and no changelog to check before an upgrade, only the version in the coordinates. The last push was on 2026-04-24 and the project is not archived, so the source remains available and you can always build 2.1.0 yourself from the tag. What you do not get is a signal about whether the library still tracks the Kafka and Spark versions you intend to run, and the compatibility table in the README is the only guidance on that.

Editorial conclusion

Adopt kafka-spark-consumer if you run Spark Streaming, need Kafka offsets tracked outside the Spark driver so a crash does not lose or replay work, and are still on the Spark 2.x and Kafka 0.10 to 0.11 generation that the README's version matrix covers. Do not adopt it for a modern deployment without checking that matrix, since the documented compatibility stops well behind current Kafka and Spark releases and the repository has no GitHub releases to fall back on. Verify first that the version you plan to use matches your Kafka client and Spark Scala version, confirm that the Zookeeper ensemble you point the offset storage at is the same one the library expects, and decide whether the block interval and block size settings give you the RDD parallelism you intend before you put it in front of production traffic.

Frequently asked questions

Official sources

1. dibbhatt/kafka-spark-consumer on GitHub
2. Issues
3. License: Apache-2.0
4. README