Spark-Redis: a connector that knows about cluster resharding

Redis data structures reach Spark as RDDs and DataFrames

Spark-Redis is a library for reading and writing data in Redis using Apache Spark, and the README is specific about scope: it provides access to all of Redis' data structures, String, Hash, List, Set and Sorted Set, from Spark as RDDs, and it also supports reading and writing with DataFrames and Spark SQL syntax.

That list of five structures is the substance of the project. A connector that only handled strings would be a thin wrapper; handling hashes, lists, sets and sorted sets means the Spark side can work with the shapes Redis actually stores, and the sorted set in particular is the structure that pipelines tend to reach for when they need ordering that a hash does not give.

The audience is data engineers who have Redis as a serving store and Spark as a processing engine, and who need to move between the two. The library can be used both with Redis stand-alone as well as clustered databases, and it also supports Spark Streaming in the DStream form and Structured Streaming, so the same connector covers batch and continuous reads.

The library is published to Maven Central under the coordinate com.redislabs/spark-redis_2.12, with javadocs available, and the repository carries badges for an integration workflow and for codecov coverage.

Cluster awareness means reacting to resharding and node failure

The one capability that distinguishes this connector from a hand written Redis client inside a Spark job is what the README calls cluster awareness. When used with a Redis cluster, Spark-Redis is aware of its partitioning scheme and adjusts in response to resharding and node failure events.

The mechanism follows from how a cluster stores data. A Redis cluster splits the keyspace across nodes by hash slot, so a Spark partition that maps to one range of slots can be read from the node that owns those slots. When slots move during a resharding, a connector that cached the mapping at startup will read from the wrong place or from a node that no longer holds the keys. A connector that listens for the cluster events and updates its mapping keeps working. The same applies when a node fails and its slots are taken over.

The practical question for an adopter is what happens to a long running job when the cluster moves underneath it. The README says the library adjusts, and the cluster documentation is filed separately under doc/cluster.md, which is where the details of that behaviour live. What the README does not claim is any tolerance for a job that must not lose or duplicate data across a reshard, so the honest reading is that the connector keeps working, not that it offers exactly once semantics.

One branch per Spark version, and the table stops at Spark 3.2.x

The compatibility table is the most important section of the README, and it is worth reading line by line. The library has several branches, each corresponding to a different supported Spark version, with the example given that branch-2.3 works with any Spark 2.3.x version, and the master branch containing recent development for the next release.

| [master](https://github.com/RedisLabs/spark-redis/)                       | 3.2.x | >=2.9.0          | 2.12                     |
| [3.0](https://github.com/RedisLabs/spark-redis/tree/branch-3.0)           | 3.0.x | >=2.9.0          | 2.12                     |
| [2.4, 2.5, 2.6](https://github.com/RedisLabs/spark-redis/tree/branch-2.4) | 2.4.x | >=2.9.0          | 2.11, 2.12               |
| [2.3](https://github.com/RedisLabs/spark-redis/tree/branch-2.3)           | 2.3.x | >=2.9.0          | 2.11                     |
| [1.4](https://github.com/RedisLabs/spark-redis/tree/branch-1.4)           | 1.4.x |                  | 2.10                     |

Read across the rows and the picture is clear. The newest Spark version the project claims support for is 3.2.x, and the newest Scala is 2.12. Every branch requires Redis 2.9.0 or later. The oldest branch, 1.4, is listed for Spark 1.4.x with no Redis version given.

That table has a consequence for anyone on a current Spark. If your platform runs Spark 3.3 or later, no row of this table covers it, and you are left choosing between pinning an older Spark or using a connector that has been written for the version you actually run. The project is not archived and the last push was on 2026-10-12, so the code is still moving, but the compatibility table and the branches are what your build will actually resolve to.

The release list stops in 2022 while the branch keeps moving

The published releases are v3.0.0 on 2021-10-19, v2.7.0 on 2022-02-10 and v3.1.0 on 2022-06-14. That is the last tagged artefact, four years before the most recent push.

The gap between a 2022 release and a 2026 commit is the kind of thing that needs a deliberate reading. A connector is versioned against the Spark it compiles against, so a change on master that adds support for a newer Spark cannot be consumed through the old artifact; it has to be built from the branch. The README states that the master branch contains the recent development for the next release, which confirms that the branch, not the tag, is where new work lands.

There is also a blunt warning in the additional considerations section: the library is a work in progress so the API may change before the official release. That sentence, sitting in the documentation of a library with a 3.x version, tells you to pin your dependency and to read the documentation on the branch you actually depend on. The README repeats that instruction elsewhere, asking users to make sure they use documentation from the correct branch, and the doc/ tree is laid out per topic, with getting-started, rdd, dataframe, streaming, structured-streaming, cluster, java, python, configuration and dev files.

The Makefile starts standalone, TLS and three cluster nodes for tests

The repository ships the test infrastructure as configuration rather than as documentation, and reading the Makefile tells you what the test suite expects to find running. The standalone node is configured with a port, a pidfile and a logfile under /tmp, persistence turned off through an empty save setting and appendonly no, and a requirepass directive:

daemonize yes
port 6379
pidfile /tmp/redis_standalone_node_for_spark-redis.pid
logfile /tmp/redis_standalone_node_for_spark-redis.log
save ""
appendonly no

A second standalone variant adds TLS, with the server itself listening on port 0 and the TLS port set to 6380, and the certificate files read from the test resources directory:

tls-auth-clients no
tls-port 6380
tls-cert-file ./src/test/resources/tls/redis.crt
tls-key-file ./src/test/resources/tls/redis.key
tls-ca-cert-file ./src/test/resources/tls/ca.crt
tls-dh-params-file ./src/test/resources/tls/redis.dh

The cluster nodes are three separate definitions with cluster-enabled yes, on ports 7379, 7380 and 7381, each with its own cluster config file under /tmp. Three nodes is the minimum for a meaningful resharding test, which tells you the integration workflow exercises topology change rather than only static reads. An ACL line for a user named alice is included in the same templates, so the suite also covers authenticated access.

No Java, Python or R bindings, and that is the first limitation to read

The known limitations section contains a single entry: Java, Python and R API bindings are not provided at this time.

That sentence deserves more attention than its length suggests, because the doc/ tree lists files named java.md and python.md, which invites the assumption that bindings exist. The limitation contradicts that reading, and the two facts together mean the Scala API is the only one to rely on. A PySpark team cannot use this connector from Python, and a DataFrame-based job in a JVM language other than Scala has no route in either. That is a considerably narrower audience than a general purpose Redis connector, and it is the reason a batch export approach is often the pragmatic choice instead.

The repository is a Maven project with a pom.xml, a scalastyle-config.xml for style checks, a project/ directory for the build definition, a build/ directory, a src/ directory and a Makefile. A LICENSE file sits at the root and the licence is BSD-3-Clause, which is permissive and places no copyleft obligation on code that depends on it, though redistributing the library itself means shipping that licence text.

The alternative is moving Redis data out before Spark touches it

There is a competing approach that requires no connector at all, and it is worth taking seriously before adding a dependency on a project whose newest release predates four years. You can read the Redis data out with a client of your choice, write it to a file or object store in a form Spark reads natively, and let Spark work on that. The difference in approach is where the work happens and what you get for it.

The export approach decouples the two systems. Redis is read at the moment of export, the data becomes a file, and the Spark job is a plain batch read with no knowledge of Redis at all. The connector instead runs inside the job: the RDD and DataFrame readers talk to Redis as the job executes, which means the data seen by a partition is the data present when that partition runs rather than a snapshot taken earlier, and the connector handles cluster topology on its own.

The trade is real. Export costs a staging step, a copy of the data, and a moment of staleness at the export boundary, and it loses the streaming options entirely, since neither DStreams nor Structured Streaming has an equivalent in a file pipeline. The connector avoids all of that but brings its own costs: a branch per Spark version, no non-Scala bindings, and an API the documentation warns may change. The choice turns on whether you need fresh reads and streaming against a clustered Redis, or whether a periodic snapshot into a file is good enough.

Editorial conclusion

Adopt Spark-Redis if your Spark jobs need to read or write Redis data and especially if you run a Redis cluster, since the connector reacts to resharding and node failure rather than assuming a fixed topology. Do not adopt it for a Spark version above 3.2.x, since the compatibility table lists no branch for it, and do not plan a Python or Java job around it, because those bindings are explicitly absent. Verify first which branch your Spark version maps to, that the doc/ files on that branch describe the API you need, and that the BSD-3-Clause licence in the LICENSE file covers how you intend to redistribute the connector.

Frequently asked questions

Official sources

1. Issues
2. License: BSD-3-Clause
3. README
4. RedisLabs/spark-redis on GitHub
5. Releases