holdenk/spark-testing-base for Spark Tests

Why spark-testing-base Exists

spark-testing-base addresses a specific problem in Spark application development: tests often repeat the code required to create a local Spark context and clean it up for each suite. The README describes this as code that developers must place between suites, with the complaint that it obscures the test itself. The project packages that setup and teardown work into base classes that a test can extend.

The intended users are developers writing tests for Spark programs, especially Scala projects whose tests launch Spark in local mode. The repository is primarily Scala, and its published Maven coordinates include Scala-version-specific artifact names. The README also documents Python package distribution through PyPI and Conda, so the project has Python-facing distributions, although it does not describe the Python API or class structure.

This is a narrow testing library rather than a general Spark compatibility layer. Its value comes from reducing repeated test infrastructure. It does not claim to replace application test assertions, generate application-specific test data, or validate Spark code correctness by itself. Those concerns remain in the test code that uses the library.

The Base-Class Test Model

According to the README, the normal integration pattern is to add the library as a test-scoped dependency, extend one of the project's base classes, and write tests against the inherited setup. The README does not list the available class names, lifecycle methods, configuration options, or cleanup behavior. It identifies the project wiki as the place to find usage details, so an adopter would need to consult that wiki before writing code against a specific class.

The documented example for a project using Spark 4.0.0 uses this Scala dependency:

"com.holdenkarau" %% "spark-testing-base" % "4.0.0_2.1.2" % "test"

The artifact suffix reflects the Scala binary version visible in the example, while the Spark version appears before the underscore in the version string. This matters because the dependency must match both the Spark version and Scala version used by the test project.

The library is based partly on test-suite bases from Apache Spark that the README says are not accessible, and partly on work inspired by sscheck's ScalaCheck generators for Spark. That provenance explains the focus on test foundations and generators, but the README does not state which generators or base classes are present in the current release. Based on the repository layout, the project contains a `core` module and separate directories for Python, Kafka, logging, and a shaded connect client, but the README does not map those directories to public APIs.

Installing Scala and Python Packages

For an sbt project using Spark 4.0.0, the test dependency shown above is the first integration step. The README also gives a Maven-style dependency with a Scala 2.13 artifact:

<dependency>
  <groupId>com.holdenkarau</groupId>
  <artifactId>spark-testing-base_2.13</artifactId>
  <version>${spark.version}_2.1.2</version>
  <scope>test</scope>
</dependency>

Both examples use test scope. That keeps the testing library out of the production artifact, although it does not prove that every dependency brought into the test classpath is safe for production use.

The README documents two Python installation paths. A PyPI user can install the package with:

pip install spark-testing-base

A Conda user can install it through conda-forge with:

conda install -c conda-forge spark-testing-base

Those commands establish package availability only. The README does not show a Python import, a Python base class, or a Python test example, so a Python user would need the project wiki or source files to identify the first usable API. The repository says it is built with sbt, but the README does not provide a complete local build command for contributors.

Test JVM Memory for Local Spark

The most concrete operational requirement is memory configuration. The README warns that the default sbt testing Java options may be too small for many tests because Spark runs in local mode. Its documented example enables forking and sets an 8 GB minimum and maximum heap:

fork in Test := true
javaOptions ++= Seq("-Xms8G", "-Xmx8G", "-XX:MaxPermSize=4048M", "-XX:+CMSClassUnloadingEnabled")

For JDK 17 and later, the README says `PermSize` and `ClassUnloadingEnabled` have been removed, so it gives this shorter form:

fork in Test := true
javaOptions ++= Seq("-Xms8G", "-Xmx8G"),

The comma after the closing parenthesis is part of the README example. It presents the memory values as examples used by the project's own tests, not universal minimums. A smaller build may require different values based on available memory and test size, but the README does not provide a sizing formula. Adopting the original options unchanged on JDK 17 or later would also conflict with the README's warning about removed options.

For Maven Surefire, the documented configuration is:

<argLine>-Xmx2048m -XX:MaxPermSize=2048m</argLine>

That Surefire example includes an option the README identifies as removed on newer JDKs. It should therefore be treated as version-sensitive configuration rather than copied into every modern build.

Serial Tests and Code-Generation Settings

The README explicitly instructs users to disable parallel test execution. For sbt, the setting is:

parallelExecution in Test := false

For Surefire, the README says to set `forkCount` to 1 and `reuseForks` to true. It does not provide a complete XML or property example for those settings, so it would be unsafe to invent one. The documented requirement still identifies the intended execution model: one test process at a time, with forks reused rather than repeatedly created.

Spark SQL code-generation tests have an additional requirement. The README says to set `SPARK_TESTING=true` when testing code generation. It also gives this command for running the project's own tests:

SPARK_TESTING=yes ./build/sbt clean +compile +test -DsparkVersion=$SPARK_VERSION

The README states that `SPARK_TESTING` is set in the project's GitHub Actions. It does not explain the variable's internal effects, nor does it say that this command is the recommended path for every consumer. The concrete requirement is that code-generation tests need an environment setting beyond the dependency and base-class changes.

A useful first Scala test therefore has three parts: add the correctly suffixed dependency, extend the base class named by the project wiki, and ensure the build has adequate heap and serialized execution. If that test launches code generation, also set the documented environment variable. A Python first use is less clearly documented because the README stops at installation.

Limitations, Compatibility, and Alternatives

The main limitation is incomplete API guidance. The README explains why the library exists and how to install it, yet it sends readers to the wiki for actual class usage. It does not enumerate the base classes, explain inheritance requirements, document fixture cleanup, or show a complete Scala test file. This makes adoption harder for maintainers who need a stable public API visible in the README.

Version selection is another concern. The release notes linked by the README state that JDK 17 support exists only for Spark 3.4.0 and Spark 4 previews. That wording makes the compatibility boundary explicit even though repository activity has continued. A team should not infer support for every newer Spark or JDK combination from the existence of a newer commit.

The README lists `juanrh/sscheck`, `hammerlab/spark-tests`, and `wdm0006/DummyRDD` as alternatives. Their approaches differ from spark-testing-base's reusable local Spark test bases. The README describes sscheck as providing ScalaCheck generators for Spark and identifies it as inspiration for parts of spark-testing-base. It does not explain the internals of the other alternatives, so a precise feature comparison would require their own documentation. The defensible distinction is scope: spark-testing-base centers on shared Spark test setup, while its listed alternatives cover generators or narrower testing facilities.

Maintenance, Upgrades, and Apache-2.0

The repository is not archived, and its last push was on 2026-09-28. That is recent repository activity, but it does not by itself guarantee compatibility with a new Spark or JDK release. The listed releases are older: `v1.4.3` was released on 2023-04-25, `v1.1.2` on 2022-06-29, and `v0.3.3` on 2016-05-11. Upgrade planning should account for both repository activity and the contents of the selected published release.

The repository identifies `Apache-2.0` as its license. This identifies the project's stated licensing terms, but it does not settle questions about bundled dependencies, build plugins, or distribution-specific obligations. The README points to `SECURITY.md` for security disclosure instructions and to `CONTRIBUTING.md` and `RELEASE_NOTES.md` for related repository information.

The project does not document a semantic-versioning policy, a supported-release window, or rollback steps for a failed upgrade. Its own upgrade cost is likely to appear in dependency selection and test configuration. The artifact version embeds Spark and Scala information, the test JVM may need substantial memory, and test execution may need to be serialized. Those changes should be checked against the existing build before the base classes replace local setup code.

Editorial conclusion

Adopt it when Spark tests repeatedly need local-mode setup and teardown and the build can enforce the documented memory and execution settings. Do not adopt it without checking Spark, Scala, JDK, build-tool, and code-generation compatibility first. Verify the selected artifact version, test JVM heap, and serialized test execution before integrating it.

Frequently asked questions

Official sources

1. holdenk/spark-testing-base on GitHub
2. Issues
3. License: Apache-2.0
4. README
5. Releases