high-performance-spark-examples: the working code behind a Spark tuning book

What the repository actually contains

high-performance-spark-examples holds the examples for a book on high performance Spark, and the primary language of the repository is Scala. The README is brief and there is a lot in the directory listing that the prose does not spell out. There is a core/ directory holding the Scala examples, plus c/, native/ and accelerators/ for the parts that touch native code, and sql/ and python/ for the SQL and PySpark content, including a high_performance_pyspark/ directory. There is a spark-pipeline-ex/ directory for pipeline examples and an iceberg-workshop-solutions/ directory, which tells you the project has moved beyond plain Spark SQL into the lakehouse ecosystem. There is a migration/ directory, a target-validator/ directory and a conf/ directory, alongside shell-scripts/, data/ and resources/. The Dockerfile at the root installs a Python data stack and then compiles the Scala with sbt.

ARG base
FROM $base

USER root
RUN pip install --no-cache-dir pyarrow pyiceberg[pandas,snappy,daft,s3fs] avro fastavro
USER dev
RUN sbt clean compile

The Python packages are the giveaway about direction. pyarrow, pyiceberg with several extras, avro and fastavro cover columnar data and table formats, so the examples have grown from tuning Spark SQL to working with Iceberg tables, and the sbt clean compile at the end of the Dockerfile means the image is built as a development environment rather than a slim runtime.

The book is the point, and the second edition is in progress

The README states that the project is in the progress of updating it for Spark 4, with some parts that depend on external libraries like Iceberg and Comet still on 3.x, and the second edition of the book. That is the most important fact about the repository for anyone planning to use it, because it tells you the code is a moving target with a stated destination rather than a finished artifact. The version skew is specific and worth planning for. The core Spark 4 work is under way, but the parts that depend on libraries which have not yet caught up remain on the 3.x line, so a reader of a chapter on Iceberg integration will be looking at Spark 3 code while a chapter on tuning the core engine will be looking at Spark 4. The repository has no GitHub releases, so there is no tagged version to check out, and the last push to the repository was on 2026-05-03 with the repository not archived. The top level also carries a Dockerfile and a Dockerfile-mini, a build_container.sh and a run_container.sh, an env_setup.sh, several run scripts named for the kinds of example they launch, and an appveyor.yml, which dates the CI configuration to a period when AppVeyor was still a common choice.

Building means sbt plus a C and Fortran toolchain

The building section is short and splits the work in two. Most of the examples can be built with sbt, and the C and Fortran components depend on gcc, g77 and cmake. That split is a hint about what the examples are measuring, since native code in a Spark job is a common source of performance problems that a JVM-level profiler cannot see, so the book needs runnable native components to demonstrate the point. The project layout matches that split, with separate c/ and native/ directories, and the presence of a .jvmopts file and .scalafix.conf and .scalafmt.conf files alongside a scalastyle-config.xml shows that the Scala side is linted and formatted as well as compiled. The build definition is a single build.sbt at the root, and the presence of a project/ directory is the standard sbt convention for plugins and build settings. The test section adds one more requirement, that the full test suite depends on having the C and Fortran components built as well as a local R installation available, which is a heavy requirement for a test suite and reflects how the examples are paired with statistical analysis of the timing data.

The event log is how the examples are meant to be read

The history server section is the most useful operational part of the README, and it tells you how to see what the examples are actually doing. By default the history server writes to /tmp/spark-events, so you need to create that directory if it is not set up.

The command is `mkdir -p /tmp/spark-events`, run once.

The scripts for running the examples generally run with the event log enabled, and you can set the environment variable before running the Scala tests and you get the history server too.

The command is `SPARK_EVENTLOG=true sbt test`.

Not every run is a test though. Some are not the byproduct of tests but instead come from main, and the README gives the assembly and submit sequence for one of them.

sbt assembly
${SPARK_HOME}/bin/spark-submit --name more  --class com.highperformancespark.examples.perf.SimplePerfTest --conf spark.eventLog.enabled=true ./core/target/scala-2.13/core-assembly-0.0.2-SNAPSHOT.jar

The class name tells you the shape of the examples, with a perf package containing a SimplePerfTest, and the Scala 2.13 target and the snapshot assembly name in the path. To view the history server you launch it with the script from the Spark distribution and then browse to the local port.

The launch command is `${SPARK_HOME}/sbin/start-history-server.sh`, and the history server is then on the local port 18080.

The README also says the most accurate way of seeing how the tests are run is to look at the GitHub workflows, which is the right advice for a repository whose commands differ per example.

The repository spans several Spark surfaces at once

The directory names show the scope, and it is broader than a tuning book would suggest. The sql/ and core/ directories cover the two main programming surfaces, and python/ with high_performance_pyspark/ covers the Python one, so the examples exist in all three of the languages a Spark user is likely to reach for. The run scripts are named accordingly, with run_pyspark_examples.sh, run_scala_spark_ex.sh, run_sql_examples.sh and run_pipeline_example.sh, and there is a separate spark-pipeline-ex/ directory for pipeline level examples. The iceberg-workshop-solutions/ directory and the migration/ directory point at a second concern, which is moving existing workloads onto newer table formats, and target-validator/ is presumably something that checks the targets a pipeline is meant to reach. The accelerators/ directory is the most open-ended of the lot, since hardware acceleration is where the largest gains usually live, and the se_simple.json and se_complex.json files at the root, with a pretty variant of the complex one, look like saved Spark event or session configuration used by the examples. The conf/ and resources/ directories hold the shared configuration and data, and the shell-scripts/ directory at the root alongside the top level scripts suggests that some examples are driven by shell rather than by sbt.

Where this repository is the wrong choice

The first limit is that it is an examples repository rather than a library, so nothing here is packaged for you to depend on. The code is meant to be read, run and modified, and the absence of tagged releases means there is no version to pin, so a course that assigns a chapter will find the code moving underneath the student unless they work from a commit. The second is the Spark 4 transition, where the mixed version state is the current condition rather than a temporary one, and a reader who copies an example without checking which Spark it targets may not reproduce the numbers in the book. The third is the toolchain cost. The full test suite needs the C and Fortran components built and a local R installation, and the native components need gcc, g77 and cmake, so this is not something you can check out and run in five minutes on a laptop. The fourth is that the examples are only as good as the environment they were measured in, since the whole point is performance, and a Spark job tuned on one cluster with one data size tells you little about another, so the numbers need re-measuring even when the code runs unchanged. The fifth is the topic's own nature, since performance work in Spark is as much about data layout, file formats and cluster configuration as it is about code, and a repository can only cover the last of those.

Editorial conclusion

Adopt this repository if you are tuning Spark and want worked examples that come with the profiling setup, since the event log workflow is the part most tuning advice leaves out. Skip it if you are looking for a maintained library, since it is an examples repository with no tagged releases and an update in progress for Spark 4. Verify first which Spark version your build targets, since the README says some parts that depend on external libraries such as Iceberg and Comet are still on the 3.x line, and that you have the C and Fortran toolchain if you intend to run the full test suite.

Frequently asked questions

Official sources

1. high-performance-spark/high-performance-spark-examples on GitHub
2. Issues
3. README