sparkMeasure turns Spark task metrics into records you can aggregate and export

Spark listeners as the collection mechanism, not a UI scraper

sparkMeasure is described as a tool and a library for easing performance measurement and troubleshooting of Apache Spark jobs, aimed at developers and data engineers who need to understand why a job ran slowly rather than merely watch it finish. The README describes it as a tool and a library, and the practical difference between those two words is the difference between a command you run and code you call from your own pipeline.

The collection mechanism is Spark listeners. The README lists as one of its highlights that it serves as a practical example of implementing Spark Listeners for the collection of detailed Spark task metrics, and the documentation set includes a section on the main concepts underlying the implementation, with an architecture diagram and an FAQ.

That choice matters. A listener receives events from the Spark scheduler as stages and tasks start and finish, so the measurements come from inside the framework rather than from scraping the Spark UI or inferring behavior from logs. The metrics it produces are task level and stage level, which is the granularity you need to answer the questions that matter after a regression: which stage consumed the time, how many tasks skewed, what the input and output sizes were.

The authors are explicit about the intended workflow. Interactive troubleshooting in notebooks and in spark-shell or pyspark environments is the first use case, then development and CI/CD integration where you measure and compare execution metrics of Spark jobs under various configurations or code changes, then batch analysis through the Flight Recorder mode, and finally monitoring with external systems.

Flight Recorder mode records batch jobs without a live session

The batch story is handled by a mode the README calls Flight Recorder mode, in which sparkMeasure transparently records batch job metrics for later analysis. The word transparently is doing real work there, because the alternative for a production batch job is to attach a notebook to it, which is not an option for a job that runs on a schedule and exits.

The model is straightforward. The same listener machinery that powers the interactive case can be switched into a recording mode where events are written out as they happen, and the resulting data is analyzed afterwards. The documentation set lists a memory report and a CLI example for task metrics alongside the Python CLI and the Scala CLI, which tells you the output is designed to be inspected from a shell as well as from code.

For a platform team this is the feature that turns sparkMeasure from a debugging aid into a monitoring component. A nightly job that occasionally takes twenty minutes longer than usual is exactly the kind of failure that never appears in an interactive session, and recording mode means the stage breakdown still exists the next morning.

What the recording contains is stage and task metrics aggregated and exportable, which is a deliberate narrowing of scope. It is not a profiler, it does not capture thread stacks or heap dumps, and it does not replace the Spark history server. It tells you which stage was slow and how the tasks within it behaved.

Sinks for Kafka, Prometheus Push Gateway, JMX Exporter and InfluxDB

The monitoring integrations are named in the README: Apache Kafka, Prometheus Push Gateway, Prometheus JMX Exporter and InfluxDB. That is a short list, but the choice of systems says something about the intended deployment.

Prometheus appears twice and in two different modes. The Push Gateway model is for batch jobs, where a process runs to completion and must push its final metrics to a gateway that Prometheus scrapes, rather than exposing an endpoint that Prometheus can scrape while the job is alive. The JMX Exporter path instead runs a Java agent inside the Spark processes and exposes the metrics as JMX beans for a JMX scrape, which fits a long running service where Prometheus can keep pulling.

Kafka and InfluxDB cover the other two common shapes. Kafka carries the events out of the Spark job for a downstream consumer, and InfluxDB stores time series data for dashboards and alerting.

This is where the choice between tool and library matters most. A one-off interactive analysis needs nothing more than a Python object you can query in a cell, while a monitoring deployment needs a durable sink and a configuration for it. The repository structure supports both: a python/ directory that the Dockerfile installs with pip install /opt/src/python, and a src/ directory compiled into a jar that gets copied into /opt/spark/jars.

Getting sparkMeasure into a Spark session from the command line

The README documents both a Python and a Scala command line path. The Python sequence installs the client, then points pyspark at the packaged jar:

  # Python CLI
  # pip install pyspark
  pip install sparkmeasure
  pyspark --packages ch.cern.sparkmeasure:spark-measure_2.13:0.28

The Maven coordinates are the important part of that line: the artifact is ch.cern.sparkmeasure, the cross build targets Scala 2.13, and the version is 0.28, which matches the latest release tag.

The Python API that follows is short, and the README shows it calling a Spark SQL query and measuring it:

  # Import sparkMeasure
  from sparkmeasure import StageMetrics
  stagemetrics = StageMetrics(spark)
  # Simple one-liner to run a Spark SQL query and measure its performance
  stagemetrics.runandmeasure(globals(), 'spark.sql("select count(*) from range(1000) cross join range(1000) cross join range(1000)").show()')

So the pattern is to construct a metrics object bound to the Spark session, then wrap the code you want to measure. The Python package is on PyPI as sparkmeasure and the Scala library is on Maven Central, which means both a notebook user and a build pipeline can add it without vendoring source.

The repository also ships notebooks you can read as examples: a Jupyter Colab example, a local Python getting started notebook, a Spark task metrics analysis notebook, Databricks examples for both Scala and Python, and a Python test script.

Scala 2.13 and 2.12 decide which artifact you use

The README has a dedicated version compatibility section, and the setup examples are split into Spark 4 with Scala 2.13 and Spark 3 with Scala 2.12. That split is the practical constraint for anyone adding sparkMeasure to a build.

The artifact naming follows the usual Scala convention, with the suffix in the coordinates telling you the binary version, so spark-measure_2.13 is the build for Spark 4 on Scala 2.13 and a _2.12 artifact serves Spark 3 on Scala 2.12. Because the library is compiled against the Spark APIs of a given major version, mixing the wrong artifact with your Spark distribution is the first thing to check when a job fails to start.

The build itself is sbt based, with build.sbt and a project/ directory at the repository root, and the Dockerfile compiles the project in a builder stage with sbt ++${SCALA_VERSION_LONG} package, where the Scala version arrives as a build argument. Unit tests have their own entry point in run_pytest.sh, and the .github/ directory holds a workflow that runs the Scala and Python tests together.

The project also publishes a DOI through Zenodo, which is the convention for research software, and a Maven Central badge and a PyPI badge are shown in the README, so both distribution channels are maintained releases rather than ad hoc uploads.

The container image ships the Prometheus JMX exporter agent

The Dockerfile is the clearest statement of what a monitoring deployment needs. The build stage starts from a Scala sbt builder image, copies build.sbt, the project directory and src, and packages the jar. The runtime stage starts from the official Spark image, with the tag defaulting to 4.0.0-scala2.13-java17-python3-ubuntu and the Scala version passed as an argument.

The interesting part is what gets added on top of plain Spark. The JMX exporter Java agent jar is downloaded at version 1.1.0 and placed into /opt/spark/jars, with the file mode set so the Spark user can read it. That is the mechanism behind the Prometheus JMX Exporter integration, and the comment in the Dockerfile says as much: it exposes metrics sent to the JmxSink to Prometheus. The local python directory is added at /opt/src/python and installed with pip install /opt/src/python, so the Python client ships in the same image as the jar.

Two more details show the intent to make this image testable. An end-to-end rootfs is added into the image because it contains Python scripts for running end to end tests, and the Spark UI port 4040 is exposed. The container runs as the Spark user with uid 185 rather than as root.

What sparkMeasure does not measure, and where its releases stand

The scope limit is worth stating plainly. sparkMeasure collects Spark task and stage metrics. It does not capture JVM internals, it does not give you a flame graph, and it does not profile the code inside your executors. If the question is why a specific function is slow, this is the wrong tool. If the question is which stage consumed the time and how the tasks in it were distributed, it is the right one, and the README makes the same distinction by calling it a practical example of implementing Spark listeners.

There is also a size limit inherent to the approach. Metrics are accumulated in memory and aggregated, so a job with an enormous number of tasks produces a correspondingly large result set, and the aggregation step is where that pressure shows up. The README describes the tool as simplifying collecting, aggregating and exporting, which is accurate, but it does not document a sampling mode or a cap on the number of task records kept.

On maintenance, the project is clearly alive. The last push was on 2026-10-16 and the repository is not archived. The tagged releases are v0.28 on 2026-05-19, v0.27 on 2025-10-03 and v0.26 on 2025-08-29, so the release cadence is a few times a year with the last tagged version several months behind the last commit. The licence is Apache-2.0 with the text in the LICENSE file at the repository root, and the main author and contact address given in the README belongs to Luca Canali at CERN.

Editorial conclusion

sparkMeasure fits data platform teams that need per task and per stage metrics in a form they can aggregate, compare across runs and push into existing monitoring, and it does not fit workloads that need per executor JVM profiling, since the project collects Spark level metrics rather than a profiler's view; before adopting it, match the artifact coordinates to your Spark and Scala version, and note that the newest release is v0.28 from 2026-05-19 even though the last push was on 2026-10-16.

Frequently asked questions

Official sources

1. Issues
2. License: Apache-2.0
3. LucaCanali/sparkMeasure on GitHub
4. README
5. Releases