Apache SystemDS and the runtime plan it builds before Spark sees it

One language across preparation, training, debugging and serving

Apache SystemDS is a machine learning system that describes itself as covering the end-to-end data science lifecycle: data preparation and cleaning, efficient model training, then debugging and serving. The audience is therefore not only the person fitting a model but also the person who has to clean the table beforehand and run the trained artefact afterwards.

The interesting part is where the work happens. Pipelines are written in a high-level language with R-like syntax, or through the related Python and Java APIs, with many built-in primitives available. The system then works out the runtime plan by itself, and that plan is hybrid: part of the computation runs locally in memory, part runs distributed on Apache Spark. The design bet is that the person writing the pipeline should not have to decide, per statement, whether that statement is worth shipping across the network.

This is the classic argument for a plan-generating layer, and it is also the main risk. A separate system in the middle means an extra layer between the code you write and the execution you get, and a plan that picks local over distributed on your behalf can change how a job behaves as the data grows. The documentation site, linked from the repository as the install, quick start and hello world page, is where those decisions are meant to be examined.

Three front ends over one set of primitives

The API surface is broad. There is a high-level language with R-like syntax, a Python API and a Java API, and the README states that many built-in primitives are available to them. The documentation is split accordingly, with a general SystemDS documentation site and a separate Python SystemDS documentation section under the API path.

For a Java shop this matters more than the language label suggests. SystemDS is written in Java, and the Java API is therefore the path with the least impedance from an existing JVM codebase, whether that means calling into a model training routine from a service or embedding the runtime in a batch job. The Python API exists for the same reason every Java system ends up with one, the same reason the repository runs a separate Python test workflow and publishes separate Python coverage reporting.

The cost of that breadth is that the three surfaces are not obviously interchangeable, and the README says nothing about how the R-like syntax maps onto the Python and Java calls. Anyone adopting this should expect to read the documentation for the specific front end they intend to use, because the high-level language is where the plan generation is described, and the API documentation is where the individual primitives are.

Spark for the distributed part, with GPU and federated backends alongside

The distributed half of the hybrid plan targets Apache Spark, and the README describes the backends as local, in-memory operations plus distributed operations on Spark, with additional backends existing for GPUs and for federated learning.

The two extra backends are the most interesting feature for a research team and the least documented in the repository itself. Federated learning changes the deployment story, because a federated job spans sites rather than a cluster, and GPU execution changes the cost model of the local part of the plan, since a GPU-resident dataset is not something you spill to disk. Neither is described further in the README, so from the repository alone you cannot tell what the backends support, how they are selected, or whether they are at parity with the Spark path.

There is also a practical constraint implied by the architecture. The local half of the plan runs in memory, which means the machine running it has to hold its share of the data, and the distributed half needs a Spark cluster. For a workload that fits comfortably in one machine's memory, the plan generator may well keep everything local, and the only visible cost is the system that generated that decision. For a workload that does not fit, the same pipeline needs a cluster configured, which is where most of the setup effort goes.

Every release tag is a candidate, and they are a year apart

The release list is short and thin. The three most recent tags are 3.2.0-rc2 from 2024-03-23, 3.3.0-rc1 from 2025-04-27 and 3.4.0-rc2 from 2026-07-24. Every one of them carries a release candidate suffix. There is no 3.3.0 or 3.4.0 final tag in the list, so a team looking for a versioned, non-prerelease artefact to pin will not find one in the release history.

The gaps between those candidates are about thirteen months and about fifteen months. That is a slow rhythm for a project whose last push was on 2026-10-13, and it means the repository is a moving target: the main branch moves, and the most recent tag sits several months behind it. For most users that is fine, because the plan is to take a release candidate and live with it, but it does change what an upgrade costs. There is no patch release stream to absorb fixes from.

The naming history is worth one sentence, because searches for the old name still surface. SystemDS was renamed from SystemML, and it is an Apache Top Level Project. The rename also means older documentation and blog posts refer to a different name for the same codebase.

No package manager commands in the repository, only pointers

This repository does not contain the installation steps. The README links an install, quick start and hello world page on the documentation site, and separately an install-from-source page, and that is where the commands live. There is no install snippet in the README to copy, so a reader who wants the exact commands has to leave the repository for the site.

What the repository does show is the shape of a build. The root holds a pom.xml, a .mvn directory, and folders for bin, conf, docker, dev, scripts, docs and src. That is a Maven project with a container directory, a configuration directory and a scripts directory, which tells you a build is expected to produce more than a jar: there is a config layer and a deployment layer, consistent with a system that runs distributed rather than being imported like a library.

For Python users there is a second path. The README carries badges for total and monthly PyPI downloads pointing at a PyPI project named systemds, and a dedicated Python documentation section exists under the API path. So the Python surface is packaged for pip, while the Java surface is built from source with Maven. The repository README does not say which one the quick start page leads you through.

Four CI workflows, a licence check, and coverage for two languages

The build badge block is unusually informative about how the project is policed. There are separate GitHub Actions workflows for the build, for the documentation, for a licence check and for the Java tests, plus a Python test workflow. Coverage is tracked separately for the Java side and the Python side through codecov, with a distinct flag for each.

The licence check workflow is a detail worth naming, because it reflects how the project handles its own Apache Software Foundation obligations. The root contains LICENSE and NOTICE alongside the usual Apache project files, and the ASF header is present in the source. The licence is Apache-2.0, which is the permissive Apache grant most suitable for embedding the runtime inside a commercial product, though the exact obligations of the NOTICE file are a matter for your own legal review rather than for a review page.

The rest of the root is ordinary for a large Apache project: .asf.yaml for the ASF metadata, CONTRIBUTING.md, SECURITY.md, CITATION, doap.rdf, AGENTS.md, .github for the workflows, .gitmodules for the submodules the build pulls in, and a docs directory that is built and published by the documentation workflow.

Where a plan-generating ML system is the wrong shape

There is a clear case where SystemDS is a poor fit, and it is the common one: a small script on a laptop. A system whose selling point is generating a hybrid runtime plan across local memory and a Spark cluster charges you in setup, in a JVM and Spark footprint, and in an extra language to learn, and for a dataset that fits in a single process that trade never pays back.

A second mismatch is stability. With every published tag a release candidate and no final release in the list, a team that must pin a version, audit it, and receive security fixes on a predictable cadence is working against the release pattern rather than with it. The pragmatic move is to pin a specific candidate and track the main branch yourself.

A third mismatch is documentation depth where it matters. The README is precise about the architecture, the language family and the backends, and thin about the operational details: the GPU and federated backends are named but not explained, and the repository README says nothing about rollback, on-disk formats, or how a plan behaves when the Spark cluster is undersized. The installation, quick start and hello world page on the documentation site is where those questions need answers, and it is worth reading that page end to end before committing a team to the system.

Editorial conclusion

Apache SystemDS fits teams that already keep their data science stack on the JVM and Spark and want one language covering preparation, model training, debugging and serving, with the plan generation step taken off their hands. It does not fit small-script work, teams that must ship against a stable numbered release, or projects that need the GPU and federated backends documented before they can plan around them, since the README names those backends without describing their setup. Before adopting it, read the install and quick start page linked from the repository rather than assuming a package manager path, and check which release candidate you would pin, because the tags run 3.2.0-rc2, 3.3.0-rc1 and 3.4.0-rc2. The last push was on 2026-10-13.

Frequently asked questions

Official sources

1. apache/systemds on GitHub
2. License: Apache-2.0
3. Project website
4. README
5. Releases