cudf-spark, running Spark SQL on GPUs without changing the answers

A plugin library, not a fork of Spark

The NVIDIA cuDF plugin for Apache Spark provides a plugin library that leverages GPUs to accelerate processing via the cuDF libraries, which are the CUDA DataFrame implementations. The important word in that sentence is plugin. The project does not fork Spark, does not patch the engine and does not ask you to change your build. It plugs into the existing extension points, and the same Spark application runs with or without it.

That design is what makes the compatibility requirement reasonable. Because the plugin sits inside Spark's execution layer, the observable behaviour of a query has to match what Spark would have produced, or every existing test and every downstream dashboard changes. The README puts it as a single sentence: the SQL plugin tries to produce results that are bit for bit identical with Apache Spark.

Bit-for-bit is a strong claim and it is the one an engineer should hold the project to. Not equal within a tolerance, identical. Floating point addition is not associative, and a different summation order on the GPU produces a different answer in the last bits. For a project that trains a model on those results, or that reconciles two systems, a last-bit difference is a real incident, so the requirement is not pedantry, it is the price of being allowed to switch the hardware underneath.

The practical consequence is that the acceleration is per operator rather than blanket. There is a compatibility document in the repository at docs/compatibility.md that records which operators are supported, and a reader's first question should be whether the operators their pipeline depends on are in it.

The documentation for the current release is published on the project's own site at nvidia.github.io/cudf-spark, and the getting started guide is the entry point for trying the plugin out.

A rename that has not finished propagating

The project used to be called something else, and the transition is documented in a paragraph that is worth reading carefully because it affects coordinates, links and expectations.

The NVIDIA cuDF plugin for Apache Spark was formerly known as the RAPIDS Accelerator for Apache Spark. The RAPIDS name will be sunset over time. GitHub links from spark-rapids will redirect to cudf-spark. And the artefact names will remain the same for now.

That last clause is the one that matters practically, and the repository shows it. The dependency example for an external project declares the artefact as rapids-4-spark_2.12, with the group com.nvidia and the scope provided. So the Maven coordinates still carry the old name long after the project, the documentation and the GitHub links have moved on.

A reader should not be surprised by that, and should not read the old name in a coordinate as evidence of an old version. The dependency example for an external project shows the shape of the declaration, and the artefact name still carries the old branding:

<dependency>
    <groupId>com.nvidia</groupId>
    <artifactId>rapids-4-spark_2.12</artifactId>
    <version>26.12.0-SNAPSHOT</version>
    <scope>provided</scope>
</dependency>

The scope there is provided, which is the recommendation the README gives to anybody building functionality on top of the plugin, because the plugin is provided by the cluster rather than bundled with the application.

The version in that same snippet, 26.12.0-SNAPSHOT, follows the calendar versioning the project uses, with the year and month in the major and minor positions and the patch level last. The published releases follow it: v26.08.1 in August 2026, v26.08.0 earlier the same month, and v26.06.1 in July.

The build target naming in the artefact tells you the other thing the coordinates carry, which is the Scala binary version the plugin is built against, in this case 2.12, while the repository contains a scala2.13 directory for the Scala 2.13 build. A team planning an upgrade has to watch both the plugin's release cycle and Spark's, because the artefact name encodes one of them.

Configuration, tuning, and the tools that prove the parity claim

Three separate pieces of documentation and tooling exist for this project, and each answers a different question.

The configuration is Spark configs, and they are documented in the repository at docs/configs.md. Since the plugin is a set of Spark plugins, the knobs are the ordinary Spark configuration mechanism, which means the usual advice applies: a Spark config that changes behaviour is a config that changes behaviour on the CPU path too unless it is specific to the plugin.

Tuning is documented separately in a tuning guide on the NVIDIA documentation site, and it is a separate document because tuning a GPU DataFrame engine is not the same activity as tuning Spark. The amount of work a query does, where the partitions fall, whether an operation can stay on the GPU at all, these are the questions the guide addresses.

The third piece is the one that matters most for trust, and it lives in a different repository, NVIDIA/cudf-spark-tools, with separate qualification and profiling documentation. Both tools are documented separately, with a qualification guide and a profiling guide on the NVIDIA documentation site, and the readme points to the tools repository for them.

The split into a separate tools repository is not unusual, and it is the right split, because these are diagnostic tools with their own release cycle, aimed at a user who is deciding whether to adopt the plugin rather than at a user who has already adopted it. But it does mean the answer to the question is not in the plugin's own tree, and a team evaluating a migration has to know the tools exist before they go looking for them.

Zero copy export to an ML library, and the memory caveat

The integration section describes a specific capability and a specific problem.

The cuDF plugin provides some APIs for doing zero copy data transfer into other GPU enabled applications, and the documentation for it is on the NVIDIA site under additional functionality and machine learning integration. The project says it is currently working with XGBoost to try to provide that integration out of the box, and the word try is doing honest work in that sentence.

The problem is RMM caching. You may need to disable RMM caching when exporting data to an ML library, because that library will likely want to use all of the GPU's memory, and if it is not aware of RMM it will not have access to any of the memory that RMM is holding.

That is a real and specific failure mode, and it is worth understanding rather than working around blindly. The cuDF stack manages GPU memory through a caching allocator, which is what makes repeated allocations cheap and is central to the performance of the plugin. An XGBoost process that does not know about that allocator will try to allocate the memory it wants, find the cache holding it, and either fail or be pushed into a smaller allocation. The advice is to disable the caching behaviour for the export path, trading some speed for interoperability.

Zero copy here means the data moves from the Spark GPU memory into the ML library without a host round trip, which is the difference between a pipeline that keeps the data on the device and one that pays a copy for every step. It is also why the memory caveat matters so much: the whole benefit depends on there being one pool of device memory, and a second library that does not respect the pool breaks the assumption.

The repository is a family of plugins, not one artefact

The root listing is the clearest evidence of scale, and it is worth walking through because the shape of the tree is the shape of the product.

There is a sql-plugin/ directory and, beside it, a set of sibling directories that look like the parts of it: sql-plugin-api, sql-plugin-columnar, sql-plugin-fileio, sql-plugin-format and sql-plugin-shims. A separate shuffle-plugin/ directory suggests the shuffle path is a distinct component from the SQL path, which is a sensible split because the shuffle is where a GPU engine has the most freedom and the most risk.

Then the tools around it. There is an aggregator/ for the published artefact, an api_validation/ directory that suggests the public API is checked against something, an udf-compiler/ and a DF_UDF_README.md for user defined functions on data frames, a datagen/ directory for test data, an integration_tests/ directory, a jenkins/ directory, and a tests/ directory whose README the top-level readme points to for how the tests are described.

The format integrations have their own directories, iceberg/, iceberg-common/ and delta-lake/, which tells you the plugin is expected to work with the table formats that matter in practice, not only with plain files.

The supporting material is modern and worth noting: an AGENTS.md, a .claude/ directory, a skills/ directory, a .greptile/ configuration, a CODE_REVIEW_GUIDELINES.md, a .pre-commit-config.yaml, a scalastyle-config.xml for the Scala style rules, and both a NOTICE and a NOTICE-binary file, which is the pair a project needs when it ships a binary artefact with third-party components. There is also a python/ directory, a scripts/ directory and a thirdparty/ directory.

Where the cuDF plugin is the wrong choice

The first limit is hardware. The whole project is about NVIDIA GPUs, and a cluster without them gains nothing from it. That is obvious, and it is still the first question, because Spark's plugin model means the same jar can be present on a mixed cluster and silently do nothing useful on the nodes without accelerators.

The second is that the acceleration is per operator. The compatibility document is the contract, and an operation that is not on it runs on the CPU, which means a query can end up with a mixed execution plan where part of the work is on the device and part is not, and the cost of moving data between the two can exceed the gain. The bit-for-bit guarantee is per operator too, so the parity claim only covers what the table covers.

The third is the version coupling. The artefact name encodes the Scala version, the plugin has its own release cadence, and Spark and the plugin build have to line up. A team that upgrades Spark has to find the plugin build for that Spark line, and a team that pins the plugin has to pin Spark, which is a constraint on the platform team rather than on the data team.

The fourth is the memory model. The RMM caveat in the integration section is not an isolated problem, it is the shape of the whole approach. A caching allocator that makes the engine fast also makes it incompatible with libraries that do not know about it, and every downstream GPU library in a data stack has to be checked for that.

The fifth is the migration itself. Nothing changes in the application, which is the appeal and the trap: the same query runs on CPU and on GPU, so a pipeline can be moved without a code change and without a failing test, and the only way to know it is still correct is to run the qualification tool on the data. A team that does not budget for that step will find out from a downstream discrepancy.

Editorial conclusion

The plugin fits a team that already runs Spark, has NVIDIA GPUs in the cluster, and needs the same queries to keep returning the same numbers on CPU and GPU, since a difference in floating point results is what stops most Spark-on-GPU projects from being adopted. It does not fit a team without NVIDIA hardware, and it does not fit a workload whose operators are not in the compatibility table, because the acceleration is per operator rather than global. Before switching a pipeline over, read docs/compatibility.md for the operators you rely on and run the qualification tool on your own data, because bit-for-bit parity is a claim about specific operators and specific versions. The licence is a custom one that GitHub cannot classify, the release cadence is monthly, with v26.08.1 published on 2026-08-28, and the last push was on 2026-10-19.

Frequently asked questions

Official sources

1. Issues
2. NVIDIA/cudf-spark on GitHub
3. Project website
4. README
5. Releases