# DataFusion Comet swaps Spark operators for Rust on Arrow, and keeps a JVM escape hatch everywhere

> Apache DataFusion Comet is a Spark accelerator that keeps queries in Arrow columnar format end to end, executing operators in Rust through DataFusion or in the JVM on Arrow batches. It installs as a plugin jar with no code changes, supports Spark 3.4 through 4.2, and claims about 1.8x on TPC-DS at scale factor 1000.

**apache/datafusion-comet** — Apache DataFusion Comet Spark Accelerator

- Repository: https://github.com/apache/datafusion-comet
- Website: https://datafusion.apache.org/comet
- Stars: 1,316 · Forks: 414
- Language: Scala
- License: Apache-2.0
- Published: 2026-10-17 · Updated: 2026-10-17 · Language: en
- Canonical page: https://hysenlabs.com/projects/apache-datafusion-comet

## It replaces operators, not your jobs

Comet's claim is that a Spark query can stay in Arrow columnar format from scan to shuffle to output, avoiding the per-row overhead of Spark's row-based engine. Operators and expressions then execute either as Rust code through the Apache DataFusion query engine, or as JVM code that operates directly on Arrow batches.

That second path is the part worth understanding. Comet is not a Rust rewrite of Spark, and it is not a foreign data source bolted onto it. The same pipeline contains native operators and JVM operators, and the boundary between them is where the design work sits.

The substitution happens at the operator level. The README lists a native Parquet reader integrated with Spark's query planner, accelerated Parquet scans when reading Apache Iceberg tables, an Arrow-IPC columnar shuffle, hundreds of supported Spark expressions across math, string, datetime, array, map, JSON, hash and predicate categories, hash aggregation with support for FILTER (WHERE ...) clauses, and hash, sort-merge and broadcast joins.

Because the substitution is per operator, coverage is not uniform. The README is careful to point at two authoritative pages, one for supported expressions and one for supported operators, rather than claiming completeness. That is the first page to read with your own workload list in hand, because an unsupported operator means a fallback and a fallback means part of the query is running the old way.

## The shuffle is Arrow-IPC, and it has a JVM fallback

Shuffle is usually where a columnar engine has to make a hard choice, and Comet describes its answer precisely: an Arrow-IPC columnar shuffle with support for hash and range partitioning, implemented in native Rust, paired with a JVM fallback for unsupported partition key types.

The fallback is the interesting part of the sentence. A columnar shuffle needs a fixed representation for the partition key so rows can be grouped, and not every partition key type has a cheap Arrow representation. Rather than refuse those types, Comet routes them to a JVM implementation. The consequence for an operator is that a query shuffling on a supported key type is fully accelerated and a query shuffling on an unsupported one is partially accelerated, with no error and no configuration to get wrong.

That design is why a fallback mechanism runs through the whole project. A configuration flag, spark.comet.explain.fallback.enabled, turns on explanation of fallbacks, and it is enabled in the example configuration the README gives. An accelerator that silently declines to accelerate part of your query is not deployable in production, so the tool that tells you which part was skipped is not a convenience feature. It is the first thing to read when you evaluate a speedup figure.

Scala and Java scalar UDFs are the other escape hatch, and they work through a different mechanism. Spark's whole-stage codegen keeps them inside the Comet pipeline rather than ejecting the query back to Spark's engine, so a UDF-heavy query keeps the accelerated parts it can.

## The plugin block, line by line

Installation is a jar on the classpath plus a plugin declaration. The README gives a shell invocation that shows the whole shape:

```shell
export COMET_JAR=/path/to/comet-spark-spark3.5_2.12-<version>.jar

$SPARK_HOME/bin/spark-shell \
    --jars $COMET_JAR \
    --conf spark.driver.extraClassPath=$COMET_JAR \
    --conf spark.executor.extraClassPath=$COMET_JAR \
    --conf spark.plugins=org.apache.spark.CometPlugin \
    --conf spark.shuffle.manager=org.apache.spark.sql.comet.execution.shuffle.CometShuffleManager \
    --conf spark.comet.explain.fallback.enabled=true \
    --conf spark.memory.offHeap.enabled=true \
    --conf spark.memory.offHeap.size=4g \
    --conf spark.executor.memoryOverhead=2g
```

What each line is doing is worth stating, because two of them are not optional in practice.

The artifact name encodes both versions you must match, comet-spark-spark3.5_2.12, which is the Spark line and the Scala binary version. The Maven Central badge in the README points at an artifact of the same shape for Spark 4.0 with Scala 2.13, so the naming convention is confirmed across versions.

The jar goes to the driver and to every executor, which is why both extraClassPath settings appear. spark.plugins registers the plugin class, and spark.shuffle.manager points Spark's shuffle at Comet's implementation rather than Spark's own. The last four lines are the memory story: off-heap Arrow memory is enabled and sized at 4g, and executor memory overhead is raised to 2g, because Arrow batches live outside the JVM heap and the memory has to be accounted for somewhere.

Comet supports Spark 3.4, 3.5, 4.0, 4.1 and 4.2, and the installation guide is named as the source for the version, Java and Scala compatibility matrix.

## The performance claim, read precisely

The README leads with a number, and the number deserves to be quoted exactly rather than paraphrased: about a 1.8x speedup over Spark 4.2 for TPC-DS at scale factor 1000, which is a one terabyte dataset, resulting in about 45 percent cost savings. Two benchmark images are linked from the 1.1.0 results directory, one per query and one of speedup per query.

What the claim does and does not cover matters. It is a single benchmark suite at a single scale factor against a single Spark version, published by the project about itself, with a benchmarking guide linked for methodology. It is not a claim about your workload.

The second half of the argument is the part that generalises better. The same acceleration means finishing the same workload in just over half the time on the cluster you already have, or matching current performance on just over half the resources, and the README frames that as a choice rather than a requirement. It also states that Comet runs on commodity hardware with no GPUs, FPGAs or other specialised accelerators, so the saving comes from utilisation rather than from buying something.

The repository backs the claim with tooling. A benchmarks/ directory sits at the root and the Makefile has a comment explaining that the benchmark suite is invoked through benchmarks/micro/run.py, which reads its arguments from a print-benchmark-args target rather than restating them, so that many suites run against one build instead of rebuilding for each. That is the detail to look for if you intend to reproduce the numbers: the harness is designed for repetition, and BENCH_HEAP defaults to 20g with a comment telling you to lower it on a smaller machine.

## Building it means Maven and Rust in the same tree

The top-level layout describes a project with two toolchains. The Maven side is the whole Apache project: pom.xml at the root, the mvnw and mvnw.cmd wrapper scripts, a.mvn/ directory, and the standard Apache release metadata in.asf.yaml, NOTICE.txt and LICENSE.txt. The Rust side is a rust-toolchain.toml at the root and a native/ directory, with common/, conf/, contrib/, dev/, docs/, kube/, spark/, spark-integration/ and benchmarks/ alongside.

The Makefile ties them together. The default target all is core plus jvm, so a full build produces both the native side and the JVM side, and the individual targets are named core and jvm. Test and release targets exist alongside them, plus a rat target.

The one knob worth noting is COMET_FEATURES. The Makefile passes it through as a --features argument to the native build when it is set, and the comment above it names an example, hdfs-opendal, which tells you the native library has optional capabilities compiled in rather than detected at runtime. Optional native features are the kind of thing that changes what works in a given environment, so this is the variable to check when a native path fails on your cluster.

A kube/ directory at the root is also worth a look for anyone running this in Kubernetes, since it suggests the project's own deployment material lives in the repository rather than in a separate ops repository.

## Where Comet is the wrong tool

The unsupported case first, because it is the one that bites. Comet's coverage is per operator and per expression, and the project publishes those lists rather than claiming completeness. A workload built on a long tail of expressions will fall back on parts of itself, and since fallbacks are silent unless you turn on the explain flag, the failure mode is a disappointing benchmark rather than an error.

The second case is a UDF heavy pipeline. Scala and Java UDFs are supported, and they stay in the pipeline, but they execute in the JVM. A query whose cost is dominated by the UDF body will not see a large gain, and a team that measures only an end to end time will not know why.

The third is operational, not technical. The integration is a plugin jar and a set of Spark configuration values, which means it must be added to the driver and executor classpaths, the shuffle manager must be replaced, and off-heap memory must be budgeted. That is entirely reasonable for a platform team that owns the cluster configuration, and it is a blocker in an environment where job authors are handed a preconfigured session.

The fourth is scale. A one terabyte TPC-DS workload is the case the project reports on. Whether a smaller, latency sensitive dashboard query benefits at all depends on whether fixed overhead or data volume dominates, and the README gives no figure for that case. Anyone with a workload below the reported scale factor should measure before committing, using the fallback explain flag to see how much of the query is actually being accelerated.

## Conclusion

Adopt Comet when you run Spark SQL or DataFrame workloads at a scale where operator overhead dominates and you can afford to add a plugin jar to the cluster classpath and off-heap memory. Do not adopt it if your pipeline is mostly Scala UDFs, since those stay in the JVM and only the surrounding pipeline accelerates, or if you cannot change Spark configuration, because the integration is a plugin rather than a library call. Verify first the operator and expression coverage against your actual queries, since Comet falls back per operator and the fallback explain flag exists precisely to tell you how often that happens.

## FAQ

### What Spark versions does Apache DataFusion Comet support?

The README states support for Spark 3.4, 3.5, 4.0, 4.1 and 4.2, and points to the installation guide for the detailed version, Java and Scala compatibility matrix. The jar name encodes both, for example comet-spark-spark3.5_2.12.

### Do I need to change my Spark application code to use Comet?

The README describes Comet as a drop-in accelerator requiring no code changes to existing Spark SQL, DataFrame or PySpark code. Integration is done by putting the jar on the driver and executor classpaths and setting the plugin and shuffle manager configuration.

### What happens when Comet does not support an operator or expression?

It falls back, rather than failing. The shuffle has a JVM fallback for unsupported partition key types, and the configuration flag spark.comet.explain.fallback.enabled turns on explanation of what fell back, which is how you find out how much of a query is actually accelerated.

### Can Scala or Java UDFs run inside Comet?

Yes. The README states that Scala and Java scalar UDFs are supported and can be kept in the Comet pipeline via Spark's whole-stage codegen. They still execute in the JVM, so a query dominated by UDF work gains less than one dominated by native operators.

## Sources

- [apache/datafusion-comet on GitHub](https://github.com/apache/datafusion-comet)
- [License: Apache-2.0](https://github.com/apache/datafusion-comet/blob/main/LICENSE)
- [Project website](https://datafusion.apache.org/comet)
- [README](https://github.com/apache/datafusion-comet/blob/main/README.md)
- [Releases](https://github.com/apache/datafusion-comet/releases)

---

Hysen Labs editorial analysis, written from the project's own repository and release notes. Cite the canonical page: https://hysenlabs.com/projects/apache-datafusion-comet