Apache Auron swaps Spark's execution layer for a Rust native engine

Auron takes the plan Spark already optimized and rewrites what it can

The mechanism is narrower than the marketing line suggests. The README states that Auron takes a fully optimized physical plan from a distributed computing framework, maps it into DataFusion's execution plan, and performs native plan computation. Auron then walks the resulting tree and translates the operators it supports.

That translation is partial by design, and the documentation is unusually explicit about it. A physical plan may contain both native and Spark operators, and a fallback means the operator stayed in Spark with a recorded reason explaining why. The docs go further and say a fallback does not by itself mean the SQL execution failed. This is the single most important thing to understand before enabling it: you are not switching engines, you are annotating one.

The native side is Rust, built on Apache Arrow's columnar format, with SIMD used for batch processing. The Cargo workspace lists the pieces under `native-engine/`: `datafusion-ext-commons`, `datafusion-ext-exprs`, `datafusion-ext-functions`, `datafusion-ext-plans`, `auron`, `auron-jni-bridge`, `auron-planner` and `auron-memmgr`. The JNI bridge is how a JVM executor talks to that engine, which is also where the claims about eliminating JVM overhead are actually cashed out.

[workspace]
members = [
    "native-engine/datafusion-ext-commons",
    "native-engine/datafusion-ext-exprs",
    "native-engine/auron",
    "native-engine/auron-jni-bridge",
    "native-engine/auron-memmgr",
]

The workspace pins `edition = "2024"` and `version = "8.0.0"`, with `unwrap_used` and `panic` denied under clippy.

Enabling it is four properties in spark-defaults.conf

Auron is added by dropping its JAR on the Spark client classpath and setting a handful of properties. The README points at `spark-xx.xx.xx/jars/` for the JAR and `spark-xx.xx.xx/conf/spark-default.conf` for configuration. The exact block it gives is:

spark.auron.enable true
spark.sql.extensions org.apache.spark.sql.auron.AuronSparkSessionExtension
spark.shuffle.manager org.apache.spark.sql.execution.auron.shuffle.AuronShuffleManager
spark.memory.offHeap.enabled false

# suggested executor memory configuration
spark.executor.memory 4g
spark.executor.memoryOverhead 4096

Two of those lines are load-bearing and one is a cost decision. The session extension is what installs Auron into Spark's plan pipeline, and the shuffle manager swaps the shuffle format, which the README lists among the production-hardened optimizations alongside multi-level memory management and adaptive execution. Turning off off-heap memory is not cosmetic; the native engine needs that heap for its own buffers, and `spark.executor.memoryOverhead 4096` is the documented suggestion for making room.

With that in place a first run is just a Spark submission. The README's example is a TPC-DS query:

spark-sql -f tpcds/q01.sql

The same configuration applies to `spark-thriftserver` and any other client. Since the change is in shared config rather than in your SQL, the risk profile is that a fleet-wide default turns native conversion on for queries nobody tested.

The Auron tab records fallback reasons you can actually act on

This is where the project is more candid than most, and it is the feature an operator will use daily. Turning on the UI takes two properties:

spark.ui.enabled true
spark.auron.ui.enabled true

They can also be passed per invocation, and the docs are specific that the `--conf` option needs the `key=value` form:

--conf spark.ui.enabled=true --conf spark.auron.ui.enabled=true

The option defaults to `true`, so a build that enabled the session extension but never mentioned the UI still exposes it. The direct path is `http[s]://<spark-ui-host>:<spark-ui-port>/auron/`, and behind a proxy you use the normal Spark UI link and pick the Auron tab. You need the application running while looking at it, since the records are live.

Inside, the tab carries build information and a paginated SQL execution list. Selecting a SQL ID gives you conversion snapshots, per-operator fallback reasons, physical plans, the configuration values captured at conversion time, and Spark SQL operator metrics, with a link out to the matching Spark execution page. That combination is what makes Auron diagnosable: you can tell whether a slow query was slow because the operator stayed in Spark, and the captured configuration tells you which knobs were actually in force rather than which ones you think were.

Building requires nightly Rust and a JDK, locally or in Docker

The build path is the practical objection for many teams. The README requires Rust (nightly) before compiling, recommends installing it with `rustup`, and requires a JDK with `JAVA_HOME` set correctly. Auron is regularly tested on JDK 8, 11 and 17 and is also tested on JDK 21. A `rust-toolchain.toml` sits at the repository root alongside `Cargo.toml`, `Cargo.lock`, `pom.xml` and `scalafmt.conf`, so the build is a mixed Rust and JVM project driven from both ends.

The convenience is a single script that abstracts over the OS image:

./auron-build.sh --help

Run with it, locally or inside Docker using one of the supported OS images, and the build produces a fat JAR with all dependencies in `target/` for a local build or `target-docker/` for a Docker build. Two different output directories is a small thing that catches people when a scripted pipeline assumes one path.

The repository also carries a Flink extension (`auron-flink-extension/`) and a Spark UI module (`auron-spark-ui/`), which matches the README's claim that the pluggable architecture is designed for other engines. There is a TPC-DS CI workflow in `.github/workflows/`, so the TPC-DS suite used in the quick-start above is the same one the project gates itself on.

Large ORC strings hit an Arrow offset ceiling you cannot configure away

The clearest documented failure mode concerns ORC files with large String or Binary values. `spark.auron.orc.batchSize` controls native ORC decoding, and setting it to a smaller positive row count is the mitigation:

--conf spark.auron.orc.batchSize=1024

The documentation is careful about what this does and does not buy you. The key defaults to `spark.auron.batchSize`, it only governs native ORC decoding, and downstream operators may coalesce those batches, so a smaller batch is not a guarantee of a smaller working set further down the plan. Smaller batches reduce the risk of offset overflow.

And then the hard limit: they cannot accommodate an individual value that exceeds Arrow's offset limit. No configuration fixes a single row that is too large. If your tables hold blobs of that size, Auron will fall back or fail on the scan regardless of how the batch size is tuned, and that is an architectural boundary rather than a configuration mistake.

The second constraint is the one nobody mentions until it bites: Auron is only as good as the operators it converted. A query full of fallbacks pays the overhead of the extra translation layer and gains nothing. The UI is how you find out, which makes it a required part of any evaluation rather than an optional extra.

How it differs from writing the operator yourself

The realistic alternative is not another accelerator, it is writing the hot operator yourself in Scala or Java and registering it with Spark, or restructuring the query so the expensive part is a map-side aggregate. The difference in approach is where the work sits. Writing the operator means you own the code, the build and every future Spark upgrade, and you get exactly one operator. Auron moves the same work into a shared DataFusion-based engine where the operator set, the memory manager and the shuffle format are already handled, and the extension points are the DataFusion expression, function and plan traits the README points at.

The README suggests the intended contribution route directly: extend DataFusion itself and add support in Auron with modifications in plan-serde and extension translation. The repository is structured for it, with separate crates for exprs, functions and plans.

The cost side of that comparison is the build chain. Auron is a fat JAR produced by a nightly Rust toolchain, which is a different order of operational commitment for a team whose build already takes twenty minutes

Editorial conclusion

Auron fits teams that already run Spark on a fixed hardware profile and can read a conversion snapshot to see which operators went native. It is the wrong choice for a small cluster or a team unwilling to run nightly Rust in its build, and it does not replace Spark, since fallback operators still run there. Before adopting, run one of your own heaviest queries with the Auron tab enabled and count the fallback reasons before deciding anything about speed.

Frequently asked questions

Official sources

1. apache/auron on GitHub
2. License: Apache-2.0
3. Project website
4. README
5. Releases