Gluten plans for two backends, and the Spark configurations it honours are only partly documented

The pitch is a specific diagnosis of where Spark stopped improving

The background section makes an argument rather than a claim, and it is the reason the project exists.

Spark 2.0 replaced the Volcano model with Whole-Stage Code Generation, delivering up to a 2x speedup. Since then, the README says, most subsequent improvements have focused on the query plan level while the performance of individual operators has almost stopped improving.

That diagnosis is the whole scope. Gluten is not a better query planner, a cost-based optimiser or a replacement for Catalyst. It targets the layer where progress stalled, which is why the design principle is to reuse Spark's control flow while offloading compute-intensive data processing to the native side.

The context is that several native SQL engines now exist, with ClickHouse and Velox named specifically, offering native execution, columnar formats and vectorized processing that can outperform Spark's JVM engine. The gap is that none of them speaks Spark SQL directly.

So the target user is narrow and the README states it plainly: anyone who wants to fundamentally accelerate Spark SQL. And because Gluten is a plugin, it requires no changes to the DataFrame API or to your SQL queries. Users only need to configure it. That last sentence is the adoption pitch, and it is also the one to test, because the following sections qualify it.

Substrait is the interface, JNI is the boundary

The data path has four stages, and each one is a place where work happens.

Spark's physical plan is transformed into a Substrait plan, which is a well-defined cross-language specification for data compute operations. That Substrait plan is passed to the native side through a JNI call. On the native side a chain of native operators is constructed and offloaded to the native engine. Results come back as a `ColumnarBatch`, and Spark's Columnar API, introduced in Spark 3.0, is used during execution.

The underlying representation is Apache Arrow, which is why `arrow` is one of the topics on the repository. Choosing Arrow means the data crossing the JNI boundary is a standard columnar format rather than something bespoke, which is what allows two different native backends to be swapped without changing the JVM side.

The design principles list in the README makes the remaining boundaries explicit: clear JNI interfaces for native SQL engines, easy switching between available backends, reuse of Spark's distributed control flow, management of data sharing between JVM and native, and extensibility for more engines.

The components list then maps those principles onto code. Query Plan Conversion does the Substrait translation. Unified Memory Management handles native memory allocation. Columnar Shuffle reuses Spark core's shuffle service while adding a columnar exchange operator, because Spark's existing shuffle speaks rows and Gluten's data is columnar. Metrics are collected from the native engine and displayed in the Spark UI, which is what makes a native plan observable to someone who only knows Spark.

That metrics point deserves emphasis for anyone evaluating this. A native plan you cannot inspect is a black box you have to debug blind.

The fallback path is the honest limit on the claims

The Fallback Mechanism component exists because not every operator can run natively. Gluten provides fallback to vanilla Spark for unsupported operators, and does it through two conversions: `ColumnarToRow` (C2R) and `RowToColumnar` (R2C), converting between Gluten's columnar format and Spark's internal row format to support those fallback transitions.

Think about what that means in practice. When an operator falls back, data is materialised into Spark's row format, handed to the JVM engine, and the results have to come back. That conversion is a real cost, and it means a query mixing supported and unsupported operators pays both for the native execution and for the transitions.

This is the single most important thing to know before adopting Gluten, and it qualifies the claim that users only need to configure it. Configuration is what is required of the user; the fallback cost is what the project manages.

Two backends are supported today: ClickHouse and Velox. Velox is described as a C++ database acceleration library providing reusable, extensible and high-performance data processing components. The README says Gluten is designed to be extensible for additional backends in the future.

The Shim Layer is the component that determines whether you can use it at all. It ensures compatibility with multiple Spark versions, and the README says Gluten supports the latest 3 to 4 Spark releases during its development cycle and currently supports Spark 3.4, 3.5, 4.0 and 4.1.

So when you upgrade Spark past 4.1, you are waiting on the shim, not on Spark.

Turning it on is four Spark confs and a shuffle manager

The user guide gives one configuration block, and every line in it is doing work.

export GLUTEN_JAR=/PATH/TO/GLUTEN_JAR
spark-shell \
  --master yarn --deploy-mode client \
  --conf spark.plugins=org.apache.gluten.GlutenPlugin \
  --conf spark.memory.offHeap.enabled=true \
  --conf spark.memory.offHeap.size=20g \
  --conf spark.driver.extraClassPath=${GLUTEN_JAR} \
  --conf spark.executor.extraClassPath=${GLUTEN_JAR} \
  --conf spark.shuffle.manager=org.apache.spark.shuffle.sort.ColumnarShuffleManager

`spark.plugins` is the actual switch. Both classpath entries are needed because native execution happens on executors as well as the driver. `spark.memory.offHeap.enabled` and `spark.memory.offHeap.size` are not optional extras: native execution allocates memory outside the JVM heap, and the 20g in the sample is a placeholder that has to be sized for your data.

The shuffle manager line is the one most likely to be missed. Gluten reuses Spark core's shuffle service but swaps in a columnar exchange, which means the row-based shuffle manager is incompatible. If you set the plugin and leave the shuffle manager alone, the plan does not go native.

There are two ways to get the JAR. You can download the tar package from the downloads page and extract the JAR from it, or build from source, in which case the JAR is generated under `/PATH/TO/GLUTEN/package/target/`.

Nightly builds are published from the main branch for early testing, and the README states they have been verified on CentOS 7, 8 and 9 and Ubuntu 20.04 and 22.04. That list of verified platforms is the closest thing to a support statement you get, and it is worth checking against your own distribution.

Spark configuration does not survive the boundary intact

There is a sentence in the configurations section that deserves to be quoted rather than paraphrased: the Gluten Velox backend honors some Spark configurations, ignores others, and many are transparent to it.

Three categories, and only one of them is the behaviour people expect. Honoured means it takes effect. Ignored means you set it and nothing happens. Transparent means the Velox backend has its own equivalent and the Spark name is a passthrough, which is the most confusing of the three because the setting appears to work.

The README routes you to four separate documents for this: `docs/Configuration.md` for common configurations, `docs/velox-configuration.md` for Velox-specific ones, and `docs/velox-spark-configuration.md` for how Spark configurations map across the boundary. Parquet write behaviour is documented separately in `docs/velox-parquet-write-configuration.md`.

The practical implication is that migrating a workload is a configuration audit, not a flag flip. Any Spark conf your current job depends on, whether a broadcast threshold, a partition sizing or an adaptive execution setting, is a candidate for being ignored or overridden.

Building from source is also backend-specific. Velox has its own two documents, `docs/get-started/Velox.md` and `docs/get-started/build-guide.md`, and ClickHouse has `docs/get-started/ClickHouse.md`, so the two backends have genuinely different build paths.

One repository, three backends, and a native build system

The top-level layout shows the scope, and it is larger than a Spark plugin usually is.

The backend directories are `backends-clickhouse/`, `backends-velox/` and `backends-bolt/`. The third one is not mentioned in the README's supported list, so treat it as work in progress rather than a supported target.

On the JVM side the modules are named after what they integrate with: `gluten-core/`, `gluten-substrait/`, `gluten-arrow/`, `gluten-ui/`, `gluten-ut/`, and format integrations including `gluten-delta/`, `gluten-hudi/`, `gluten-iceberg/`, `gluten-paimon/`, plus `gluten-flink/`, `gluten-kafka/`, `gluten-celeborn/` and `gluten-uniffle/`. Each of those is a format or engine you use with Spark, and each needs native code that understands it.

The native side is `cpp/` and `cpp-ch/`, which correspond to Velox and ClickHouse respectively, with `shims/` holding the Spark version compatibility code.

The `Makefile` is where the build complexity shows. It sets `BUILD_TYPE=Debug` by default, with `ENABLE_ASAN`, `LDB_BUILD`, `BUILD_BENCHMARKS`, `BUILD_TESTS`, `BUILD_EXAMPLES`, `BUILD_ORC` and `ENABLE_PROTON` as switches, plus `ENABLE_HDFS` defaulting to True and `ENABLE_S3` defaulting to False. Architecture is resolved with a shell `arch` call and mapped to `amd64`. Conan package versions default to `main` for both Gluten and Bolt. The version is read out of `package/pom.xml` with `sed` specifically so jar paths stay in sync when the project version bumps, and the comment says so.

So this is a JVM project that requires a native toolchain. That is the practical cost of the approach and it is not small.

Licence is Apache-2.0, with the standard dual `LICENSE` and `LICENSE-binary` pair plus `NOTICE` and `NOTICE-binary` that an Apache release requires, and there is a `benchmarks/` directory if you want to run your own comparisons rather than trusting published numbers.

Editorial conclusion

Adopt Gluten if your Spark SQL workload is bottlenecked inside individual operators rather than at the query plan level, you already run ClickHouse or can build Velox, and you can accept a four-release shim layer across Spark 3.4 to 4.1. Do not adopt it expecting the whole query to run natively, since the fallback mechanism converts data back to Spark's row format for unsupported operators, and do not expect your Spark configuration to behave identically, because the Velox backend honours some configurations, ignores others and treats many as transparent. Before you deploy, read docs/velox-spark-configuration.md, set `spark.memory.offHeap.enabled` and `spark.memory.offHeap.size` deliberately rather than copying the sample values, and test your specific queries against the fallback path.

Frequently asked questions

Official sources

1. apache/gluten on GitHub
2. License: Apache-2.0
3. Project website
4. README
5. Releases