Sail: A Rust Spark Connect Server That Replaces the JVM, Not the Client Code
Drop-in Apache Spark replacement written in Rust, unifying batch processing, stream processing, and compute-intensive AI workloads.
At a glance
- What is it?
- Sail is an Apache-2.0 Spark Connect server written in Rust, built on Arrow and DataFusion. It keeps your PySpark code and swaps the engine underneath. The claim of a 10x speedup comes from the project's own derived TPC-H benchmarks, so treat it as a vendor figure to reproduce, not a settled fact.
- Who is it for?
- Adopt Sail if your pipelines are already PySpark against Spark Connect and you want to remove JVM startup and memory tuning from the loop, and start with the pip install plus the compatibility check script. Do not adopt it if you depend on Spark SQL strings, on behavioral parity the checker explicitly does not test, or on RDD-level APIs.
- Can I use it commercially?
- Yes. Apache-2.0 is a permissive licence: you can use, modify and sell software built on it, as long as you keep its copyright and licence notices.
- Is it still maintained?
- Yes. The repository last received commits 1 day ago.
- What is it written in?
- Mainly Rust, according to GitHub's language statistics.
Answers come from the project's GitHub data, last synced on September 15, 2026, and from our analysis. They are not legal advice.
DEEP OPEN-SOURCE ANALYSIS
The JVM tax Sail Is Trying to Remove
Spark has been the default distributed data engine for over fifteen years, as the README itself notes. Its foundation is the JVM, and that foundation is what Sail targets. The project's stated argument is that garbage collection pauses, JVM memory tuning, and slow startup are costs inherited from the runtime rather than from the query model. Sail keeps the query model and replaces the runtime with Rust.
The audience is narrow but real. If you have PySpark notebooks or jobs that connect through Spark Connect, and the friction you feel is cluster startup time, executor memory configuration, or GC tuning rather than SQL semantics, Sail is aimed at you. If your work is Spark SQL strings issued from a BI tool, or RDD-level programming, the compatibility surface described in the README does not cover you. The README lists Spark SQL dialect support and DataFrame API support as separate items, and the compatibility checker's own warning says it does not cover Spark SQL strings at all.
Spark Connect as the Seam, Arrow and DataFusion as the Engine
The mechanism is a protocol substitution. Sail implements the Spark Connect protocol, so the client side does not change. The README's example starts a server and then points an ordinary SparkSession at it:
from pyspark.sql import SparkSession spark = SparkSession.builder.remote("sc://localhost:50051").getOrCreate() spark.sql("SELECT 1 + 1").show()
Everything above that line is PySpark. Everything below it is Rust. The server parses Spark SQL with a custom Rust parser built from parser combinators and procedural macros, executes DataFrame operations with what the README calls identical semantics, and runs the plan on Apache Arrow and Apache DataFusion. Arrow supplies the columnar in-memory representation and SIMD-friendly execution; DataFusion supplies the query engine. Workers exchange Arrow columnar data directly during shuffles, which is the project's answer to join and aggregation cost.
The Python story is the part worth reading twice. The README states that Python code runs inside Sail with zero serialization overhead because Arrow array pointers allow zero-copy data sharing, and that Python, Pandas, and Arrow UDFs, UDAFs, UDWFs, and UDTFs follow Spark conventions. That is a concrete design claim, not marketing: it says the boundary between Python and the engine is a pointer handoff rather than a pickle round trip. Whether that holds for every UDF shape is something only your own UDFs can tell you.
Getting a Server Running in Three Ways
The install path is two pip commands, and the README separates the server from the client package deliberately:
pip install pysail pip install "pyspark-client"
Note the second package. It is pyspark-client, not pyspark, which matches the Spark Connect model where the client is thin and the server does the work.
The server itself has three documented launch routes. From the command line:
sail spark server --port 50051
From Python, if you want to embed it:
from pysail.spark import SparkConnectServer server = SparkConnectServer(port=50051) server.start(background=False)
And on Kubernetes, where the README shows a manifest apply and a port-forward:
kubectl apply -f sail.yaml kubectl -n sail port-forward service/sail-spark-server 50051:50051
The README points to a deployment guide for building the Docker image and writing the manifest, so sail.yaml is your file, not something the project ships. For source builds aimed at specific hardware, there is an installation guide; the README does not reproduce those flags here.
The Compatibility Checker Is a First Pass, and Says So
Migration risk is the real cost of this kind of swap, and Sail ships one tool for it:
python -m pysail.examples.spark.compatibility_check <directory>
The README labels this experimental and then lists what it does not do, which is more informative than what it does. It checks whether referenced PySpark functions are implemented in Sail. It does not verify behavioral parity. It looks at functions used in DataFrame operations and does not cover Spark SQL strings. So a clean report means your function names exist in Sail, not that they return the same rows.
That gap is where migrations fail quietly. A function can be implemented and still differ in null handling, type coercion, or ordering. The README's own migration guide is the place to look for recommended practice, but the checker is not a substitute for running your queries against both engines and diffing results. If your pipeline has no result-diffing step, adding one is the prerequisite, not the follow-up.
Where the Lakehouse and Storage Surface Actually Sits
Sail's data surface is broad on paper. Native support for Delta Lake and Apache Iceberg table formats, with catalog integrations listed as Apache Iceberg REST Catalog, AWS Glue, Unity Catalog, Hive Metastore, and Microsoft OneLake. Storage backends listed include AWS S3, Azure, Hugging Face, Cloudflare R2, Google Cloud Storage, HDFS, plain file systems, HTTP/HTTPS, and in-memory storage.
Two things stand out. Hugging Face as a first-class storage backend is unusual for a Spark replacement and lines up with the AI workload framing in the project description. The in-memory backend is useful for tests and for the kind of local iteration the pip install path implies.
The caveat is that a listed integration is a claim about existence, not depth. Catalog providers differ in which operations they support, and Iceberg REST Catalog is a specification with room for partial implementations. If your tables live behind Unity Catalog or OneLake, the question to answer before committing is which operations Sail issues against that catalog, not whether the name appears in the README.
Performance Claims and What They Are Worth
The README states roughly 10x faster than Spark and 98% cheaper on infrastructure, and points to derived TPC-H benchmarks for the basis. It also points to ClickBench results where the project says Sail outperforms Spark, popular Spark accelerators, Databricks, and Snowflake.
These are the project's own figures. The word derived in derived TPC-H is doing work: it signals a modified benchmark rather than the official TPC-H run, and the README does not lay out the derivation in the material available here. ClickBench is a public benchmark with published queries, which makes it more checkable than a private suite, but the comparison set and hardware are not described in the README text.
The honest position is that the architecture supports the claim without proving it. Removing the JVM removes GC pauses and JVM startup, and Arrow plus DataFusion is a vectorized columnar stack, so faster than JVM Spark on scan-heavy queries is plausible. A 10x multiplier across your workload is not implied by that. Reproduce it on your own queries before you put a number in a capacity plan.
When DuckDB or Trino Is the Better Fit
The closest alternative for single-node analytical work is DuckDB. It is also columnar and vectorized, and it also targets fast analytical queries, but the difference in approach is structural: DuckDB is an embedded in-process database, so there is no server, no Spark Connect protocol, and no distributed execution. You call it from Python directly. If your data fits on one machine and your code is not already PySpark, DuckDB is less machinery for the same class of query, and Sail's distributed engine buys you nothing.
For federated SQL across large clusters with an existing Hive or Iceberg footprint, Trino is the other reference point. Trino's approach is a coordinator and worker cluster speaking its own SQL dialect and its own connector model, with no attempt at Spark API compatibility. That means no PySpark migration path, but also no compatibility checker needed, because there is no pretense of drop-in behavior. Trino is the choice when the SQL is the interface and the cluster is the product; Sail is the choice when the PySpark code is the asset you are unwilling to rewrite.
Spark itself remains the alternative for anything Sail does not cover. The README's own compatibility notes are the map of that territory.
Maintenance, Licensing, and What to Check Before You Commit
The licence is Apache-2.0, which is the same licence family as Spark and permits commercial use and modification. That removes the licensing question from the decision, though it does not remove the operational one: you are running a server, and someone has to own it. The deployment guide covers Kubernetes, and the README's Kubernetes path assumes you build the Docker image and write the manifest yourself, so container maintenance is on you.
Version cadence is visible from the release list: v0.7.1 on 2026-08-24, v0.7.0 on 2026-08-03, v0.6.6 on 2026-07-07. That is a fast minor-release rhythm, which means upgrade testing is a recurring cost rather than a one-time one. Pinning pysail and testing each bump against your query set is the practical posture.
Two facts bound the judgement. First, the compatibility checker is experimental and does not test behavioral parity or Spark SQL strings. Second, the performance figures are the project's own derived benchmarks. Everything else in the README describes a coherent design: Spark Connect as the seam, Arrow and DataFusion as the engine, catalogs and object stores as the data surface. The design is legible. The evidence for your workload is not yet in hand, and the way to get it is to run the checker, then run your queries against both engines and compare.
Editorial conclusion
Adopt Sail if your pipelines are already PySpark against Spark Connect and you want to remove JVM startup and memory tuning from the loop, and start with the pip install plus the compatibility check script. Do not adopt it if you depend on Spark SQL strings, on behavioral parity the checker explicitly does not test, or on RDD-level APIs. Verify three things first: that the compatibility script reports your DataFrame functions as supported, that your Delta Lake or Iceberg tables read correctly through your catalog, and that your own query set reproduces the TPC-H-derived numbers rather than accepting the README's 10x and 98% figures.
Community notes