Ballista: A Distributed Query Engine That Keeps DataFusion's API

The Problem Ballista Solves

DataFusion is a fast, in-process query engine written in Rust. It runs SQL and DataFrame workloads on a single machine, which is fine until your data or your query complexity outgrows one node. Ballista extends DataFusion to run the same workloads across a cluster. The README is explicit about the pitch: you can take an existing DataFusion application, change a few lines, and get parallel execution across multiple nodes. The core change is replacing the standard SessionContext with a Ballista-aware one. The rest of your code, including table registration and SQL execution, stays the same. This is aimed at three audiences: DataFusion users going multi-node, Spark users who want a Rust-native alternative with a familiar execution model, and library builders who want reusable distributed query components. For the first group, the value is obvious: no new query language, no new API, just a different context. For the second, Ballista keeps the Spark-like model of stages, partitions, and executors, which lowers the learning curve.

Architecture: Scheduler, Executors, and the Standalone Mode

A Ballista cluster has one or more scheduler processes and one or more executor processes. The scheduler accepts job submissions from clients and assigns tasks to executors. Executors fetch tasks and report status back. This is a classic shared-nothing design. The README points to an architecture guide for details, but the visible structure is clear: the scheduler is the coordinator, executors do the work, and clients submit plans. The interesting part is the standalone mode. When you create a SessionContext::standalone(), Ballista starts all required infrastructure in the background within the same process. That means you can develop and test distributed queries without deploying a cluster. This is a sensible way to lower the barrier to entry. You write the same code you would for a cluster, but you can run it locally first. The trade-off is that standalone mode hides the network and scheduling overhead, so performance you see locally may not reflect a real cluster.

Getting Started: From DataFusion to Ballista in a Few Lines

The README shows a direct comparison. A standard DataFusion program creates a SessionContext::new(), registers a CSV, runs a SQL query, and prints results. The Ballista version changes only the context creation: SessionContext::standalone().await?. Everything else, the register_csv call, the sql call, the show call, remains identical. That is the promise, and the example is compelling. To run a real distributed setup, you need to start scheduler and executor binaries. The README mentions Docker images, Docker Compose, and Kubernetes deployment guides in the user guide. For local development, the examples directory has standalone and distributed examples. There is also a Web TUI: when the scheduler HTTP endpoint is available, opening the scheduler address in a browser, for example http://localhost:50050, redirects to a hosted Web TUI for monitoring jobs, executors, and metrics. That is a practical addition for operators.

Cargo Features and Configuration Levers

Ballista is split into crates, and each crate has optional features. The client crate has a standalone feature, enabled by default, which gives you the in-process scheduler and executor. The ballista-core crate has arrow-ipc-optimizations for better shuffle performance, enabled by default. It also has spark-compat, which is off by default and enables Spark compatibility mode via datafusion-spark. The scheduler crate has build-binary for the CLI, substrait for Substrait plan support, prometheus-metrics for metrics collection, and graphviz-support for execution graph visualization. These features matter because they change what you can do. If you want Prometheus metrics, you must enable that feature explicitly. If you want to build a binary with S3 support, you need build-binary on ballista-core. This is a typical Rust approach, but it means you need to know which features you need before you build. The testing-only force_hash_collisions feature is a nice touch for debugging hash collisions, though it is clearly not for production.

The Documented Gap Between DataFusion and Ballista

The README includes a warning in an important block: there is a gap between DataFusion and Ballista, which may bring incompatibilities. The community is actively working to close the gap. This is a honest admission, and it is a real limitation. What does that mean in practice? Your DataFusion code might use a feature or API that Ballista does not yet support. The gap could be in plan serialization, function support, or execution semantics. The README does not enumerate the specific gaps, so you cannot know in advance what will break. The mitigation is to test your specific workloads early. The standalone mode is a good place to do that; if a query fails there, it will fail on a cluster too. But the gap also means that the 'same results' promise is not unconditional. You should verify that your queries produce identical results in standalone mode before scaling out.

Alternatives and the Execution Model Difference

The most direct alternative is Apache Spark. Spark also splits plans into stages at shuffle boundaries, assigns one task per partition, and uses executors with vcores. Ballista deliberately mirrors this model, so the conceptual shift is small for Spark users. The difference is the implementation: Ballista is Rust-native, while Spark runs on the JVM. That has implications for resource usage and startup time, though the README does not provide benchmarks. Another alternative is to stay with single-node DataFusion and scale vertically, which avoids the gap entirely. A third option is to use a different distributed query engine like Presto or Trino, which are JVM-based and have a different execution model based on a coordinator and workers. The key difference is that Ballista reuses DataFusion's logical and physical plan representations, so your existing DataFusion code is portable. With Spark, you would rewrite your queries in Spark SQL or DataFrame API. With Trino, you would write SQL, not DataFusion code.

Maintenance, Licensing, and Upgrade Considerations

Ballista is part of the Apache DataFusion project, which means it is licensed under Apache-2.0. That is a permissive license, so you can use it in commercial products without copyleft obligations. The project is actively maintained, with recent releases 54.1.0, 54.0.0, and 53.0.0 all pushed in August 2026. The release cadence appears to track DataFusion's versioning, which is a double-edged sword. On one hand, you get regular updates and bug fixes. On the other, you must keep Ballista and DataFusion versions in sync, since the gap between them is version-dependent. Upgrading DataFusion without upgrading Ballista, or vice versa, could introduce incompatibilities. The README does not provide a migration guide, so you should read the release notes for each version. The maintenance cost is moderate: you need to monitor the gap, test after each upgrade, and manage the scheduler and executor processes. The Web TUI helps with operational visibility, but it does not replace a monitoring stack.

Editorial conclusion

Ballista is for DataFusion users who have hit single-node limits and want a Rust-native distributed engine without rewriting their SQL or DataFrame code. It is also for Spark users who want a lighter execution model and for library builders who need reusable scheduler and executor components. Do not adopt it if you are unwilling to accept the documented gap between DataFusion and Ballista, which may cause incompatibilities, or if you need a mature, battle-tested distributed system like Spark. Before adopting, verify that your DataFusion version matches the Ballista release, test your specific workloads against the standalone mode first, and check the current state of the gap in the project's issue tracker. The project is actively maintained with recent releases, but the gap is a real risk that you must evaluate with your own queries.

Official sources

1. Official documentation
2. Official README
3. Project repository
4. Release notes