# DataFusion in Python puts a Rust query engine behind a DataFrame API

> datafusion-python binds the Apache Arrow DataFusion engine to Python, letting you run SQL or DataFrame expressions over CSV, Parquet and JSON, exchange results with Pandas and PyArrow, and run Python code inside a query. It is a foundation for building data systems, not a drop in Pandas replacement, and the packaging is a compiled extension rather than a pure Python wheel.

**apache/datafusion-python** — Apache DataFusion Python Bindings

- Repository: https://github.com/apache/datafusion-python
- Website: https://datafusion.apache.org/python
- Stars: 610 · Forks: 180
- Language: Python
- License: Apache-2.0
- Published: 2026-10-22 · Updated: 2026-10-22 · Language: en
- Canonical page: https://hysenlabs.com/projects/apache-datafusion-python

## A query planner, not a dataframe library

The distinction matters because it sets expectations. datafusion-python is described as a Python library that binds to the Apache Arrow in memory query engine DataFusion, and its stated role is to be a foundation for building new data systems in Python rather than a convenience layer over an existing one. That framing shows in the features list: queries against CSV, Parquet and JSON are executed using SQL or DataFrames, and the optimizer that rewrites them is DataFusion's own query optimizer, not something reimplemented in Python.

The list of real projects built on the bindings is short and revealing. Dask SQL uses them for SQL parsing, query planning and logical plan optimizations, then transpiles the logical plan to Dask operations for execution. DataFusion Ballista extends the bindings for distributed query use cases, and and DataFusion Ray is another distributed query engine that uses the bindings. In all three cases the Python bindings act as the planning layer and something else carries out the work, which is the clearest evidence that the value offered here is a planner you can reuse rather than a runtime you must adopt wholesale.

## One session context holds the catalog and the runtime

Everything in the API hangs off a SessionContext, and that object is more than a connection handle. It is the catalog and schema scope, the place tables are registered, the entry point for SQL and DataFrame expressions alike, and the container for the runtime and configuration that govern execution.

The context accepts a configuration and a runtime environment at construction time, both built with fluent builders:

```python
runtime = (
    RuntimeEnvBuilder()
    .with_disk_manager_os()
    .with_fair_spill_pool(10000000)
)
config = (
    SessionConfig()
    .with_create_default_catalog_and_schema(True)
    .with_default_catalog_and_schema("foo", "bar")
    .with_target_partitions(8)
    .with_information_schema(True)
    .with_repartition_joins(False)
    .with_repartition_aggregations(False)
    .with_repartition_windows(False)
    .with_parquet_pruning(False)
    .set("datafusion.execution.parquet.pushdown_filters", "true")
)
ctx = SessionContext(config, runtime)
```

That block is where the real control lives. Partition count, repartitioning behaviour for joins, aggregations and windows, Parquet pruning and a general key value configuration mechanism are all exposed, and the documentation points to a section on maximizing CPU usage for tuning parallelism. Printing the context shows the settings currently in force, which is a practical way to confirm that a configuration you passed was actually applied.

## Registering data and querying it two ways

A Parquet file can be registered directly and then queried with SQL:

```python
from datafusion import SessionContext

# Create a DataFusion context
ctx = SessionContext()

# Register table with context
ctx.register_parquet('taxi', 'yellow_tripdata_2021-01.parquet')

# Execute SQL
df = ctx.sql("select passenger_count, count(*) "
             "from taxi "
             "where passenger_count is not null "
             "group by passenger_count "
             "order by passenger_count")

# convert to Pandas
pandas_df = df.to_pandas()
```

The DataFrame API is the other door in, and it composes with expressions rather than strings. The register_view example builds a frame from a Python dictionary, filters it, and makes it visible to SQL:

```python
from datafusion import SessionContext, col, literal

# Create a DataFusion context
ctx = SessionContext()

# Create sample data
data = {"a": [1, 2, 3, 4, 5], "b": [10, 20, 30, 40, 50]}

# Create a DataFrame from the dictionary
df = ctx.from_pydict(data, "my_table")

# Filter the DataFrame (for example, keep rows where a > 2)
df_filtered = df.filter(col("a") > literal(2))

# Register the dataframe as a view with the context
ctx.register_view("view1", df_filtered)

# Now run a SQL query against the registered view
df_view = ctx.sql("SELECT * FROM view1")

# Collect the results
results = df_view.collect()

# Convert results to a list of dictionaries for display
result_dicts = [batch.to_pydict() for batch in results]

print(result_dicts)
```

The result of that last snippet is a list with a single dictionary, so the same data can be inspected in Python without a Pandas round trip.

## Running Python inside the query

Two features distinguish this binding from a plain Arrow reader. The first is executing user defined Python code from SQL, which covers ordinary UDFs and aggregate UDFs, with dedicated examples in the repository for python-udf.py, python-udaf.py and python-udwf.py. A Python function can therefore participate in a query that is planned and optimized by the Rust engine, which is the point of using DataFusion rather than reading Parquet files into Pandas directly.

The second is interchange and portability. Results can be exchanged with Pandas and with other DataFrame libraries that support PyArrow, query plans can be serialized and deserialized in Substrait format, and there is experimental support for transpiling SQL queries into DataFrame calls for Polars, Pandas and cuDF. The transpilation is labelled experimental in the features list, and treating it that way is fair: it produces calls for another library rather than executing inside DataFusion, so the optimizer work that makes the project interesting does not travel with it.

The examples directory shows how far this goes: there is a substrait.py example, a multiprocessing pickle expression example, a ray pickle expression example, and separate FFI examples for the query planner and for the datafusion-ffi crate.

## Rust in the build, Python at the interface

This is a compiled extension and the repository layout makes that plain. The Python package is built with maturin, requiring maturin 1.8.1 or newer as the build system, and the Rust side is a Cargo workspace whose members include crates/core and crates/util, with the two FFI examples as workspace members too. The workspace pins a toolchain through rust-toolchain.toml and formats Rust sources with rustfmt.toml.

The version numbers in Cargo.toml are worth reading as a compatibility statement. The workspace version is 55.0.0, the Arrow crates are on 59, and the datafusion, datafusion-substrait, datafusion-proto and datafusion-ffi dependencies all sit at 55.1.0, with pyo3 on 0.29 and the edition set to 2024 with a minimum Rust version of 1.88. The Python package metadata declares requires-python of 3.10 or newer and classifies the project as Development Status 2, Pre Alpha while listing support for Python 3.10 through 3.14, plus free threading as 2, Beta.

On the Python side the package name is datafusion and the console entry point is not a script at all, since this project has no CLI. The license file is LICENSE.txt and the distribution license field points at it, and the keywords list dataframe, datafusion, query-engine and rust.

## Where the fit stops

Three limits show up immediately for anyone choosing a tool.

First, this is not a Pandas substitute. The bindings sit on top of a Rust engine, the package is built with maturin, and the whole point of the optimizer is to plan work across partitions, which pays off on large scans and is overhead on a small in memory frame. If your task is a filter on a list you already have, the setup cost buys nothing.

Second, Python UDFs are the classic way a query engine stops being fast, because a Python callback cannot be pushed down or vectorised the way a native expression can. The repository does have a python-udf-comparisons.py example, which suggests the comparison is treated as a real topic rather than an edge case.

Third, packaging risk is not zero. The metadata calls the project Pre Alpha, the build is a Rust extension so wheels must exist for every platform and Python version you need, and the repository publishes no GitHub releases at all, so there is no tag stream to fall back on. The last push was on 2026-10-18, which shows the code is moving, but a moving code line with no published release history is a different proposition from a moving code line with stable artifacts.

## Conclusion

datafusion-python fits teams that already live in Arrow data and want a query planner with more than one execution backend, or that need a Python UDF inside a SQL statement without leaving the DataFrame ecosystem. It is the wrong choice for a small script that only needs Pandas, because the compiled wheel and the Rust toolchain are real setup cost for a project that also classifies itself as Development Status 2, Pre Alpha in its packaging metadata. Verify three things before adopting it: that the PyPI artifact for your platform and Python version is available, that your queries can be expressed as SQL or DataFrame expressions rather than arbitrary Python, and that the version of the datafusion crate in Cargo.toml lines up with the engine your plan is expected to execute. The repository is not archived and the last push was on 2026-10-18, but it publishes no GitHub releases, so there is no tag history to pin against.

## FAQ

### How do I install datafusion for Python?

The repository builds the Python package with maturin, so the wheel is produced from Rust sources rather than from a pure Python distribution, and the project declares Python 3.10 or newer. The README does not list a pip install command, so setup details are left to the online documentation at datafusion.apache.org/python.

### What is the difference between SessionContext and a Pandas DataFrame?

A SessionContext is the query engine state: it holds the catalog and schema, registered tables and views, the configuration and the runtime environment. The DataFrame it returns is a lazy query description, and results are pulled with collect or to_pandas.

### Can I use a Python function inside a SQL query with datafusion?

Yes. The features list states that queries can execute user-defined Python code from SQL, and the examples directory ships python-udf.py, python-udaf.py and python-udwf.py for functions, aggregate functions and window functions respectively.

### Does datafusion-python support Parquet pushdown and partitioned execution?

The configuration API exposes with_target_partitions, with_repartition_joins, with_repartition_aggregations, with_repartition_windows, with_parquet_pruning and a generic set call that includes datafusion.execution.parquet.pushdown_filters. The documentation points to a section on maximizing CPU usage for tuning parallelism.

### Is datafusion-python stable enough for production?

The packaging metadata classifies it as Development Status 2, Pre Alpha and lists support for Python 3.10 through 3.14 with free threading as 2, Beta. The repository is not archived and the last push was on 2026-10-18, and it publishes no GitHub releases, so there are no published tags to pin against.

## Sources

- [apache/datafusion-python on GitHub](https://github.com/apache/datafusion-python)
- [Issues](https://github.com/apache/datafusion-python/issues)
- [License: Apache-2.0](https://github.com/apache/datafusion-python/blob/main/LICENSE)
- [Project website](https://datafusion.apache.org/python)
- [README](https://github.com/apache/datafusion-python/blob/main/README.md)

---

Hysen Labs editorial analysis, written from the project's own repository and release notes. Cite the canonical page: https://hysenlabs.com/projects/apache-datafusion-python