SQLFrame runs PySpark DataFrame code on a database engine instead of Spark

What SQLFrame actually replaces

The README's first line is the whole pitch: SQLFrame implements the PySpark DataFrame API in order to enable running transformation pipelines directly on database engines, with no Spark clusters or dependencies required.

That is a narrower claim than it looks at first, and understanding the narrowness is what makes the project interesting. The code you write stays PySpark-shaped. You still import from pyspark.sql, you still use session.table, groupBy, agg, withColumn, Window and F.lag. What changes is where the work happens. Instead of a Spark session turning your DataFrame operations into a distributed execution plan, SQLFrame turns them into SQL and hands that to the engine you chose.

The README lists four audiences, and they map to four different reasons to want this. Users who want a DataFrame API that uses the full power of their engine to do the processing. Users who want to run PySpark code quickly locally without the overhead of starting a Spark session. Users who want a SQL representation of their DataFrame code for debugging or sharing. And users who want to run PySpark DataFrame code without the complexity of using Spark for processing.

The second and third of those are the ones the ordinary Spark workflow is worst at. A local Spark session takes time and memory to start for what is often a small query, and a DataFrame plan is opaque when you want to show someone why a number is wrong.

The package metadata is honest about maturity. The classifier is Development Status :: 5 - Production/Stable, requires Python 3.10 or newer, and the licence is MIT.

Installing the engine extra you actually need

Installation is one pip command with an extra per engine, and the README lists them all:

# BigQuery
pip install "sqlframe[bigquery]"
# Databricks
pip install "sqlframe[databricks]"
# DuckDB
pip install "sqlframe[duckdb]"
# Postgres
pip install "sqlframe[postgres]"
# Snowflake
pip install "sqlframe[snowflake]"
# Spark
pip install "sqlframe[spark]"
# Redshift (in development)
pip install "sqlframe[redshift]"
# Standalone
pip install sqlframe

There is a conda-forge route as well, with conda install -c conda-forge sqlframe, and the README says to see the specific engine documentation for additional setup instructions, which is where credentials and connection details live.

Two of the entries carry their own status labels in the README, and they are worth noticing before you plan around them. Redshift is marked in development, and GizmoSQL and Redshift are both described as community maintained rather than maintained by the core project. GizmoSQL is not in the install list at all because it is a separate repository, gizmodata/sqlframe-gizmosql.

The base install without an extra gives you the Standalone session, which the README says can be used to generate SQL without any connection to a database engine. That is a more useful capability than it first appears, because it lets a developer turn DataFrame code into SQL text with no credentials at all, which is exactly what you want in a test or in a code review.

The core dependencies are small: prettytable for the show output, sqlglot for SQL generation, typing_extensions and more-itertools. Each engine extra adds its own driver, so google-cloud-bigquery with pandas for BigQuery and databricks-sql-connector with pyarrow for Databricks.

Two ways to activate SQLFrame, and the difference between them

SQLFrame can either replace pyspark imports or be used alongside them, and the README describes both routes with code.

The replacement route uses the activate function to set the engine to use, and then the ordinary PySpark import gives you a session backed by that engine:

from sqlframe import activate

# Activate SQLFrame to run directly on DuckDB
activate(engine="duckdb")

from pyspark.sql import SparkSession
session = SparkSession.builder.getOrCreate()

The alongside route keeps pyspark imports intact and gives a more engine-native DataFrame API:

from sqlframe.duckdb import DuckDBSession

session = DuckDBSession.builder.getOrCreate()

The distinction is not cosmetic. With activate, a codebase can switch engines by changing one call, which is what makes a single test suite able to run against DuckDB locally and BigQuery in production. With the direct import, you are already using the engine's own session class, which means the DataFrame API is more native to that engine, and the pyspark import path is only there for the parts of the codebase that still use it.

Neither route requires pyspark to be installed for the non-Spark engines, since the Spark-specific pieces only matter if you target Spark. That is worth checking against your own project, because a project that depends on pyspark only for DataFrame transformations can drop that dependency for most engines.

The session object is a SparkSession from the pyspark point of view, so the API surface is unchanged. The difference is that the object is backed by a SQL generator rather than a distributed execution engine.

How the DataFrame code becomes SQL

The README's example is a BigQuery pipeline that computes the year-over-year percentage change in the number of families with a single child, using a window function for the lag and then ordering by the absolute percentage change.

The Python is ordinary PySpark. The interesting part is what comes out the other end. Calling df.sql(optimize=True) produces a readable SQL statement with named common table expressions, and the README notes that SQLFrame generates consistently accurate yet complex SQL for engine execution by default, while the optimize option produces more human-readable SQL.

Looking at the generated query shows the shape of the translation. The group by and the count become a first CTE, the lag over the window becomes a second CTE selecting from the first, and the final select joins the arithmetic and applies engine-specific formatting functions for the display columns, with the ordering by the absolute value of the computed percentage and a limit of five.

The README also mentions that the generated SQL output can be configured, and that the documentation covers how to configure it and to use OpenAI to enhance the SQL. That is an unusual feature to find in a query builder: an optional step where an LLM is used to tidy up the SQL it produced.

There is a second configuration axis that matters more for correctness than cosmetics. SQLFrame by default uses the Spark dialect for input and output, and this can be changed to make SQLFrame feel more like a native DataFrame API for the engine you are using, with a separate configuration page for input and output dialect. So the SQL that goes in, the Spark dialect it parses, and the SQL that comes out are separately configurable, which is how a project can keep writing Spark-flavoured SQL literals while emitting BigQuery-flavoured output.

The show method and the dialect question behind it

The df.show() output in the README is plain ASCII with the column names and the values, and it renders the BigQuery results as a year, a formatted count and a formatted percentage change.

That output format is not an accident of the example. The pyproject lists prettytable as a core dependency, pinned below version 4, and prettytable is a library for rendering tables in the terminal. So SQLFrame implements the familiar show() by building an ASCII table rather than by handing the result to a Spark UI.

This is the point where the difference in philosophy between SQLFrame and Spark becomes visible. Spark's show() is a debugging affordance layered on top of a distributed engine, and the same code under SQLFrame produces the same table from a result set the engine already computed. Nothing is being re-executed, and the table is the query result rendered locally.

There is a practical consequence for anyone moving code across. Spark's DataFrame API has a large surface, and the set of operations each engine supports is not identical. SQLFrame translates the operations it recognises into SQL, and the engine then applies its own semantics to the result. Where an operation has no equivalent in the target engine, the behaviour is determined by what the engine does with the generated SQL, which is not always the behaviour you would get from Spark.

That is the practical risk in adopting it, and it is not documented as a limitation in the README, which is worth noting. The engine-specific documentation pages are where the differences would be described, and each engine has its own page among the documentation set.

Where SQLFrame is the wrong tool

There are cases where this project is the wrong answer, and the most obvious one is distributed processing.

SQLFrame's premise is that the work happens inside one database engine. That is a win for a query that a warehouse can do well, and a loss for a job that genuinely needs to fan out over a cluster. Spark exists for the second case, and SQLFrame does not remove that need, it redirects the computation elsewhere. If your workload is a hundred-terabyte shuffle, the database engine now carries it, and the failure mode moves rather than disappears.

The second case is code that depends on Spark-specific behaviour rather than the DataFrame API shape. The README positions the project as implementing the PySpark DataFrame API, and the API is not the whole of Spark. Anything reaching for Spark internals, or for the parts of the ecosystem built on them, is outside what SQLFrame offers.

The third is Redshift specifically, where the install command in the README is labelled in development. GizmoSQL and Redshift are also described as community maintained, so their version pinning and release cadence are not controlled by the SQLFrame maintainer.

The fourth is the dependency on sqlglot, which is pinned in pyproject.toml to a range of greater than or equal to 30.0.0 and less than 30.23. A narrow upper bound on a SQL parser means an unpinned sqlglot upgrade can change behaviour or break the build, and SQLFrame pins it that way because the generated SQL is sensitive to sqlglot's output. In an environment where the resolver fights you over sqlglot, that pin is a thing to be aware of.

For comparison, the alternative approaches differ in kind. Writing SQL by hand for each engine gives you exact control over the dialect but abandons the shared codebase. Keeping PySpark but running a local session for development keeps the API identical but retains the startup cost and the memory cost of Spark on a laptop. SQLFrame is the middle position: one codebase, the engine does the work, and the SQL is the artifact you can read when something looks wrong.

Editorial conclusion

Adopt SQLFrame if your team already writes PySpark DataFrame transformations and wants them to execute on a database engine, or wants to run the same code locally on DuckDB without starting a Spark session. Do not adopt it to replace Spark for genuinely distributed workloads, since the point is to push computation into a single engine rather than a cluster, and expect the generated SQL to be your debugging surface rather than a curiosity. Before you commit, check the engine list, since GizmoSQL and Redshift are marked community maintained and the Redshift extra is marked in development, and note that the project depends on sqlglot pinned to a narrow range, so a sqlglot upgrade can break it. The licence is MIT, the latest release is SQLFrame 4.5.0 from 2026-10-14, and the last push was on 2026-10-22.

Frequently asked questions

Official sources

1. eakmanrq/sqlframe on GitHub
2. License: MIT
3. Project website
4. README
5. Releases