Fugue: One Interface for Pandas, Spark, Dask and Ray

The rewrite problem Fugue targets

A Pandas function that maps a column of letters to food names is four lines long. Moving that same function onto Spark normally means rewriting it against mapInPandas and an Iterator of DataFrames, plus converting the input, copying the schema from the Spark DataFrame, and wrapping the mapping in a generator. The README shows that PySpark equivalent side by side with the Fugue call, and the difference is not cosmetic: the PySpark version has a conversion branch, a schema extraction step, and a wrapper function, none of which exist in the original Pandas code. Fugue's claim is that the original function stays untouched and remains usable on Pandas DataFrames. The audience is data practitioners who write in Pandas or SQL, not engineers who already think in RDDs and query plans. If your team is comfortable in native PySpark, Fugue adds a layer you may not want.

What transform() actually does with your function

The mechanism is a signature contract. Fugue inspects the function you pass to transform(), sees that it takes a DataFrame and a mapping dictionary, and executes it per partition on the chosen engine. The schema argument is the part that is not optional in practice. Passing schema="*" means all input columns appear in the output, which works in the README example because map_letter_to_food mutates an existing column rather than adding one. The moment your function returns a different shape, you have to describe that shape to Fugue, because distributed engines need to know the output types before execution. Parameters travel separately through the params dictionary, so the mapping never has to be closed over or serialized implicitly. The return value is engine-native: a Spark DataFrame when the input was a Spark DataFrame, as the README's out.show() output confirms.

Switching engines with engine_context

The fugue.api module exposes load(), save(), transform() and engine_context(). The README's run() function shows the intended pattern: wrap the body in fa.engine_context(engine), then call fa.load on a Parquet path, fa.transform with the same function and schema, and fa.save to a Parquet path. Calling run() with no argument executes on Pandas. Calling run(engine="spark") or run(engine="dask") executes the identical body on those engines. This is the concrete answer to the portability question: the toggle is one string, and it applies to everything inside the context block. What the README does not show is what happens when a function behaves differently across engines, for example when a Pandas-only library is imported inside the function body. That failure would surface at runtime on the distributed engine, not at the point where you wrote the code.

FugueSQL and Python inside the same query

FugueSQL is the second half of the project. It is described as a SQL-based language for expressing end-to-end workflows on Pandas, Spark and Dask, and the distinguishing feature is that it can call Python. The README's example runs a standard SELECT, then a TRANSFORM clause that invokes map_letter_to_food with the mapping passed in as a JSON string, under an explicit SCHEMA *. The same query returns a Pandas DataFrame by default or a Spark DataFrame when run against Spark. Note the shape of the parameter: mapping is passed as map_dict_str, a JSON dump of the dictionary, not the dictionary itself. That is a real ergonomic cost of crossing the SQL boundary, and it is visible in the README rather than hidden in documentation. FugueSQL is the part of the project with the most surface area and the most tutorial material, so budget reading time for it separately from the Python API.

Where Fugue gets in the way

The schema requirement is the honest limitation. Fugue exists because distributed engines need schemas, and it does not make that need disappear, it only moves the declaration into the transform() call. Functions with dynamic or nested output will need a schema string that is harder to write than the function itself. The second limitation is that Fugue is an interface, not an engine. It calls Spark, Dask or Ray, so you still install and configure those systems, and their failure modes are still yours. A third issue is debugging depth: a stack trace that passes through Fugue's execution layer, the engine's serialization, and your original Pandas function is longer than a trace from a native engine call. For a single small dataset, plain Pandas is the right tool and Fugue is overhead. The README's own framing, that Fugue is for parallelizing or scaling existing code, implies you should already have code worth scaling.

Fugue against Ibis

The README points to a comparisons page listing dbt, Arrow and Ibis, and the Ibis contrast is the useful one. Ibis is an expression system: you build a lazy dataframe expression and it compiles to SQL or to a backend's execution plan. Fugue does not compile your Python into an engine's native representation. It runs your Python function per partition and hands the engine a declared schema, which is why the original Pandas function needs no edits. That is the real difference in approach. Ibis gives you portability for relational operations expressed in its API; Fugue gives you portability for arbitrary Python that the engine cannot express as a plan. If your work is joins, aggregations and window functions, an expression-based approach fits better. If your work is a function that does something SQL cannot describe, Fugue's per-partition execution model is the one that applies.

Licence, releases and upgrade cost

Fugue is Apache-2.0, which permits commercial use and modification, with the usual requirements around notices and the absence of a patent grant beyond what the licence text states. That is a factual note about the licence identifier, not legal advice; check the full text against your own policy. The release cadence is visible in the repository metadata: v0.9.7 in February 2026, 0.9.6 and 0.9.5 in late January 2026, and the last push in May 2026. The 0.x version number means the API is still pre-1.0, so pin your dependency and read release notes before upgrading rather than tracking the latest version. The upgrade cost is not only Fugue's own API. Because Fugue sits between your code and Spark, Dask or Ray, a breaking change in any of those engines can reach you through Fugue, and you inherit that surface without controlling it. The project is not archived, so maintenance is ongoing, but the versioning says treat it as a dependency to pin.

Editorial conclusion

Adopt Fugue if you already have Pandas functions that need to move to Spark, Dask or Ray without a rewrite, and you are willing to declare output schemas. Do not adopt it if your logic is already written against a single engine's native API, or if you cannot accept an extra abstraction between your code and the cluster. Before committing, verify two things on your own data: whether your function's output schema can be expressed in Fugue's schema syntax, and whether the engine you target is actually installed, since Fugue calls into Spark, Dask or Ray rather than replacing them.

Official sources

1. fugue-project/fugue on GitHub
2. License: Apache-2.0
3. Project website
4. README
5. Releases