Narwhals DataFrame Compatibility for Python Libraries

Why Narwhals Exists Between DataFrame Backends

Narwhals addresses a packaging and API problem faced by Python libraries that accept user-provided DataFrames. A library may need to work with pandas for conventional analysis, Polars for eager or lazy execution, PyArrow for table-oriented workflows, and distributed or query-backed systems such as Spark or Dask. Writing separate implementations of the same transformation creates duplicated logic and makes behavior harder to keep aligned. Dispatching directly on concrete backend types shifts that burden into every adapter.

Narwhals exposes a subset of the Polars API through its own typed interface. The intended user is a library author who wants reusable transformation code without requiring the host application to change its backend. The README states that Narwhals provides full API support for cuDF, Modin, pandas, Polars, and PyArrow, with lazy-only support for Dask, DuckDB, Ibis, PySpark, SQLFrame, Daft through `narwhals-daft`, and DataFusion through `narwhals-datafusion`.

The important boundary is that computation remains native to the input. According to the README, Modin can still distribute computation, cuDF can still compute on a GPU, and PyArrow still returns a PyArrow object. Narwhals does not convert every input into pandas or Polars. That design distinguishes it from a compatibility layer implemented as unconditional conversion to one engine.

Narwhals Wrapping, Expressions, and Native Return Values

The documented data flow has three stages. First, `narwhals.from_native` accepts a native DataFrame or LazyFrame and returns an object in Narwhals' abstraction. Second, code operates on that object with the supported subset of the Polars API. Finally, `narwhals.to_native` returns the result in the native flavor associated with the input.

Narwhals separates eager and lazy APIs and represents column operations as expressions. Its type annotations distinguish native inputs through `IntoDataFrameT` and `IntoLazyFrameT`. This matters for library signatures: the same public function can accept both categories while its annotations communicate which backend family is expected. Full static typing is a documented project goal, allowing IDEs and type checkers to inspect these boundaries.

The abstraction is intentionally partial. The README repeatedly refers to a subset of the Polars API, so choosing Narwhals does not mean every Polars operation is interchangeable across supported libraries. Optional dependencies in `pyproject.toml` also show backend version constraints, including `pandas>=1.3.4` and `modin>=0.22.0`. The package itself declares `dependencies = []` and requires Python 3.10 or newer.

Installing Narwhals and Defining the First Portable Function

The README recommends pip for the most up-to-date installation. Install Narwhals in the environment where the library and its caller operate. Narwhals does not require a DataFrame library as a dependency, although the environment must already provide whichever backend the caller wants to pass in.

The following command installs the package:

pip install narwhals

After installation, define a transformation using `from_native`, an expression, and `to_native`:

import narwhals as nw
from narwhals.typing import IntoDataFrameT, IntoLazyFrameT


def agnostic_function(
    df_native: IntoDataFrameT | IntoLazyFrameT,
) -> IntoDataFrameT | IntoLazyFrameT:
    return (
        nw.from_native(df_native)
        .with_columns(
            category=nw.when(nw.col("animal").str.contains("whale"))
            .then(nw.lit("whale"))
            .otherwise(nw.lit("other"))
        )
        .to_native()
    )

The caller can pass supported native objects, including `pandas.DataFrame`, `polars.DataFrame`, `polars.LazyFrame`, `duckdb.DuckDBPyRelation`, `pyspark.sql.DataFrame`, and `pyarrow.Table`. The returned object follows the input flavor. A pandas input produces a pandas result, while a DuckDB relation remains associated with DuckDB according to the README's usage description.

Supported Backends and Native Execution Boundaries

Narwhals distinguishes full API support from lazy-only support, which affects how a library should describe compatibility. Eager backends can be used for direct DataFrame transformations and return their native result. Lazy backends are available when the operation belongs to Narwhals' lazy API, but that does not make every eager method valid for them.

This distinction is useful for library maintainers because it preserves the execution characteristics chosen by the application. The README specifically identifies DuckDB, Ibis, PySpark, Dask, and the listed plugin integrations as lazy-only paths. A function that needs distributed or query-backed execution should therefore be designed with lazy semantics rather than collecting rows eagerly.

Narwhals also handles backend-specific concerns such as pandas indexes and its type system, according to the README. However, the documentation does not promise identical physical layouts, null behavior, or implementation details after `to_native`. The stable return flavor is documented; the internal representation belongs to the backend.

The package metadata requires Python 3.10 or newer and has no required dependencies. That makes installation inexpensive for a library that already handles its own optional backend dependencies. It does not remove the need to declare or document those backend requirements in the surrounding application.

Compatibility Trade-offs and a Direct Backend Alternative

The main limitation is API coverage. Narwhals describes itself as exposing a subset of the Polars API, so a library whose transformation depends on a backend-specific operation may still need a conditional implementation. The API reference and generated completeness tables are the relevant places to check before committing to a particular operation. The README does not document automatic fallback for unsupported methods, and it does not claim that every backend can express every Polars behavior.

There is also a distinction between eager and lazy programming. A library that assumes materialization, Python objects per row, or eager indexing can lose the native execution model that makes DuckDB, Spark, or Dask valuable. Conversely, an expression-based function may expose only operations supported by the lazy API. A library should not infer that successful compatibility means identical execution timing.

A direct alternative is to write one function per backend and dispatch on the input type. That approach can expose a larger backend-specific API, but it requires maintaining multiple implementations and aligning their behavior manually. Narwhals reduces that duplication by centralizing shared operations, at the cost of restricting library code to its common API. The right comparison is therefore shared portable code against backend-specific capability.

Maintenance, Licensing, and Upgrade Signals

The repository is not archived. Its last push was on 2026-10-16, and the listed releases include v2.27.1 on 2026-10-10, v2.27.0 on 2026-10-05, and v2.26.0 on 2026-09-08. These dates indicate recent project activity in the release record, although the repository's own stability policy should still be reviewed when selecting a version.

The README links to a stable API policy and describes an opt-in backwards compatibility policy. That policy is relevant for library maintainers because a dependency upgrade can otherwise change the common surface available to downstream code. The repository's `pyproject.toml` identifies the current package version as 2.27.1, while optional dependency declarations show that supported backend versions can differ by integration.

Narwhals is distributed under the MIT license, with the license file recorded in `LICENSE.md`. That permits reuse and modification under the license's terms, subject to its conditions. This is not legal advice. A downstream project should still review the license text and its own distribution obligations before shipping a combined package.

For maintenance cost, the main work is checking backend compatibility and API availability rather than maintaining a second computation engine. The repository's test and generated documentation commands can help verify changes, including `make typing`, `make docs-dynamic-content`, and `make docs-build`.

Editorial conclusion

Library maintainers should adopt Narwhals when one public function must accept several DataFrame implementations while preserving each caller's execution model. Projects that need every backend-specific feature, unconditional eager execution, or a universal DataFrame API should keep direct backend adapters. Before adoption, verify the exact operations used by the library against Narwhals' API completeness tables and test conversion boundaries for the required backend versions.

Frequently asked questions

Official sources

1. License: MIT
2. narwhals-dev/narwhals on GitHub
3. Project website
4. README
5. Releases