# chispa: PySpark assertions that show you the offending row

> chispa is a small Python library of test helpers for PySpark, built around a pair of comparison functions that print a table with the mismatching rows highlighted. It solves one narrow problem, the readability of DataFrame assertions, and it does it without wrapping the Spark API.

**MrPowers/chispa** — PySpark test helper methods with beautiful error messages

- Repository: https://github.com/MrPowers/chispa
- Website: https://mrpowers.github.io/chispa/
- Stars: 776 · Forks: 83
- Language: Python
- License: MIT
- Published: 2026-10-20 · Updated: 2026-10-20 · Language: en
- Canonical page: https://hysenlabs.com/projects/mrpowers-chispa

## The failure message is the product

chispa is a set of PySpark test helper methods whose stated purpose is descriptive error messages. The pitch is narrow on purpose. When a DataFrame assertion fails in plain pytest, the output is a Spark exception or a diff of two objects that nobody wants to read at 2am. chispa replaces that with a table where the row that broke the assertion is marked and the rows that matched are shown alongside it.

The README frames the library as a way to write high quality PySpark code, and the implementation is consistent with that narrowness. The runtime dependency list is short: prettytable at version 3 or above, and Python 3.10 to 4.0 is the supported range. The rendering of the error table is prettytable's job, which is why that library is the only non-test dependency.

The package is on PyPI, licensed MIT, and the repository carries a Makefile with install, check, test, build, publish and docs targets, a pre-commit configuration, an AGENTS.md, a CONTRIBUTING.md and an openspec directory. The documentation is built with MkDocs through mkdocs.yml and a docs directory, and the docs build is itself checked by a make target that runs mkdocs build -s, which fails on warnings.

## Column equality without building a second DataFrame

The core helper is assert_column_equality, which compares one column of a DataFrame against another column of the same DataFrame. The example in the README starts from a function that strips non-word characters out of a string, and then checks the transformed column against the expected one.

```python
from chispa import assert_column_equality
import pyspark.sql.functions as F

def test_remove_non_word_characters_short():
    data = [
        ("jo&&se", "jose"),
        ("**li**", "li"),
        ("#::luisa", "luisa"),
        (None, None)
    ]
    df = (spark.createDataFrame(data, ["name", "expected_name"])
        .withColumn("clean_name", remove_non_word_characters(F.col("name"))))
    assert_column_equality(df, "clean_name", "expected_name")
```

The point of the column-level assertion is that it avoids constructing a second DataFrame at all. You already have the expected values in the source data, so the assertion is about the values in one column against the values in another. The README makes the same point about the DataFrame-level function, noting that creating two DataFrames is slower and requires more code but is necessary for some tests.

When the comparison fails, the message is a table with the failing row highlighted in red and the matching rows in blue. The README walks through a deliberately failing case with the input matt7 against an expected matt, which is where the value of the library becomes obvious: the offending row is visible in the output instead of being buried in a schema dump.

## Whole-DataFrame comparison and the two order flags

For tests that need to compare complete DataFrames, chispa provides assert_df_equality, which takes two DataFrames and compares them. The README's example builds a source DataFrame with one column, derives an actual DataFrame with a transformed column, and builds an expected DataFrame with both columns, then compares.

```python
from chispa import assert_df_equality

def test_remove_non_word_characters_long():
    source_data = [
        ("jo&&se",),
        ("**li**",),
        ("#::luisa",),
        (None,)
    ]
    source_df = spark.createDataFrame(source_data, ["name"])

    actual_df = source_df.withColumn(
        "clean_name",
        remove_non_word_characters(F.col("name"))
    )

    expected_data = [
        ("jo&&se", "jose"),
        ("**li**", "li"),
        ("#::luisa", "luisa"),
        (None, None)
    ]
    expected_df = spark.createDataFrame(expected_data, ["name", "clean_name"])

    assert_df_equality(actual_df, expected_df)
```

The two flags that make this usable in practice are ignore_row_order and ignore_column_order:

```python
assert_df_equality(df1, df2, ignore_row_order=True)
assert_df_equality(df1, df2, ignore_column_order=True)
```

Row order matters by default, and the README gives the reason: sorting slows the function down, so ordering is not imposed unless you ask. Column order is also compared by default, but the order of fields inside struct columns is ignored regardless, including structs nested in arrays and maps. That last detail is the one that saves you from a class of false failures when your schema contains complex types, and it is the kind of behaviour that is easier to live with than to reimplement.

## Installing chispa and the shape of the test suite

Installation is a normal pip line, and Poetry users add it as a development dependency:

```sh
pip install chispa
```

```sh
poetry add chispa --group dev
```

The project itself is developed with Poetry. The pyproject.toml pins the runtime dependency on prettytable, and the development group carries pytest, pyspark at version 4.0.0, pytest-describe, pytest-cov, pre-commit and mypy. A separate optional mkdocs group holds the documentation toolchain, and the group is marked optional so it does not install by default.

Running the tests is a make target:

```sh
	@poetry run pytest tests --cov --cov-report=term-missing
```

The same pytest invocation with a different report flag produces an HTML coverage report, and another produces XML for a CI service to ingest. The check target runs the pre-commit hooks across the repository and then runs mypy against the chispa package, so the type checker is part of the standard loop rather than an optional extra.

Ruff handles both linting and formatting in the project configuration, with a target version of py310 and a line length of 120, and the formatting rules are in preview mode. Together with the pre-commit setup, that means the library's own code style is enforced by tooling that most Python teams already run.

## What chispa does not compare, and when pytest alone is enough

The limits come from the design. These assertions compare content and, optionally, order. They do not compare schemas, do not compare column types, and do not ignore a subset of columns. There is a section in the README about ignoring specific columns, so the feature exists in some form, but the two documented flags are for row order and column order only, and it would be wrong to assume a third one behaves the way a test author might guess.

The second limit is that the comparison is exact on values. There is no tolerance for floating point drift, no normalisation of null versus empty string, and no option to compare a projection of the DataFrame. The examples do pass a None alongside an expected None, so nulls are handled as ordinary values, but that is different from providing a tolerance or a comparison mode.

So the case where chispa is the wrong tool is a small suite with one or two DataFrame assertions. Adding a dependency to get a prettier failure message for a test that fails twice a year is a poor trade, and a hand-rolled assert comparing collect() output would be enough. The library earns its place in a suite with many DataFrame assertions, where the failure message is read often and the repeated debugging of a diff is the real cost.

The alternative worth comparing is not another library but a different assertion style. Comparing sorted collected rows yourself in a plain pytest assert gives you control over tolerance and projections, at the price of the table rendering that makes a failure readable at a glance. chispa is the option that trades that control for legibility.

## Maintenance pace, packaging metadata and the alpha label

The release history is a good sign for a small library. Version 0.12.0 shipped on 2026-03-24, 0.11.1 on 2025-04-12, and 0.12.1 on 2026-10-12, with the release note for the last one described as maintenance and bug fixes. The last push was on 2026-10-19, so the project is active, and the fix release came a week before that push.

One inconsistency is worth noticing. The pyproject.toml still carries version 0.12.0 as the package version even though 0.12.1 has been published, which suggests the version field in the manifest is not the single source of truth for the published version. The Makefile has separate build and publish targets, and publish runs a dry run of poetry publish before the real one, so a release is a deliberate sequence of steps rather than a single command.

The classifier for development status is Alpha, and the package declares support for Python 3.10, 3.11 and 3.12 with an operating system of OS Independent. The licence is MIT, and there is a NOTICE-binary.md in the tree alongside the standard LICENSE.md, which is worth reading if you vendor the package into a distribution.

The declared keywords are apachespark, spark, pyspark and pytest, and the maintainer list names two people alongside the original author, so the project is not a single-maintainer effort.

## Conclusion

Adopt chispa if your PySpark test suite fails on DataFrame comparisons and the resulting error message is your bottleneck, since the library does that one job in a dozen lines of setup and stays compatible with Python 3.10 to 3.12 on PySpark 4.0.0. Stay away if you need schema-level or semantic comparison beyond row content and order, since the library's assertions are content and order comparisons with a prettytable rendering. Before you depend on it, run pytest tests --cov the way the Makefile does and check that the Python 3.10 floor in pyproject.toml matches the interpreters your CI matrix actually uses.

## FAQ

### What is chispa and what does it do for PySpark tests?

It is a Python test helper library for PySpark that provides assertion methods producing descriptive error messages. The failures are rendered as a table where the mismatching row is highlighted in red and the matching rows are shown in blue.

### How do I install chispa?

Run pip install chispa, or add it to a Poetry project as a development dependency with poetry add chispa --group dev. The package requires Python 3.10 or newer and depends on prettytable for rendering the failure table.

### How do I compare two PySpark DataFrames ignoring row order?

Call assert_df_equality with the ignore_row_order flag set to True. Row order is compared by default because sorting slows the function down, and the same helper offers ignore_column_order for the column order, while the field order inside struct columns is ignored either way.

### Does chispa compare DataFrame schemas as well as values?

The documented behaviour covers the content of the rows and their order. The README does not document schema comparison, type comparison or value tolerance, so treat those as out of scope and compare them yourself if your test needs them.

## Sources

- [License: MIT](https://github.com/MrPowers/chispa/blob/main/LICENSE)
- [MrPowers/chispa on GitHub](https://github.com/MrPowers/chispa)
- [Project website](https://mrpowers.github.io/chispa/)
- [README](https://github.com/MrPowers/chispa/blob/main/README.md)
- [Releases](https://github.com/MrPowers/chispa/releases)

---

Hysen Labs editorial analysis, written from the project's own repository and release notes. Cite the canonical page: https://hysenlabs.com/projects/mrpowers-chispa