DataComPy: comparing DataFrames across Pandas, Polars, Spark and Snowflake

It started as a PROC COMPARE replacement for Pandas

DataComPy's stated origin is worth knowing, because it explains the shape of the whole library. The README says it was created to be something of a replacement for SAS's `PROC COMPARE` for Pandas DataFrames, with some more functionality than just `Pandas.DataFrame.equals(Pandas.DataFrame)`.

That comparison names the gap precisely. `equals` returns a single boolean. When it returns False you learn that the two frames differ, and nothing about where, how many rows, how many columns, or whether the difference is a rounding artifact or a genuine mismatch. A data engineer reconciling two extracts of the same table needs the second thing, and writing that reconciliation report by hand is exactly the chore this package removes.

The second clause in the README's description is the important one: it prints out some stats, and lets you tweak how accurate matches have to be. Tolerance control is the reason to use a comparison library rather than a set operation, because two datasets built by different code paths will disagree on floating point values in the last few digits, and a comparison that reports those as mismatches is useless.

The supported types are Pandas, Polars, Spark in both classic and Spark Connect flavors, and Snowflake. The repository topics add dask, fugue and numpy to that list, which points at the same design: a comparison of tables rather than of a specific dataframe library.

The v1 release is a deliberate break from 0.19.x

The README leads with an Important note about versioning, and it is worth reading before you install anything.

datacompy has released `v1`. The `v0.19.x` line is no longer supported, and users should upgrade to `v1` going forward. The `support/0.19.x` branch is archived and will only receive critical security fixes on a best-effort basis, with no new features and no regular maintenance. All active development targets `main`.

That is an unusually blunt deprecation notice, and it is the sort of thing that bites teams with a long-lived environment. A pipeline pinned to the 0.19 series is now on a branch that is archived, and the escape hatch is a major version upgrade that will bring API changes.

The release history is short and recent. v1.1.0 was tagged on 2026-09-30, v1.0.4 on 2026-07-31 and v1.0.3 on 2026-07-29, so the v1 line has been getting patch releases within days of each other. The last push was on 2026-10-19, and the repository is not archived.

The packaging is modern Python. `requires-python` is `>=3.10.0` with classifiers for 3.10 through 3.13, the build backend is `setuptools.build_meta` requiring `setuptools>=64`, and the version is dynamic. Runtime dependencies are jinja2, numpy, ordered-set, pandas and polars with the pandas extra. The presence of jinja2 explains the HTML output, and polars being a hard dependency rather than an extra means the import cost is paid by every user.

The CLI turns a data reconciliation into a CI gate

The command line interface is the part of the library that has grown most, and it is the reason to look at DataComPy even if you never import it.

The README says DataComPy ships a `datacompy` command so that ad hoc checks and CI pipelines do not need a throw-away script. The simple form compares two files and prints a report:

# Compare two files and print a report
datacompy compare --left before.csv --right after.csv --on id

The second example is the one worth copying, because it shows what a build gate looks like:

# Fail a build on any difference, with a JSON report saved as an artifact
datacompy compare \
    --left before.parquet --right after.parquet \
    --on account_id,as_of_date \
    --abs-tol balance=0.01 \
    --max-unequal-rows 0 \
    --report-format json --output reports/diff.json --quiet

Per-column absolute tolerance, a cap on the number of unequal rows, JSON output to a file for archiving as a build artifact, and a quiet mode that suppresses the human-readable report so the log is not spammed when the gate passes.

The exit codes make it usable as a gate: 0 when the datasets match, 1 when they differ, and 2 on error. A two-column join key, a tolerance on one numeric column, and a zero tolerance on row count is a comparison that would be painful to write as a script and trivial to write as a command.

For inputs, CSV, Parquet and JSON are supported on the pandas, polars and Spark backends, including tab separated CSV through a `.tsv` extension, and Snowflake tables can be compared in place, which is to say without exporting them first.

build_report_data returns typed data instead of a string

Historically, a comparison produced a printed report. DataComPy now exposes a typed report object, and the README argues the case for it directly: every compare object exposes `build_report_data()` which returns a typed `ReportData` object, useful for dashboards, JSON export or custom rendering without relying on the string report.

The example shows the shape of the API:

import pandas as pd
from datacompy import PandasCompare

df1 = pd.DataFrame({"id": [1, 2, 3], "val": [10, 20, 30]})
df2 = pd.DataFrame({"id": [1, 2, 3], "val": [10, 99, 30]})

compare = PandasCompare(df1, df2, join_columns="id")

# Access structured data directly
data = compare.build_report_data()
print(data.row_summary.unequal_rows)  # 1
print(data.mismatch_stats.stats[0].column)  # 'val'

print(data.render())  # same text as compare.report()
data.save("report.html")  # HTML file
data.to_dict()  # JSON-serializable dict

The access pattern is a nested structure rather than a blob of text. `row_summary.unequal_rows` is a number, `mismatch_stats.stats` is a list of per-column records, and the first one's `column` attribute is the column that differed. That means a caller can branch on the value instead of scraping the printed report, which is the difference between a report you read and a result a program consumes.

The rendering methods live on the ReportData object itself, so `render()` reproduces the text of `compare.report()`, `save()` writes an HTML file and `to_dict()` gives a JSON-serializable dict. A full report reference is available in the Report API documentation.

Four backends, four separate documentation pages

The backend support is the part of the design that makes this more than a Pandas utility. The README lists Pandas, Spark, Polars and Snowflake or Snowpark, and each has its own documentation page under the project's documentation site.

The point is the comparison across engines. A reconciliation that has to move from a Pandas frame to a Spark DataFrame, or from a local extract to a Snowflake table, is a reconciliation that has to be rewritten, and each rewrite is a place for the comparison logic to drift. A single comparison API means the same reconciliation runs against any of them.

The implementation reality is more demanding than the API suggests. The repository's Makefile shows the test matrix, and it is large.

The default `test` target runs the suite against a classic Spark session with ANSI mode off, and `test-ansi` runs it again with `spark.sql.ansi.enabled=true`, because a comparison library can produce different results depending on how Spark handles SQL semantics. `test-connect` runs the Spark suite against a Spark Connect session using a separate pytest config, and the comment explains that only the Spark tests are included since the rest are backend-agnostic. `test-connect-regression` is a separate pytest process on purpose, because starting a local Connect server sets `SPARK_LOCAL_REMOTE` and every later `SparkSession.builder.getOrCreate()` in the same process would return the Connect session.

There is a `test-cov` target for coverage and a `test-all` aggregate that needs Java 17, pyspark with connect, and a live Snowflake session. The Makefile also states plainly that the Snowflake suites error without a live session, and that Snowpark's local testing mode is an emulator and not an alternative.

Comparing Pandas with Polars against a hand-written merge check

The realistic alternative for a quick equality check is not another library, it is a few lines of Pandas.

A hand-written check merges the two frames on the key columns, computes a boolean mask of rows that differ, and counts them. The difference in approach is that this produces a number, while DataComPy produces a number plus a per-column breakdown plus a report plus a set of structured objects. If you only need the number, the hand-written version is faster to write, has no dependency beyond Pandas, and is easy to reason about.

The hand-written version also has the failure modes DataComPy is built around. It tends to break on duplicate keys, since a merge with duplicates produces a cartesian expansion that quietly inflates the row count. It needs explicit handling for nulls, where a null equals a null in one comparison and not in another depending on how you wrote it. And it has no tolerance, so the usual workaround is to round both columns before comparing, which is a decision that quietly changes what you are checking.

DataComPy's approach is to own those decisions in one place, across four backends, with configurable absolute and relative tolerance, and to give you the results in a form you can assert on. The cost is a dependency and a learning curve on a report object with a nested structure.

For a project that compares data in CI on a schedule, that trade is usually worth it, since the reconciliation logic is written once and the failure output is a report an on-call engineer can read. For a test suite that only needs to assert two frames are identical, it is overhead, and `DataFrame.equals` is the right tool.

Licence, contribution rules and the local development loop

DataComPy is Apache-2.0 licensed, with the licence given in the `pyproject.toml` metadata as the Apache Software License and the full text in the LICENSE file at the repository root. That is a permissive licence with an explicit patent grant, which is a straightforward commercial position for a company that wants to embed a comparison step in a pipeline, and it is not legal advice.

Contributions require signing a Contributor License Agreement, and the README links to the CLA assistant for capitalone/datacompy. The project also adheres to the Open Source Code of Conduct published by Capital One, and the README states that by participating you are expected to honor it.

For contributors, the CI matrix can be run locally, and the README points to the Developer Instructions page for running the Spark and Spark Connect tracks without a hand-managed JDK. The repository layout supports that: a `Makefile` with the test targets, `tox.ini`, three separate pytest configuration files in `pytest.ini`, `pytest-ansi.ini` and `pytest-connect.ini`, a `.pre-commit-config.yaml`, a `benchmarks/` directory, a `ROADMAP.rst`, and a `CLAUDE.md` at the top level.

The `pytest-ansi.ini` and `pytest-connect.ini` split is worth noting for anyone extending the library. The ANSI mode difference is a real behavioral fork, since a comparison that passes with ANSI off can fail with it on, and a test suite that only runs the default configuration would miss that class of bug.

Editorial conclusion

Adopt DataComPy when you have to prove two datasets agree, in a pipeline or a CI job, and you want tolerance handling, a per-column mismatch breakdown and machine-readable output rather than a boolean. Do not adopt it for a quick equality check inside a test, since setting it up costs more than DataFrame.equals, and treat the v0.19.x line as dead, because the README states it is no longer supported and the branch is archived. Verify first which extras you need, since Spark and Snowflake require explicit extras and the Snowflake suites need a live session to run at all.

Frequently asked questions

Official sources

1. capitalone/datacompy on GitHub
2. License: Apache-2.0
3. Project website
4. README
5. Releases