Data Validation Tool: comparing source and target tables after a migration

A migration step that needs proving, not assuming

The Data Validation Tool, shortened to DVT, is an open sourced Python CLI tool based on the Ibis framework that compares heterogeneous data source tables with multi-leveled validation functions.

The problem it addresses is specific. Data validation is a critical step in a data warehouse, database or data lake migration project, where data from both the source and the target tables are compared to ensure they are matched and correct after each migration step, and the migration steps it names are data and schema migration, SQL script translation and ETL migration. The failure being guarded against is the quiet one: a migration script that runs without error and produces a table with the wrong row counts, a truncated column or a subtly different aggregate. DVT provides an automated and repeatable solution to that task.

The word heterogeneous is doing real work in that description. The tool is not limited to comparing two copies of the same database. Its point is that you can validate a move from one engine to another, where the SQL dialects differ, and the comparison logic still works.

The audience is a data engineer or database administrator in the middle of a migration, and the platform team that has to sign off on the result. The secondary audience is anyone who has inherited a migration and needs evidence, rather than assurances, that the data arrived.

The project states its own status in a disclaimer that is worth reading before anything else. This is not an officially supported Google product, bugs may lurk, and the maintainers reserve the right to make small backwards-incompatible changes. The tool is Apache 2.0 licensed, the primary language is Python, and the last push was on 2026-10-19 with release v8.10.0 on 2026-10-02.

Five validation kinds and sixteen connection types

The validation types are listed plainly, and the list tells you what the tool considers a comparable difference.

Column validation covers count, sum, avg, min, max, stddev and group by. Row validation compares row sets, with the note that it is not supported for FileSystem connections. Schema validation compares the structure. Custom Query validation lets you write the comparison yourself. Ad hoc SQL exploration sits alongside them for interactive poking.

The connection list is long, which is the clearest sign of the intended use. It includes AlloyDB, BigQuery, Db2 LUW, Db2 z/OS, FileSystem, Hive, Impala, MSSQL, MySQL, Oracle, Postgres, Redshift, Spanner, Snowflake, Sybase ASE and Teradata. That combination is telling: it is not a tool for one engine, it is a tool for the case where your source is on one of them and your target is on another.

Two constraints are stated up front. The tool is designed to run in an environment connected to GCP services, specifically BigQuery, GCS and Secret Manager, and if you are running it on-premises or in an environment with restricted access to GCP services there is a separate section on running DVT at on-prem. And the documentation notes plainly that nested or complex columns are not supported for column or row validations, which in practice means a column holding a struct or an array is outside the tool's reach for those validation types.

The FileSystem exclusion for row validation is a related kind of boundary. A filesystem connection is useful for comparing files, and column and schema checks work there, but there is no row set to compare in the way the tool needs.

Connections first, then the CLI

The usage sequence in the README is: create connections to the source and target tables, then run validations on those tables.

Connections can be stored locally or in a GCS directory, and the Connections page in the docs covers how to create and list them. This matters because the connections are the part with credentials in them, and the tool is designed to pull those from Secret Manager when it is running in a GCP environment, which is why the GCP dependency appears at the top of the setup story rather than being an optional extra.

The CLI is the main interface, with several commands for creating and running validations. The column validation command takes the source and target connections and a table mapping, and the shape of that mapping is the piece to understand:

  --source-conn or -sc SOURCE_CONN
                        Source connection details
                        See: *Data Source Configurations* section for each data source
  --target-conn or -tc TARGET_CONN
                        Target connection details
                        See: *Connections* section for each data source
  --tables-list or -tbls SOURCE_SCHEMA.SOURCE_TABLE=TARGET_SCHEMA.TARGET_TABLE
                        Comma separated list of tables in the form schema.table=target_schema.target_table. Or shorthand schema.* for all tables.
                        Target schema name and table name are optional.
                        i.e 'bigquery-public-data.new_york_citibike.citibike_trips'

The tables-list is a source-to-target mapping, and the target schema and table names are optional, so the same table name in both places is the common case. The schema.* shorthand expands to all tables in a schema, which is how you validate a whole migrated schema rather than a hand-picked list.

The aggregate flags follow the same pattern, one per comparison type, each accepting a comma separated column list or * for all columns. The count flag is the only one that is not restricted to numeric columns, and the stddev flag comes with an explicit warning in the help text: not all supported SQL engines give results from STDDV_SAMP, or the engine specific equivalent, that are comparable across other supported engines, and the option may produce unreliable results.

That warning is the honest summary of the whole tool. A cross-engine comparison is only as good as the least comparable function in the set.

Results to stdout or to BigQuery, and configs you can keep

Validation results can be printed to stdout, which is the default, or output to BigQuery, which the documentation calls recommended.

The choice matters for how you use the tool. Stdout is what you want interactively, and when a validation fails in a script you want the message in the pipeline output where a human will see it. BigQuery is what you want when the validations are part of a repeated process, because the results become rows you can query, compare over time and build a dashboard on. The result handler options in the CLI take a connection name, schema and table, or the legacy BigQuery format of project id, dataset and table, and one of the flags is marked deprecated.

The other half is configurability. DVT allows you to save and edit validation configurations in a YAML or JSON file, which the documentation describes as useful for running common validations or updating the configuration. That is the feature that turns the tool from a one-shot command into something you can put in a repository and run repeatedly, which is the difference between a validation you did once and a validation that survives the next migration.

There is also a set of flags for narrowing the comparison, including grouped columns for a group by, a list of columns to exclude, and two special cases for wildcard aggregations. You can specify a list of string columns for aggregations in order to calculate an aggregation over length(string_col), and timestamp or date columns for an aggregation over unix_seconds(timestamp_col). Running an aggregation over all columns, with *, only runs over numeric columns, unless the wildcard-include-string-len or wildcard-include-timestamp flags are present. That is a precise and useful default, because comparing string lengths and epoch seconds across engines is a common way to find that a datetime column was truncated.

Ibis in the middle, and what it costs

The choice of Ibis as the framework is the architectural decision that makes the cross-engine comparison possible, and it is worth understanding what it does for you.

Ibis lets DVT express a validation once and have it translated into whatever SQL the source and target engines speak. The setup.py pins ibis-framework at exactly 7.1.0, which is a notable thing to see: an exact pin on the framework that does the dialect translation, rather than a range. Since the translation is where the subtle differences live, a moving framework version is a moving risk.

The rest of the dependency list describes the shape of the tool. google-cloud-bigquery and google-cloud-storage are the GCP integration, google-cloud-secret-manager for credentials, and google-cloud-spanner for one of the supported engines. impyla and PyMySQL cover the MySQL-family connections, psycopg2-binary for Postgres, SQLAlchemy at 2.0 with an upper bound below 2.1 because of snowflake-sqlalchemy, and pyarrow and pandas for the data handling. There is a Flask dependency, with a comment explaining that some Airflow versions depend on flask below 2.3 and above 2.2, which shows the tool is expected to run inside Airflow as well as on a laptop. There is also jellyfish for column name matching, which is how a schema validation decides that two differently cased or punctuated names are the same column.

The repository layout shows a project with real engineering around it. There is a tests/ directory, a noxfile.py for session automation, a .coveragerc and a .flake8, a cloudbuild.yaml for continuous build, a terraform/ directory for infrastructure, ci/ for pipelines, a samples/ directory, a CHANGELOG.md and a RELEASING.md, and SECURITY.md.

The samples directory is the fastest way to see intended usage, and it names the deployment targets directly: airflow, cloud_functions, cloud_run, cloud_run_jobs, docker, allow_list, bq_result_handler, bq_result_handler_grouped, bq_utils, object_comparisons, oracle and postgres.

Running it outside GCP, and the deployment options

The tool is not CLI-only in the sense of being restricted to a terminal, and the README says so. Alternatives to running DVT in the CLI include deploying DVT to Cloud Run, Cloud Functions or Airflow, with examples in the samples directory.

The Cloud Run and Cloud Functions samples exist because the natural home for a validation step in a mature data platform is a managed service that a pipeline can trigger, and because a validation that runs on a schedule after every load is worth more than one you have to remember to run. The Airflow sample fits the same pattern for teams who run their migrations in Airflow.

Running on-premises or with restricted access to GCP services is explicitly covered by a section of its own, which tells you the maintainers expected the tool to be used in environments that are not fully on Google Cloud. The plausible reading is that connections can be stored locally rather than in GCS, and that credentials come from somewhere other than Secret Manager.

For the on-premises path, the samples/docker directory and the two Dockerfiles in the repository are the reference, and the fact that a Docker sample is provided rather than only a container image reference suggests the container route is the supported way to run the tool in a locked-down network.

The licence is Apache 2.0, the package is published to PyPI as google-pso-data-validator at version 8.10.0, and the release status in setup.py is declared as Development Status :: 4 - Beta. The project also ships a Python version pin in a .python-version file.

Editorial conclusion

Adopt the Data Validation Tool if you are migrating a warehouse, database or lake between engines and need a repeatable way to prove the target matches the source, especially when both ends speak different SQL dialects. Do not adopt it for a single one-off table check, for a real-time continuous check, or if you need a hard guarantee, because the project states it is not an officially supported product and reserves the right to make small backwards-incompatible changes. Verify first that your source and target engines are both in the supported connection list, that your target supports the aggregate functions the validation needs, and that you can read the validation results out of stdout or BigQuery in a form your pipeline can act on.

Frequently asked questions

Official sources

1. GoogleCloudPlatform/professional-services-data-validator on GitHub
2. Issues
3. License: Apache-2.0
4. README
5. Releases