Datavines: data quality checks, profiling and a catalog on a plug-in architecture

Data quality as the core of a DataOps pipeline

The premise is stated in one sentence and it sets the scope of everything else. Data quality is used to ensure the accuracy of data in the process of integration and processing, and it is also the core component of DataOps. Datavines is described as an easy-to-use data quality service platform that supports multiple metrics.

So this is a data observability platform in the sense of checking data, not in the sense of tracing pipelines. The repository topics list data quality, data profiling, data catalog work, metadata management and monitoring, alongside connectors for Spark, Flink and Doris.

The audience is data engineers who own a warehouse or lakehouse and need checks to run on a schedule rather than by hand. The three feature groups map onto the daily work of such a team.

The data catalog obtains data source metadata regularly to construct a data directory, monitors metadata changes regularly, and provides tag management with support for metadata. In other words, it tells you what tables exist, what changed, and what someone labelled them.

The data quality module is the core. It has 27 built-in data quality check rules and supports 4 rule types: single table-column check, single table custom SQL check, cross table accuracy check, and two table value comparison check. That last pair is the interesting one, because it covers the checks that catch a discrepancy between two tables, which is where the expensive data incidents usually come from. It also supports scheduled tasks for checks and SLA for check result alert, so a failing check can page you rather than sit in a report.

The data profile module handles detection on a schedule, outputs a data profile report, automatically identifies column types to match appropriate profile indicators, monitors table row number trend, and provides a data distribution view.

Everything pluggable: sources, rules, engines, alerts, storage, registry

The plug-in design is stated as the basis of the platform, and the modules that support user-defined plug-ins are enumerated, which makes the extension surface unusually legible.

Data sources already supported are MySQL, Impala, StarRocks, Doris, Presto, Trino, ClickHouse and PostgreSQL. Check rules are plug-in too, with 27 of them built in, such as null value check, non-null check and enumeration check.

The job execution engine is where the interesting trade-off sits. Two engines are supported, Spark and Local. The Spark engine currently only supports the Spark2.4 version, which is a real constraint given how old that line is. The Local engine is a local execution engine developed based on JDBC, without relying on other execution engines.

That Local engine is the reason you can run this without a cluster, and the readme makes the sizing guidance explicit: if the data volume is small, or the goal is merely functional verification, you can use the JDBC engine, while if you want to run Datavines based on Spark you need to ensure your server has Spark installed.

Two more plug-in points have limits worth knowing. Error data storage supports MySQL and local files, but only the Local execution engine is supported there, so if you want to keep the rows that failed a check, you are on the Local engine. The alert channel currently supports Email, and the registry supports MySQL, PostgreSQL and ZooKeeper.

The registry choice is the one with architectural weight. ZooKeeper gives you a distributed coordination service for the service nodes, while MySQL or PostgreSQL gives you a single shared database, and the two models of horizontal scaling are not the same thing.

The module names in the tree are the architecture

The repository layout gives you the internal structure without opening a design document, and it is a proper multi-module Maven project.

The core pieces are datavines-server, which is the service itself and holds the configuration, datavines-core for the domain logic, and datavines-spi, which is the extension point other modules implement. datavines-engine holds the execution engines, datavines-runner is the job running side, datavines-connector is where the data source connectors live, and datavines-metric handles the metrics.

Around those sit the interfaces to the outside world. datavines-registry is the registry implementation, datavines-notification handles alerting, datavines-client is the client library, datavines-cli is the command line tooling, datavines-ui is the frontend, and datavines-dist is the distribution assembly, which is what produces the release package you deploy. datavines-common holds shared code, and there are deploy, docs, scripts and tools directories.

The presence of a separate spi module alongside a separate connector module is the clearest signal of the design. It means a new data source is a plugin against a published interface rather than a change to the server, and the same pattern is reused for engines, rules, registry implementations and notification channels.

The environment requirements are modest for a service like this: a java runtime environment on jdk8, plus Spark if you intend to use the Spark engine. The deployment claim is that the platform has less dependency, is easy to deploy, and minimally relies only on MySQL to start the project and complete data quality check operations.

Building it and picking a metadata database

The build needs Maven 3.6.1 and later, and the packaging command is:

$ mvn clean package -Prelease -DskipTests

The -Prelease profile produces the release package, and -DskipTests skips the test phase. The repository also carries the Maven Wrapper as mvnw with a .mvn directory, so a consistent build does not depend on the Maven version installed on your machine.

The metadata database is the decision to make before anything runs. The default datavines-server configuration uses PostgreSQL, and the schema is initialised from scripts/sql/datavines-postgresql.sql. If you want to run with MySQL, you switch to the mysql Spring profile and initialise with scripts/sql/datavines-mysql.sql. The readme also notes that the release package now contains both scripts in the scripts directory of the installation, which is a small but useful change for people deploying from a tarball rather than from source.

So there are two supported metadata stores and the choice is a profile switch plus a different schema file, not a code change.

After that the quick start and development setup are documented on the project website rather than in the readme, with separate pages for the user guide quick start and for environment preparation in development. The readme covers the build, the database and the feature set, and then links out for the operational steps, which is the point where you will need the documentation site.

High availability claims, and what decentralised actually means here

The deployment section makes several claims that deserve a careful reading rather than a nod.

There is support for horizontal expansion with automatic fault tolerance, and job automatic fault tolerance, to ensure that jobs are not lost or repeated. The server node supports horizontal expansion to improve performance, and the design is described as decentralised.

Decentralised here means the server tier is a set of interchangeable nodes rather than a single host, with the registry as the coordination point. That is the design the plug-in registry choices point to, and choosing ZooKeeper rather than a database-backed registry is the switch that turns that from a single database with a stateless frontend into genuinely distributed coordination. If you run one server node against PostgreSQL, you get the same feature set with a much simpler mental model, at the cost of the scaling story.

The error data storage constraint interacts with this. Failed rows can be kept in MySQL or local files, but only through the Local engine. On a horizontally scaled deployment that means error data is tied to whichever node ran the job, which is consistent with the automatic fault tolerance goal of not repeating jobs, and inconsistent with a shared error store. The readme does not say how to reconcile error data across nodes, so if you plan to run several servers, that is a question to answer before you rely on the failure output.

The web interface covers configuring check jobs, running jobs, viewing job execution logs, viewing error data and viewing check results. There is also an online generation path for job running scripts, submitted through datavines-submit.sh, which can be used in conjunction with the scheduling system, so Datavines can be driven by an external scheduler rather than only by its own.

Where Datavines is the wrong tool

Several situations are outside what this platform is built for.

The first is a single table with a single check. The minimum viable deployment is a service, a metadata database and a set of jobs. If your requirement is a nightly assertion that one column is not null, that is one query in your existing scheduler, and the operational weight of a platform is not repaid. The readme's own sizing note points the same way: for small data volumes or functional verification, the Local engine is enough, which is an admission that the full platform is not the minimum.

The second is the Spark engine's version constraint. It only supports Spark 2.4. If your estate runs a modern Spark, the Local engine based on JDBC is the only path, and that pushes work onto the databases themselves. For heavy scans over large tables, that is a meaningful architectural constraint rather than a detail.

The third is the alert channel list. Only Email is supported, though the feature description mentions SLA support for check result alert, so a platform whose only delivery channel is email will not slot into a team that runs its incident response on chat. You would route the email onward yourself.

The fourth is data scope. The supported data sources are MySQL, Impala, StarRocks, Doris, Presto, Trino, ClickHouse and PostgreSQL. There is no generic JDBC plug-in mentioned as a way to reach anything else, so an engine you run that is not on that list needs a connector written for it, against the spi interfaces.

And the fifth is the state of the project itself. The repository is not archived, but the last push was on 2026-07-27, the default branch is dev rather than main, and there are no GitHub releases, so the version you would deploy is whatever the build produced rather than a published tag. For a platform that will hold your metadata catalog and your check history, that is worth weighing before you adopt it as the system of record.

Against a warehouse-native approach, and against building the checks yourself

Two comparisons make the trade clearer.

The first is against doing the checks in the warehouse. A data platform that already has constraints, assertions or scheduled transformation tests can do much of this without a separate service. The difference in approach is where the metadata and the history live. A native approach keeps everything inside the engine, so there is no cross-system catalog to keep in sync, and there is no separate availability problem. Datavines moves that concern out: it polls metadata from many engines, keeps a unified catalog, applies a rule set that is the same regardless of engine, and stores the history in its own metadata database.

That buys you a single place to see checks across MySQL, Doris, Trino and Spark at once, which is exactly the multi-engine case. It costs you a service to run, a plugin contract to learn, and a second place where the truth about your data lives.

The second comparison is against writing the checks yourself. The built-in rules are unremarkable individually, a null check and a non-null check and an enumeration check. The value is not the rule content, it is the surrounding machinery: scheduling, SLA on results, alerting, error data capture, a web UI to inspect failures, and profiling that identifies column types so that the right profile indicators are applied automatically. Writing 27 rule queries is a day of work; building the scheduler, the failure store and the alerting around them is not.

So the honest summary is that Datavines is a framework for running data checks across engines, not a source of clever rules. If your estate is one engine and a handful of tables, the framework costs more than the checks. If it is a dozen engines and a scheduled pipeline that keeps breaking silently, the framework is the cheaper path.

Editorial conclusion

Adopt Datavines if you run a heterogeneous data estate and want one place to schedule quality rules, profile columns and track metadata changes, especially since the plug-in design lets you add a source or an engine without touching the core. Skip it if you only have one small MySQL table, since a single quality check is less work in a query. Verify first which execution engine fits your data volume, since the Spark engine only supports Spark 2.4 and error data storage works only with the Local engine, and confirm the metadata database choice, PostgreSQL by default or the mysql Spring profile.

Frequently asked questions

Official sources

1. datavane/datavines on GitHub
2. Issues
3. License: Apache-2.0
4. Project website
5. README