Zingg: entity resolution that blocks before it compares

The quadratic comparison problem is the reason the tool exists

Zingg starts from a data problem rather than a feature list. Real world data holds several records for the same customer, in one system or in several, with variations across fields, and combining them gets harder as the volume grows. The consequences are named in the README in the language of a data team: lifetime value, loyalty programmes and marketing channels are impossible to establish when the base data is not linked, no segmentation algorithm can produce the right answer while duplicate copies of one customer sit in the data, and a warehouse cannot deliver on its promise if the dimension tables have duplicates in them.

The technical heart of the project is that comparing every record with every other record does not scale, because the number of comparisons grows quadratically with the number of input records. Zingg answers this by learning a clustering, or blocking, model that indexes near similar records, so the tool does not compare every record with every other one. The README states the outcome as a range: typical Zingg comparisons are 0.05 to 1 percent of the possible problem space.

That is a saving of two to three orders of magnitude, and it is the difference between a job that finishes and a job that does not. It is also the reason the tool is shaped the way it is. The blocking model is learned from your data rather than configured by hand, so the project spends most of its effort on training, not on connectors.

Two models: one decides what to compare, the other decides if it matches

The similarity model is the second half of the machinery. It is a classifier that predicts similarity between records that are not exactly the same but could belong together, and it runs only on records that fall inside the same block or cluster. The two models split the work cleanly: the blocking model decides which pairs are even worth examining, the similarity model decides whether each surviving pair is a match.

The project says this split is deliberate, and names the two approaches it is trying to get past: the Fellegi Sunter model and entity centric matching, both of which the Zingg team has written about separately. The claim is that the machine learning here is purpose built for entity resolution rather than a general model applied to it, which is a claim you would have to test on your own data, but it does explain the architecture.

Both models need training data, and this is where a user of the tool spends real effort. Zingg ships an interactive learner that asks you to mark candidate pairs, with the choices being yes, no and cannot say, and the project describes it as active learning that reaches high accuracy from frugally small training samples. In practice this is a labelling round: the tool proposes pairs, a person with domain knowledge judges them, and the model is rebuilt from those judgements.

The prebuilt models that ship with the project exist for the same reason. They let you evaluate the pipeline on a known dataset before you have invested in your own labels, which is a sensible way to find out whether the tool fits your problem at all.

The Docker path is three commands and a prebuilt model

The README calls Docker and the prebuilt models the easiest way to get started, and the sequence is short enough to type straight through:

docker pull zingg/zingg
docker run -it zingg/zingg bash
./scripts/zingg.sh --phase match --conf examples/febrl/config.json

The third command is the one worth reading twice. The script takes a phase and a configuration file, and the phase in the example is match, which is the step that pairs records rather than the step that trains a model. The configuration points at examples/febrl/config.json, so the shipped example uses the FEBRL sample data, one of several example directories in the repository alongside databricks, fabric, amazon-google, iTunes-amazon, beerAdvo-rateBeer, febrl120k, febrl5M and ncVoters5M.

A configuration file rather than a pile of flags is the shape to expect. Zingg is a pipeline tool, and the pipeline is described in configuration: which sources to read, which fields to compare, which model to use, where to write the result. The step by step guide in the project documentation is the place to learn that structure, and the README does not attempt to summarise it.

The example data is worth a second look before you run anything, because it tells you what the tool is being tuned for. The febrl and ncVoters directories are the two classic record linkage benchmarks, and the presence of febrl120k and ncVoters5M variants tells you the same data is offered at a hundred and twenty thousand records and at five million.

The image pins Spark 3.5.5 and checksums its own download

The Dockerfile shows how the release artefact is assembled, and it is stricter than most. The base image is apache/spark:3.5.5-python3 with a sha256 digest pinned to the FROM line, and the runtime environment is then set to a local Spark master and a Zingg home directory matching the release version:

FROM apache/spark:3.5.5-python3@sha256:39321d67b23e2e0953f81b60778f74bf40c40a18dfb0e881e6a38593af60afa1
USER 0
RUN apt-get update && \
    apt install -y curl vim 
ENV SPARK_MASTER local[*]
ENV ZINGG_HOME /zingg-0.7.0
ENV PATH $ZINGG_HOME/scripts:$PATH
ENV LANG C.UTF-8
WORKDIR /
USER root
WORKDIR /zingg-0.7.0

The Zingg distribution itself is fetched from the release page of the repository and verified before it is unpacked:

RUN curl --fail --show-error --silent --location \
    --output /tmp/zingg-0.7.0-spark-3.5.5.tar.gz \
    https://github.com/zinggAI/zingg/releases/download/v0.7.0/zingg-0.7.0-spark-3.5.5.tar.gz && \
    echo '4765f43e84f81728078037c65a12d119d5703366328ccabbd2357d8fecf0c5c3  /tmp/zingg-0.7.0-spark-3.5.5.tar.gz' | sha256sum --check --strict && \
    tar --extract --gzip --file /tmp/zingg-0.7.0-spark-3.5.5.tar.gz --strip=1 && \
    rm --force /tmp/zingg-0.7.0-spark-3.5.5.tar.gz

Two things follow from this. The image is tied to one Spark version, 3.5.5, and to one Zingg version, 0.7.0, so a cluster on a different Spark release is a different exercise. And the packaging installs the Python requirements from python/requirements.txt and then installs the zingg package from the Python index, which is why there is a python/ directory at the repository root, before it hands ownership of the models directory to the spark user and drops to that user for the running container.

The repository is a mixed project: assembly/, common/, config/, models/, scala/, spark/ and test/ directories next to a pom.xml, a log4j2.properties, a spark-defaults.conf and a spark.env. The Scala and Java sources under scala/ and the Spark configuration files are what make it a Spark application rather than a standalone matcher.

Connectors and file formats decide what you can feed it

Zingg reads and writes across on premise and cloud sources, and the README lists Snowflake, Cassandra, S3, Azure, Elastic, major relational databases and anything Spark itself supports, plus the file formats Parquet, Avro, JSON, XLSX, CSV and TSV. It runs on any private or cloud based Spark service, so the deployment choice is your own cluster rather than a hosted product.

Those sources and sinks are reached through an abstraction the project calls the pipe, and the documentation for it lives in the repository under docs/dataSourcesAndSinks/pipes.md. A pipe is the unit of configuration that binds a source, a transformation and a sink together, and it is what makes a Zingg job look like a data pipeline job rather than a script that reads a table and writes a table.

The domain is not limited to customers. The README lists customers, patients, suppliers and products as entities the tool can handle, and it points at a patient deduplication document and a households document, which suggests that the same engine is reused for different subject domains with different configuration. It also names concrete scenarios: unified views of customers and suppliers across systems for master data management, large scale entity resolution for AML, KYC and other fraud and compliance work, deduplication and data quality, identity resolution, integrating silos during mergers and acquisitions, enrichment from external sources, and establishing customer households.

On language, the project claims out of the box support for English as well as Chinese, Thai, Japanese, Hindi and other languages, and it advertises the ability to define domain specific functions to improve matching, which is the escape hatch for the case where a field needs a comparison rule that the defaults do not cover.

AGPL-3.0 is the decision that is not technical

The licence of the repository is AGPL-3.0, and for a data tool that teams often deploy inside a network, that is the constraint to read before the architecture. The obligation that the AGPL is known for is source sharing when a modified version is offered to others over a network, so a vendor who modifies Zingg and runs it for its own customers is in a different legal position from one who uses it unmodified inside a closed internal platform. That is a question for a lawyer and for your own compliance team, and it is worth asking before the work rather than after it ships.

The rest of the project is structured for a company rather than a loose community. Zingg is developed by the Zingg.AI team, and the README points to their consulting page for custom help around the product, to a Slack workspace for the community, and to a video demo. The documentation lives on its own site under docs.zingg.ai, and the repository also holds a gitbook configuration, a readthedocs configuration, a CNAME file and a favicon, which is what a documentation site built from the repository looks like.

The licence in practice is the same for all users, and the consulting offer is the same for all users. What changes with the AGPL is who may modify the code and distribute the result, which is why the licence question and the modification question belong in the same conversation.

Where Zingg is the wrong tool

The first case is a single table with a handful of duplicates. Zingg runs on Spark, the Docker image starts a local Spark master, and the pipeline expects sources, models and sinks. Solving a small deduplication with a handful of rules does not need a cluster, and starting one is a poor trade.

The second case is anything close to real time. The unit of work in Zingg is a batch job over a dataset, and the output is a mapping of records into clusters. A request path that must resolve an identity while a customer waits is a different problem with a different architecture, and a Spark job is not it.

The third case is data you cannot get labels for. The accuracy of the result is bounded by the training pairs, and the interactive learner is designed to be used by someone who knows the domain. A team with no subject matter expert available to mark candidates will not get a good model, and the prebuilt models only help if your data resembles the sample sets they were trained on.

The fourth is a Spark version mismatch. The published artefacts are built against Spark 3.5.5, and the repository ships performance test workflows named for the hundred and twenty thousand and five million record datasets, which says the tool is measured at that scale and not at the scale of an interactive session.

Editorial conclusion

Zingg fits a data team that has to build one trusted customer or supplier view across several systems and can afford to run a Spark job and to label training pairs, because the blocking model is what makes the comparison count manageable at millions of records. It is the wrong choice for a single table, for a near real time lookup, or for a vendor that will not accept AGPL-3.0 terms, and the prebuilt model path only helps if your data resembles one of the example sets. If you try it, check two things first: that the Spark version in your cluster matches the 3.5.5 base the Docker image is built on, and that you have budgeted for the labelling round, since the accuracy of the result is bounded by the training pairs you supply.

Frequently asked questions

Official sources

1. Issues
2. License: AGPL-3.0
3. README
4. Releases
5. zinggAI/zingg on GitHub