SemHash: semantic deduplication, outlier filtering and representative sampling

Three jobs, one idea: judge by meaning

SemHash is a lightweight, multimodal library for semantic deduplication, outlier filtering and representative sample selection, and the word that matters is semantic. Cleaning a dataset by exact match is a solved problem with a hash function. Cleaning it by meaning is not, because a paragraph that has been lightly reworded, a question rephrased, or a template filled in with different values is the same example wearing different clothes, and near-duplicates like that are exactly what inflates a training set and skews an evaluation. The library takes a set of records, embeds them, and works on the geometry of those embeddings. The three operations are natural siblings. Deduplication finds the near-duplicates and keeps one copy. Outlier filtering finds the records that do not look like the rest of the corpus. Representative sampling picks a subset that covers the variety of the whole, which is what you want when a dataset is too large to train on and you would rather curate than sample at random. The package is on PyPI, MIT licensed, and it is still marked as beta in its classifiers, with a development status of 4 and Python 3.10 to 3.12 supported.

Model2Vec for text, your own encoder for everything else

The text path works out of the box with fast Model2Vec embeddings, which is the reason this library is fast rather than merely correct. Model2Vec embeddings are cheap to compute, so embedding a large corpus does not cost what a sentence-transformer pipeline would. Other modalities, images, audio and whatever else, are supported with custom encoders, which means you pass a model object into the constructor. The image example in the readme uses a CLIP vision model from sentence-transformers:

from sentence_transformers import SentenceTransformer
from semhash import SemHash

model = SentenceTransformer('clip-ViT-B-32')
semhash = SemHash.from_records(list(dataset), columns=["img"], model=model)

The columns argument is what tells the library which field of a record to embed, so the same code handles a list of plain strings and a table of rows with several fields. The cost of that flexibility is obvious: for text you get the fast path for free, and for images you get the speed of whatever encoder you supply. The dependency list in the project file is short, model2vec, vicinity with the usearch extra, frozendict and pyversity, and everything else in the stack is what you bring.

Self operations and cross-dataset operations, and the leakage case

The library supports both single-dataset operations, cleaning a training set, and cross-dataset operations, deduplicating test against train, and the second is the one people underestimate. If a test set contains near-duplicates of training examples, your benchmark number is measuring memorisation, and no amount of shuffling or stratification fixes it. The cross-dataset API mirrors the self API with the records argument added:

semhash = SemHash.from_records(records=train_texts)
deduplicated_test_texts = semhash.deduplicate(records=test_texts, threshold=0.9).selected

The same pattern covers filtering outliers from the test set against the training set and finding representative samples there. The self variants are named for what they act on, so self_deduplicate acts on the corpus the instance was built from, while deduplicate acts on a corpus you pass in. The naming is a little confusing for an hour and then obvious forever, which is the usual trade for an API that covers both cases with the same three methods. Multi-column datasets are handled by naming the columns to embed, so a question and answer dataset can be deduplicated on the question and context fields together, as the squad example does with columns of question and context.

The three parameters you will actually tune

Three numbers control the behaviour, and each one corresponds to one of the operations. The threshold is the similarity above which two records are considered duplicates, and the cross-dataset example uses 0.9, which is conservative, since a high threshold only catches records that are very close. The outlier percentage is how much of the corpus to treat as outlier, with the example using 0.1, and the readme notes that this can be a specific percentage rather than an absolute count, which is what you want when the corpus size changes between runs. The selection size is how many representative samples you want, with the example using 10. Each of these is a judgement call, and none of them is a default you should accept without looking, because the right value depends on your corpus. The library gives you the tools to look: the result object keeps the records that were removed alongside the ones that caused the removal, so you can read a sample of the decisions instead of trusting a number.

The result objects are the real product

The readme spends more space on inspection than on cleaning, and that is the design decision that makes the library usable. The deduplicate and self_deduplicate functions return a DeduplicationResult, which stores the deduplicated corpus, a set of duplicate objects along with the objects that caused the duplication, and several functions for further inspection. The filter_outliers, self_filter_outliers, find_representative and self_find_representative functions return a FilterResult, which stores the found outliers and representative samples. On either object, the selected attribute gives the records that were kept, and the filtered attribute gives the ones that were dropped, so viewing the outliers is semhash.self_filter_outliers().filtered. The statistics are on the result too, with duplicate_ratio and exact_duplicate_ratio, and the second of those is the number you compare against the first to see how much of the duplication your embedding found that an exact hash would have missed. The sharpest tool is get_least_similar_from_duplicates, which returns the least similar items among the duplicates it found, and that is the fastest way to see whether a threshold is too loose, because if the least similar duplicate pair looks like a different example then your threshold is wrong.

Vicinity underneath, and the plumbing around it

All the operations use Vicinity for efficient similarity search, which is the library that makes the approach scale to more than a toy dataset. That is the one architectural decision the whole package rests on, since deduplication by embedding is a nearest neighbour problem and a brute force comparison of every pair is quadratic in the corpus. The project also handles a detail that is easy to miss: it works with simple lists of strings and with complex multi-column datasets, and the quickstart examples assume you have the datasets library installed, which is a separate pip install. The data models live in a single module, datamodels.py, which holds the DeduplicationResult and FilterResult classes the readme links to, so the shape of what you get back is documented in one place. The result objects are ordinary Python objects with attributes, not a special framework, which means you can drop them into a pandas workflow and carry on with your own analysis.

A young package with strict habits

The project is young, with a beta classifier and a release history of v0.4.1 in January 2026, v0.4.2 in September and v0.5.0 a few days later, and the last push was on 2026-10-04, so it is moving quickly. What is interesting is how disciplined the tooling is. The project uses uv for the environment, with a Makefile that creates a virtual environment, syncs all extras and installs the pre-commit hooks, a fix target that runs pre-commit over all files, and a test target that runs pytest with coverage measured against the semhash package itself. Ruff does the linting with a line length of 120 and Python 3.10 as the target, and the selected rules are the strict ones: annotations are enforced, cyclomatic complexity is capped, docstrings are required through the pydocstyle rules, import order is checked, and the numpy style convention is applied. There is a benchmarks directory with its own text and image runners wired into make targets, which is unusual for a package this size and means the fast claims can be checked. The tree also carries a CITATION.cff, a codecov badge and a Discord for the project, and the authors are Thomas van Dongen and Stephane Tulkens.

Editorial conclusion

SemHash is worth trying if your training data has near-duplicate or boilerplate contamination, because exact-match dedup will not touch a paragraph that has been lightly reworded, and because the inspection tools let you see what a threshold actually removed before you commit to it. It is the wrong choice if your duplicates are exact, since a hash is cheaper, or if you need the dedup to run on a dataset far larger than fits in memory, since the approach embeds everything first. Before you trust a threshold, look at the least similar items the deduplication called duplicates, because that is where a wrong threshold shows up first.

Frequently asked questions

Official sources

1. License: MIT
2. MinishLab/semhash on GitHub
3. Project website
4. README
5. Releases