fastdup: local duplicate, outlier and label-error detection for image and video datasets
fastdup is a powerful, free tool designed to rapidly generate valuable insights from image and video datasets. It helps enhance the quality of both images and labels, while significantly reducing data operation costs, all with unmatched scalability.
At a glance
- What is it?
- fastdup is a Python package that scans a folder of images or videos and reports duplicates, near-duplicates, outliers and suspect labels. The mechanism is a C++ similarity engine driven from Python, and the licence is the first thing to check before you build a pipeline on it.
- Who is it for?
- Adopt fastdup if you already have a folder of images or videos on a Linux or macOS machine and you want a local, scriptable pass over duplicates, outliers and blur before training. Do not adopt it for a commercial product without resolving the licence question first, because the README badge points at CC BY-NC-ND 4.0 while the repository metadata reports NOASSERTION, and those two do not say the same thing.
- Can I use it commercially?
- Check first. The repository uses a licence we do not classify automatically, so read its LICENSE file before any commercial use.
- Is it still maintained?
- Yes. The repository last received commits 23 days ago.
- What is it written in?
- Mainly Python, according to GitHub's language statistics.
Answers come from the project's GitHub data, last synced on September 15, 2026, and from our analysis. They are not legal advice.
DEEP OPEN-SOURCE ANALYSIS
The dataset problem fastdup is aimed at
Most image projects start with a folder that nobody has audited. It contains duplicates from repeated scrapes, near-identical frames from video, images that failed to download properly, and labels that point at the wrong class. Training on that folder wastes compute and quietly caps accuracy, and the cost grows with the size of the set because manual review does not scale past a few thousand files. fastdup targets exactly this step: the audit that happens before training. The README describes it as analyzing image and video datasets and as handling labeled or unlabeled data, which matters because the duplicate and outlier work does not require labels at all. The intended user is a machine learning engineer or data curator who has a directory of images and wants a ranked list of what is wrong with it. The project's own framing puts quality, scale, speed and privacy first, and the privacy point is structural: the README states it runs locally or on your cloud infrastructure, so the images do not leave the machine. The stated scale target is 400M images on a single CPU machine, with a claim of scaling to billions. Those numbers come from the README, not from any run I performed.
What the Python API actually does
The entry point is a class instance built from a directory path, followed by a run call. The README gives this example: fastdup.create(input_dir="IMAGE_FOLDER/") assigned to a variable, then fd.run(). That is the whole setup. The object then exposes a vis namespace with separate gallery methods for duplicates, outliers, connected components, image statistics such as blur and brightness, and similar images. Each is a single call with no arguments in the README example, and each produces a static gallery for inspection. The connected-component gallery is the interesting one architecturally: grouping images into components implies a graph built over similarity edges, so a chain of near-duplicates that are not directly identical to each other can still land in one group. The similarity threshold is a parameter rather than a fixed constant, which the README makes explicit in the removal path by naming distance as the control. The heavy work sits in a C++ engine according to the README, with Python as the interface. That split explains the CPU-only positioning: no GPU is mentioned anywhere in the material, and the performance claim is tied to optimized C++ on low-resource CPU machines.
Installing and running the first scan
Installation is one command from PyPI: pip install fastdup. The README badge lists Python 3.9, 3.10, 3.11 and 3.12, and the supported operating systems badge reads macOS, Linux and Windows via WSL2. WSL2 is a real constraint rather than a footnote: native Windows is not in that list, so a Windows user is running a Linux environment underneath. The README also links to a separate installation page for more options, which is where platform-specific builds would live. After install, the minimal script is the two lines above. There is no config file, no environment variable and no service to start in the material provided. The one destructive call to know about is fastdup.remove_duplicates("IMAGE_FOLDER/"), which the README says finds and deletes duplicate images directly from disk using a similarity above 0.96 by default. Two safety parameters are documented: dry_run=True to preview which files would be removed, and distance to change the threshold. The default of 0.96 sits close to identical, so lowering distance is how you widen the net to near-duplicates, and that is also how you start deleting images that are merely similar rather than redundant.
The licence is the adoption blocker, not the code
The repository metadata reports the licence as NOASSERTION, which is what GitHub shows when it cannot map the licence file to a known identifier. The README badge says something more specific: CC BY-NC-ND 4.0. Those two signals disagree in practice, and the badge points at a licence with a non-commercial restriction and a no-derivatives restriction. For an engineer evaluating this for a commercial training pipeline, that is the first question to settle, ahead of any benchmark. I am not giving legal advice here, and the README badge is not the licence file. What I can say from the material is that the two sources shown do not match, and that the named licence is not a permissive software licence. Anyone planning to ship fastdup inside a product, or to fork and modify it, should read the LICENSE file in the repository and get an answer from whoever handles licensing at their organisation before writing it into a pipeline. The release history adds a second consideration: the most recent releases listed are v2.2_3.8 and v2.2_3.7 from June 2024, both labelled Centos 7.0.9 stable, and v1.119 from April 2024 labelled Ubuntu 18. Those platform labels are old distributions, and the last push to the default branch is dated August 2026, so the release tags and the repository activity are not telling the same story. Check which wheel your platform actually resolves to before assuming the newest tag applies to you.
Where fastdup is the wrong tool
The removal helper is the sharpest edge in the project. It deletes files from disk. Similarity above 0.96 by default is a narrow definition of duplicate, but any dataset with legitimately near-identical images, such as consecutive video frames, burst photography or augmentation output, will present candidates the tool cannot distinguish from true redundancy. The dry_run parameter exists for exactly this reason and should be treated as mandatory on a first pass. Beyond deletion, the tool answers a narrow question. It reports duplicates, near-duplicates, outliers, mislabels, broken images and low-quality images. It does not train a model, does not score a model against a validation set, and does not tell you whether removing a cluster will help your particular task. A dataset with no duplicates and clean labels gets nothing from the run beyond confirmation. There is also a hardware boundary: the README positions the engine for CPU, and nothing in the material describes GPU acceleration, so a team expecting to borrow training GPUs for this step should not assume it. Finally, the gallery outputs are static visualisations. They are for a human to look at, not for a pipeline to consume automatically, and the material does not describe a programmatic report format beyond the galleries.
How this differs from embedding-based deduplication scripts
The common alternative is a hand-rolled pipeline: run every image through a pretrained embedding model with a framework such as PyTorch or TensorFlow, store the vectors, then do a nearest-neighbour search with FAISS or a similar index and threshold the distances yourself. That approach is flexible. You choose the embedding model, you can fine-tune it on your domain, and you can wire the output into whatever training loop you already have. The cost is that you own the whole stack: batching, memory management over millions of vectors, the index build, and the threshold tuning. fastdup makes the opposite trade. It ships the embedding and the search as one C++ engine behind a Python object, which is why the API is two lines and why the scale claim is about a single CPU machine rather than a GPU cluster. You give up control of the embedding and the internals in exchange for not building or maintaining that pipeline. The choice comes down to whether your similarity notion is standard or domain-specific. If off-the-shelf visual similarity is what you need, the two-line path is hard to argue with. If your notion of duplicate depends on a fine-tuned model, a custom pipeline is the honest answer, and fastdup is not trying to be that.
Maintenance and upgrade cost
The upgrade surface is small. The package installs from PyPI, the API shown in the README is a handful of methods, and there is no server, database or config schema to migrate. The friction sits in two places instead. First, Python version support is bounded at 3.9 through 3.12 by the README badge, so an environment on 3.13 or on an older interpreter is outside the stated range. Second, the release tags in the material are tied to specific distributions, Centos 7.0.9 and Ubuntu 18, both of which are past their mainstream support windows. Pinning a fastdup version that matches your platform is likely to be part of the setup, and that pin is what you carry forward. The repository is not archived and the last push is recent, but the release list shown stops in June 2024, so the gap between repository activity and tagged releases is worth understanding before you assume a fix has shipped. Because the licence badge names a no-derivatives licence, the option of patching the engine yourself and redistributing the result is not something to assume either. Budget for a pinned version and a licence review, not for a fork.
Editorial conclusion
Adopt fastdup if you already have a folder of images or videos on a Linux or macOS machine and you want a local, scriptable pass over duplicates, outliers and blur before training. Do not adopt it for a commercial product without resolving the licence question first, because the README badge points at CC BY-NC-ND 4.0 while the repository metadata reports NOASSERTION, and those two do not say the same thing. Verify the licence file, then run fastdup.remove_duplicates with dry_run=True on a copy of the dataset and read the preview list before anything is deleted.
Community notes