# rdfind, and the art of finding duplicate files by content > rdfind is a C++ command line tool that finds duplicate files by comparing their content rather than their names, and it does so with an elimination funnel that reads as little of the disk as possible. The tool is packaged for most distributions, has an unusually thorough CI matrix, and still keeps the author reachable by email. **pauldreik/rdfind** — find duplicate files utility - Repository: https://github.com/pauldreik/rdfind - Stars: 1,319 · Forks: 95 - Language: C++ - License: NOASSERTION - Published: 2026-10-19 · Updated: 2026-10-19 · Language: en - Canonical page: https://hysenlabs.com/projects/pauldreik-rdfind ## Duplicates are decided by content, not by file name Rdfind is a command line tool that finds duplicate files, useful for compressing backup directories or just finding duplicate files, and the important property is that it compares files based on their content, not on their file names. That single decision shapes everything about how you use it. A name based duplicate finder tells you that two files are called the same thing, which is often true and occasionally a coincidence, and it cannot find the same photo saved under two different names at all. A content based tool finds both cases, at the cost of having to read the files to compare them. The tool's whole design is about making that cost manageable. The output of a typical run is a funnel, because the algorithm eliminates candidates in cheap stages before it does anything expensive. ``` Now scanning "/home/pauls/bilder", found 3301 files. Now have 3301 files in total. Removed 0 files due to nonunique device and inode. Now removing files with zero size...removed 3 files Total size is 2861229059 bytes or 3 Gib Now sorting on size:removed 3176 files due to unique sizes.122 files left. Now eliminating candidates based on first bytes:removed 8 files.114 files left. Now eliminating candidates based on last bytes:removed 12 files.102 files left. Now eliminating candidates based on md5 checksum:removed 2 files.100 files left. It seems like you have 100 files that are not unique Totally, 24 Mib can be reduced. ``` Out of 3301 files, 3176 disappear on size alone and only 100 remain to be checksummed. That is the whole design in five lines, and it is why a run over a large directory is fast. ## The algorithm, in nine steps The project description spells the algorithm out, and it is worth reading because each step is a different way of ruling a file out cheaply. The effort required is in worst case O(Nlog(N)) for N files, and because it sorts files on inodes prior to disk reading, it is quite fast, and it also only reads from disk when it is needed. First, loop over each argument on the command line and assign each argument a priority number, in increasing order. Second, for each argument, list the directory contents recursively and assign it to the file list, with a directory depth number starting at 0 for every argument. Third, if the input argument is a file, add it to the file list. Fourth, loop over the list and find the sizes of all files. Fifth, if the flag -removeidentinode is true, remove items from the list which are already added, based on the combination of inode and device number, so a group of files hardlinked to the same file is collapsed to one entry. There is a pointer to the comment on hardlinks in the caveats section, which is worth following. Sixth, sort files on size and remove files from the list which have unique sizes. Seventh, sort on device and inode, which speeds up file reading, and read a few bytes from the beginning of each file. Eighth, remove files from the list that have the same size but different first bytes, and this step can be disabled with -firstbytessize 0. Ninth, the same again with the last bytes of each file, and then the checksum comparison finishes the job. The ordering is the design. Size first, because it is free once you have the directory listing. Then the first and last bytes, because a few bytes of each file rule out the overwhelming majority of candidates. Then the full md5 checksum, which is the only step that reads whole files, applied to the small residue that is left. ## The results file, and the decision it leaves to you The output is a text file, and its format is the interesting part of the design. ``` # Automatically generated # duptype id depth size device inode priority name DUPTYPE_FIRST_OCCURENCE 960 3 4872 2056 5948858 1 /home/pauls/bilder/digitalkamera/horisontbild/.xvpics/test 001.jpg.gtmp.jpg DUPTYPE_WITHIN_SAME_TREE -960 3 4872 2056 5932098 1 /home/pauls/bilder/digitalkamera/horisontbild/.xvpics/test 001.jpg ``` The header line lists the columns, and the two dup types are what make the file usable rather than just a list. A row marked DUPTYPE_FIRST_OCCURENCE is the copy the tool considers the original, and the rows with the same identifier and a negative id are the other copies of it. In the example the author walks through, the same file appears in two directories, and the two rows are the two copies. The author then says that he can now remove the one he considers a duplicate by hand if he wants to. That sentence is the design decision. Rdfind does not delete anything by default. The tool produces a report and the deletion, when it happens, is a separate deliberate act, which is the correct default for a program pointed at a photo library or a backup directory. The DUPTYPE_WITHIN_SAME_TREE label carries a second piece of information. Files that are duplicates of each other but sit in the same directory tree are marked differently from those in different trees, which is a hint about how the duplicates arose, and it is the difference between a file you copied twice and a file that ended up in two places by accident. Without options, a results file is created in the current directory, and the man page has the full options list. ## Hardlinks, and the flag that decides what counts as a duplicate One detail in the algorithm deserves its own attention, because it is the one that changes the meaning of the output. If the flag -removeidentinode is true, items are removed from the list which are already added, based on the combination of inode and device number, and a group of files hardlinked to the same file is collapsed to one entry. The project description points at a comment on hardlinks in the caveats section for the details. The reason this is not a trivial detail is that a hardlink is not a duplicate in the sense a user usually means. Two paths pointing at one inode are one file with two names, and reporting them as duplicates would be wrong, and removing one would be harmless but pointless. Collapsing them by inode and device is the correct behaviour, and it is also the behaviour that makes the tool safe to point at a directory tree that uses hardlinks deliberately, which on Unix systems is common for deduplicated archives. The flag is off by default, so the conservative behaviour is not applied unless you ask for it. That is the right default in a tool whose output someone might act on, and it is a reminder to read the options rather than accept the first run. The file information the tool collects, which the report exposes, is the size, the device number, the inode and the priority number, and the priority comes from the command line order, so a run over several directories in a deliberate order can express which copy you want to keep. The tool is telling you what it found and leaving the policy to you. ## Installation, and building it yourself with C++17 Rdfind is packaged for the usual places, and the project is happy to be installed from a distribution rather than compiled. On Debian and Ubuntu it is a standard package. ```bash apt install rdfind ``` On Fedora it is dnf install rdfind, and on Mac there are two routes, MacPorts with a direct link to the Portfile, or brew with a direct link to the homebrew formula. On Windows it is used through cygwin. If you want to compile the source yourself, that is fine. Rdfind is written in C++17 and should compile under any Unix-like system. The repository shows both build systems. There is the autotools path, with configure.ac, Makefile.am, a bootstrap.sh script and an INSTALL file, and there is an inofficial_cmake directory, whose name is a small joke about the state of the CMake support. That naming is worth a moment, because it tells you what to expect from the build. The autotools build is the supported one, and the CMake directory is where a contributor can experiment without the project committing to it. The build requirements are therefore modest. A C++17 compiler, autotools, and nothing exotic. There is no dependency on a hashing library, since the md5 implementation is in the tree, with Checksum.cc, Checksum.hh and ChecksumTypes.hh at the top level, and EasyRandom.cc for the random number generation the first and last byte sampling uses. ## A CI matrix that treats a file utility like security code The continuous integration section is longer than the feature list, and for a small command line tool it is remarkable. There are two static analysers, codeql and cppcheck, each with an issues list and a build badge, and both are run against two branches, main and devel. There is a build and test workflow for Ubuntu 20.04, 22.04 and 24.04 with the default compiler and settings, and one that compiles on the latest Mac OS X. Then there is the quality workflow, and its description is the one to read. It builds and executes tests on Ubuntu 24.04 with multiple versions of gcc and clang. It runs builds with address and undefined sanitizers and with valgrind. And it also performs the tests with a binary compiled in 32 bit mode. The 32 bit build is the detail that shows the author's habits. A program that compares files, walks directories and parses command lines has plenty of places where a pointer width assumption or an integer size assumption can hide, and compiling and running the test suite in 32 bit mode catches the ones that a 64 bit build cannot. Running under valgrind and the sanitizers catches the memory errors that a functional test would not notice. The repository scripts mirror that discipline locally. There is do_quality_checks.sh, do_clang_format.sh, do_cmake_format.sh, do_shellcheck.sh, do_shellfmt.sh and do_yamllint.sh, a .clang-format file for the C++ style, a .yamllint configuration, a cppcheck directory, and separate testcases and unittests directories. For a tool that reads a directory tree and, in the end, unlinks files, this is exactly the amount of care you would want, and it is a strong argument for trusting the output of a run on a large tree. ## The source layout, the licence, and the release history The repository is a flat C++ project with the source files at the top level, which is what a small autotools project looks like before anyone splits it up. There is rdfind.cc as the main program, and the pieces are named for what they do: CmdlineParser.cc for the option handling, Dirlist.cc for the directory traversal, Fileinfo.cc for the per file record, Options.cc for the configuration, UndoableUnlink.cc for the deletion path, Checksum.cc for the hashing, EasyRandom.cc for the sampling and Rdutil.cc for the odds and ends. The types are split into header files with the same names, and RdfindDebug.hh holds the debug output. The UndoableUnlink name is the one to notice, because it describes a promise the tool makes about its most dangerous operation. A file removal that can be undone is a different thing from a plain unlink, and the presence of a module named for it says the deletion path is designed with recoverability in mind. The supporting files are the usual ones for a project of this vintage and shape: AUTHORS, COPYING, ChangeLog, NEWS, TODO, INSTALL, VERSION.txt, a release_new_version.txt, a man page at rdfind.1, and the .github directory. The licence is recorded as a custom one that GitHub cannot classify, and the COPYING file in the repository is the authoritative text to read. That is the normal arrangement for a project that predates the licence classifier, but it is worth checking the file rather than assuming. The releases are 1.8.0 from 2026-02-22, 1.7.0 from 2025-02-07 and 1.6.0 from 2023-06-17, and the last push was on 2026-07-25 with the repository not archived, so the tool is still being worked on. The author asks readers to email him about how they actually use it, and suggests a gift card if they want to give something back. ## Conclusion Adopt rdfind if you are cleaning up a backup directory, a photo library or a build tree and want duplicates identified by content rather than by name, since the incremental elimination means it reads only what it must. Do not adopt it as a deletion tool on its own, because the tool reports duplicates and leaves the decision to you, which is the right default for data you cannot recreate. Before you run it over anything large, read the caveats about hardlinks and check the -removeidentinode flag against your filesystem, and remember that the tool does not delete anything until you tell it to, so the first run should be on a copy. ## FAQ ### How does rdfind decide that two files are duplicates? By content rather than by file name. It sorts candidates by size, eliminates those with unique sizes, compares the first and last bytes of what remains, and then compares md5 checksums of the residue, so it only reads whole files for the few candidates that survive. ### Does rdfind delete the duplicate files? By default it does not. Without options it writes a results file in the current directory, and the tool's author describes removing a duplicate by hand after reading the results. ### What do the DUPTYPE values in the results file mean? DUPTYPE_FIRST_OCCURENCE marks the copy the tool treats as the original, and DUPTYPE_WITHIN_SAME_TREE marks another copy of the same file, with the identifier shared between them and the priority coming from the order of the command line arguments. ### What does the -removeidentinode flag do? When it is true, files that are hardlinked to the same file are collapsed into one entry, based on the combination of inode and device number, so that a hardlinked pair is not reported as a duplicate. ### How do I install rdfind? From a distribution package with apt install rdfind on Debian and Ubuntu, dnf install rdfind on Fedora, through MacPorts or brew on Mac, or through cygwin on Windows. Building from source needs a C++17 compiler, and the project says it should compile under any Unix-like system. ## Sources - [Issues](https://github.com/pauldreik/rdfind/issues) - [pauldreik/rdfind on GitHub](https://github.com/pauldreik/rdfind) - [README](https://github.com/pauldreik/rdfind/blob/main/README.md) - [Releases](https://github.com/pauldreik/rdfind/releases) --- Hysen Labs editorial analysis, written from the project's own repository and release notes. Cite the canonical page: https://hysenlabs.com/projects/pauldreik-rdfind