bees, a btrfs deduplication agent that works on live data

Block-oriented dedupe aimed at large btrfs filesystems

bees stands for Best-Effort Extent-Same, and it is a btrfs deduplication agent written in C++. The design goal is scale. It is described as a block-oriented userspace deduplication agent designed to scale up to large btrfs filesystems, and it combines an offline dedupe pass with an incremental data scan capability, so the time data spends on disk between being written and being deduplicated is minimised.

That two-mode description is the heart of the tool. An offline dedupe walks the filesystem looking for equal extents. The incremental scan exists because waiting for a maintenance window is a long time for new data to sit un-deduplicated, so bees can pick up what has appeared since the last pass as it appears. The daemon mode is the daemonised form of that incremental behaviour.

The scope is btrfs only, and that is not a limitation of implementation effort but a design dependency. bees works on extents, the unit btrfs stores data in, and the relationship between extents, snapshots and reflinks is what lets it look at data only once. That is why the README claims a whole-filesystem dedupe that scans data only once even with snapshots and reflinks, and why it says bees recovers more free space from btrfs than naive dedupers. A tool that operated at the file level would find nothing useful in a filesystem where identical content is already stored once and shared.

The strengths and weaknesses the project lists itself

The README states strengths and weaknesses separately, which makes it easy to see the trade-off being made. On the strength side: a space-efficient hash table that can use as little as 1 GB of hash table per 10 TB of unique data, which works out to about 0.1 GB per TB. The hash table size is constant, so RAM use does not grow when the data set grows. A persistent hash table allows a rapid restart after shutdown. The largest extents are processed first, so more free space is recovered inside a fixed maintenance window. bees works with btrfs compression and can dedupe any combination of compressed and uncompressed files. It works on live data with no scheduled downtime, and it throttles itself automatically to reduce system load.

The weakness list is equally concrete. Whole-filesystem dedupe means there are no include or exclude filters, and it does not accept file lists. It requires root privilege, with CAP_SYS_ADMIN plus the usual filesystem read and modify capabilities. The first run may increase metadata space usage if many snapshots exist, which is documented as a gotcha. The constant hash table size also means RAM use does not go down if the data set becomes smaller. And it is btrfs only.

Those two lists together define the buyer. If you need to dedupe a specific set of directories, bees is the wrong shape of tool, and the absence of filters is not going to change.

The Makefile build and what lands on the filesystem

The build is a plain Make setup with overridable install locations. The defaults are:

PREFIX ?= /usr
ETC_PREFIX ?= /etc
LIBDIR ?= lib
BINDIR ?= sbin

A localconf file can be included to override those values, and the top level of the repository carries the standard pieces of a C++ project, with Defines.mk, include/, lib/, src/, scripts/, test/, docs/ and bin/ directories. The targets are separated by concern: lib builds the libraries, src builds the binaries and depends on lib, test runs the test suite, doc builds the documentation, and scripts turns the templates in scripts/ into the beesd daemon and the [email protected] unit.

The install step is where the layout becomes visible. The agent binary goes into the library execution directory under the prefix, and the daemon and a sample configuration file are installed to the sbin and etc locations:

install -Dm755 bin/bees
install -Dm755 scripts/beesd

The systemd unit is installed only when pkg-config reports a systemd system unit directory, since that variable is queried during configuration. The version string embedded in the binaries comes from git describe, falling back to UNKNOWN when the version cannot be determined, which is a useful detail for a tool you deploy into a fleet. The default target builds everything including docs and tests, and the fly target is simply an alias for install.

Daemon mode, persistent hash tables and self-throttling

The daemon exists because a filesystem does not stop growing between maintenance windows. In daemon mode bees incrementally deduplicates new data as it appears, and the hash table that records what has been seen is persisted, so a restart after a shutdown is fast rather than starting from nothing. The documentation splits these concerns across several pages: docs/running.md for running the tool, docs/config.md for configuration, docs/options.md for the command line options, and docs/how-it-works.md for the mechanism itself.

Self-throttling is the other behaviour that makes daemon mode usable on a live system. bees reduces its own system load when it detects that the machine is busy. Combined with working on live data and no scheduled downtime requirement, that is what separates this design from a batch deduper you run at night. The trade-off is that the tool needs the privileges and the whole-filesystem reach described earlier, and the operational documentation is part of the deal rather than an optional extra.

For anyone who wants to know what the tool measures about itself, docs/event-counters.md documents the event counters, and docs/scan-one-algorithm.md explains how scan_one matches extents. Those two pages are the closest thing to a design document and the natural reading order for an engineer deciding whether the matching behaviour suits their data.

Snapshots, compression and the metadata cost of the first run

Two of the interesting claims concern how bees handles data that is already shared or already compressed. The whole-filesystem scan sees data once even with snapshots and reflinks, because a snapshot is a reference to existing extents rather than a second copy. And because btrfs compresses data per extent, dedupe has to work on any combination of compressed and uncompressed files, which is what the claim about deduping compressed and uncompressed files means.

The cost of that behaviour shows up in the first run. The weakness list warns that the first run may increase metadata space usage if many snapshots exist. That is the kind of line that is easy to skim past and expensive to discover later, because metadata growth on a filesystem with many snapshots is exactly where free space is hardest to win back.

This is why docs/gotchas.md is the first link under Recommended Reading, alongside docs/btrfs-kernel.md, which collects btrfs kernel bugs and carries a data corruption warning for old kernels, and docs/wrong.md, which covers what to do when something goes wrong. docs/btrfs-other.md compares bees against other btrfs features, which is the document to read if your plan was to solve the problem with built-in btrfs functionality instead of a separate agent.

A userspace agent against the filesystem's own features

The alternative to bees is not another dedupe tool, it is doing nothing and relying on btrfs's own capabilities, which is exactly the comparison docs/btrfs-other.md is written to support. That difference in approach is worth spelling out, because both routes live in the same filesystem and solve overlapping problems.

btrfs itself deduplicates at the extent level as it writes, so new data that matches existing data can share an extent without anything external running. What it does not do is go back over data that was written before that matching data existed, and it does not walk the filesystem afterwards looking for equal extents that arrived in different orders. bees does that retrospective pass, across the whole filesystem, and processes the largest extents first so that a fixed window of downtime yields the most space.

The cost of that approach is everything the weakness list describes. bees runs as a userspace agent with root privileges, it looks at the entire filesystem rather than a subset, and it can increase metadata usage on the first pass. In exchange it can recover space that btrfs's write-time behaviour has already missed. Whether that exchange is worth it depends on how much duplicate data has accumulated, and the project is careful to describe itself as best-effort rather than promising a result.

Editorial conclusion

bees suits administrators of large btrfs filesystems who have already accepted root privilege and a whole-filesystem scope, and who want space recovered without scheduling downtime. It does not suit ext4 or XFS users, teams that need to deduplicate a chosen set of directories, or anyone who cannot read docs/gotchas.md and docs/btrfs-kernel.md before the first run. Before you deploy it, check your kernel against the btrfs kernel bug notes, confirm your hash table budget against the roughly 1 GB per 10 TB of unique data figure, and read docs/gotchas.md, because the last push to the repository was on 2026-05-03 and there are no GitHub releases to consult.

Frequently asked questions

Official sources

1. Issues
2. License: GPL-3.0
3. README
4. Zygo/bees on GitHub