axiomhq/hyperloglog: count-distinct sketches with LogLog-Beta bias correction and a sparse mode

The repository description still advertises TailCut, the code no longer does

The repository description on GitHub reads HyperLogLog with lots of sugar (Sparse, LogLog-Beta bias correction and TailCut space reduction), and that third item is the interesting one, because the README says the current implementation has evolved away from it. The implementation history note states that the initial version, tagged v0.1.0, was based on the paper Better with fewer bits by Qingjun Xiao, You Zhou and Shigang Chen, and that the current implementation has moved away from the tailcut method. The current implementation is instead based on LogLog-Beta, as described by Jason Qin, Denys Kim and Yumei Tung in 2016. A reader who reads only the one line description and then the release notes is going to look for a tailcut encoder that is not there. This is the kind of drift that makes a project worth reading the README of rather than the summary of.

How the sketch works: registers, a hash function and a bias correction

The library approximates the number of distinct elements in a multiset, the count-distinct problem, with a HyperLogLog style structure. The current design has seven stated features. A Metro hash replaces xxhash, so the hash is the Go Metro hash from dgryski/go-metro rather than the more common xxhash choice. Registers are 8-bit, which the README presents as a matter of convenience and simplified implementation rather than of maximum density. Bias correction uses LogLog-Beta, described as dynamic across all cardinalities. Insertions and merging are order-independent, so the result does not depend on the order data arrives in. A sparse representation handles lower cardinalities, in the manner of HyperLogLog++, and the tailcut method has been removed in favour of a more straightforward approach. The module dependencies match that description: dgryski/go-metro for the hash, kamstrup/intmap for the sparse map, and stretchr/testify for the tests.

Precision is a dial: 16 bytes to 256 KB depending on the register count

The precision choice is the main decision a user makes, and the README gives concrete numbers for it. Sketches can be built with any register count between 2^4 and 2^18, and memory usage scales directly with it: the minimum of 2^4 registers costs 16 bytes, the default of 2^14 registers costs 16 KB, and the maximum of 2^18 registers costs 256 KB. The stated trade-off is the familiar one, memory against estimation accuracy, and the design intent is that you pick a point on that curve for your use case. The default of 2^14 is the balanced choice, and it is worth understanding as a number rather than as a word. The precision is a parameter of the sketch, not a global setting, so a single program can hold sketches of different precisions side by side.

Getting the code and running the checks the CI runs

The README does not give an install command, so the module path in go.mod is the thing to note: the package is github.com/axiomhq/hyperloglog, the module requires go 1.23 with a toolchain of go1.23.4, and it is a normal Go module you pull into a project the way the toolchain expects. What the README does document is the local check workflow, which matters if you plan to send a pull request. The contributing guide explains how to propose bug fixes and improvements, and the instruction is to run the same checks that CI runs before opening a pull request. The Makefile in the repository root shows what that means, with a ci target composed of fmt-check, lint, test-race and build, and separate targets for formatting, linting, tests, race-enabled tests and coverage. The linter is golangci-lint v2, installed into a .tools directory at the pinned version recorded in the .golangci-lint-version file.

make ci

make fmt
make lint
make test
make test-race
make coverage

Race and shuffle are in the test targets, which is a hint about the code

The Makefile is short and the detail worth noticing is in the test targets. The coverage target runs go test with the race detector, with -shuffle=on, and with atomic coverage mode, and test-race does the same minus the coverage. Shuffling the test order is a deliberate guard against a test that only passes because it depends on the order in which map entries or subtests happen to run, and a sketch library is exactly the kind of code where a hidden ordering dependency would be a correctness bug rather than a flaky test. The fact that the race detector is part of the default check sequence, not an optional extra, tells you the maintainers consider concurrent access part of the normal usage of the package. The repository layout supports that reading: the implementation is split across hyperloglog.go, beta.go for the bias correction, sparse.go for the sparse representation, compressed.go and utils.go, with test files covering the sketch and the sparse representation.

Order-independent merging is the feature that matters in a distributed pipeline

The README lists order-independent insertions and merging among the key features, promising consistent results regardless of data input order. For a single process counting distinct values in a stream, this matters less than it might appear, because a stream has an order anyway. It matters a great deal when the sketch is used as a mergeable structure, which is the usual reason to reach for HyperLogLog in the first place. A sketch per shard, per region or per collector, merged at the end, is the pattern that makes count-distinct affordable on a large dataset, and it only works if merging is commutative and associative in practice. The same property also removes a class of nondeterminism bugs, where a result changes because goroutines happened to deliver events in a different order. If your design depends on merging sketches, this is the property to test first against your own expectations, and the presence of a shuffle-enabled test target suggests the maintainers treat ordering as a property to be defended rather than assumed.

Maintenance, versioning and the licence

The last push to the repository was on 2026-09-14, and the most recent tags are v0.3.0 and the v0.3.0-rc.1 pre-release, both from that date, with v0.2.6 before it on 2025-12-16. The version numbers are still in the zero series, which is worth keeping in mind when you depend on it: pre-1.0 versions carry no compatibility promise in the Go ecosystem's own conventions, and this one is explicit that it remains backwards compatible with previous versions, which is a statement about the v0.1.0 era rather than a promise about the next major change. The release process is automated, with the .goreleaser.yaml and a release workflow that validates tags matching v*.*.* before publishing a GitHub Release with generated release notes, and tags carrying a pre-release suffix such as v1.0.0-rc.1 publishing as pre-releases. The licence is MIT, with the copyright held by Axiom, Inc. for 2026, and the README credits Prof. Shigang Chen and his team at the University of Florida for the surrounding research.

Editorial conclusion

The library suits a Go service that needs distinct counts over a large stream on a fixed memory budget, and that is willing to pick a register count deliberately rather than accept a default. It is the wrong choice when you need exact counts, when you need to merge sketches produced by a different library, or when a tailcut representation is required for a specific reason, because the implementation moved away from that method. Before adopting it, read the implementation history note in the README, because the repository description advertises TailCut space reduction that the current code no longer uses, and check the register count against your accuracy requirement, since the default of 2^14 registers costs 16 KB while the minimum of 2^4 costs 16 bytes and the maximum of 2^18 costs 256 KB.

Frequently asked questions

Official sources

1. axiomhq/hyperloglog on GitHub
2. License: MIT
3. Project website
4. README
5. Releases