# This domain list is 1.7 billion names in 4.6 gigabytes, and the crawler behind it respects robots.txt

> The largest public list of Internet domains, maintained by crawling and DNS checks over years of traffic, with the public version on Git LFS and the unfiltered version behind a subscription. The interesting engineering is the crawl budget: petabytes of traffic for a compressed text file.

**tb0hdan/domains** — World’s single largest Internet domains dataset

- Repository: https://github.com/tb0hdan/domains
- Website: https://domainsproject.org
- Stars: 1,178 · Forks: 186
- Language: JavaScript
- License: BSD-3-Clause
- Published: 2026-10-19 · Updated: 2026-10-19 · Language: en
- Canonical page: https://hysenlabs.com/projects/tb0hdan-domains

## The list is one text file per country, and the size is the point

The Domains Project describes itself as processing petabytes of data so you do not have to, and the dataset is a public, freely available, sorted list of Internet domains. The badge in the README says 1.7 billion domains.

The format is as plain as it gets. After unpacking, domain lists are just text files, about 49Gb at 1.7 billion, with one domain per line. The sample given is for the Afghanistan directory.

```
1tv.af
1tvnews.af
3rdeye.af
8am.af
aan.af
aaa.gov.af
```

Each country has its own file, with the file named `domain2multi-af.txt` in the `data/afghanistan/` directory. The `domain2multi` prefix is the interesting part: it means the file contains second level domains plus the subdomains beneath them, which is why the sample includes entries like `acaa.gov.af` and `acku.edu.af` alongside the plain `af` domains.

The compression is remarkable and is stated as a fact in the README. Only 4.6Gb of disk space is required to store 1.7 billion domains in compressed form. That is a compression ratio far beyond what a gzip on a text file would give, and it is why the README mentions XZ specifically and why the file layout is the way it is.

The repository also carries the derived reports. There is `STATS.md` with the GitHub dataset statistics, `REPORTS.md` with the active domains graphical reports, `NEWS.md` for the project news, and `SUBSCRIPTIONS.md` for the support tiers. A `logos/` directory holds the trusted-by logos and a `reports/` directory sits alongside the `data/` directory.

## Cloning it means Git LFS and xz, and the milestone list explains why

The dataset is stored in Git LFS, so a plain clone is not enough. The README gives the procedure.

```bash
git clone https://github.com/tb0hdan/domains.git
cd domains
git lfs install
./unpack.sh
```

Three steps, and the third is a script in the repository root rather than a command you have to know. The `unpack.sh` script is what handles the XZ decompression, which the README recommends because the files use XZ.

The milestone list explains why the setup is the way it is, and it is a history of the format changing under the users. After reaching 9 million domains the repository was switched to compressed files, and the README asks users to use the freely available XZ to unpack them. After reaching 30 million records the files were moved to `/data` so the repository does not have its README at the very bottom.

The scale milestones are equally informative. The public list went 10 million, 100 million, 1 billion, and then 1.7 billion on GitHub, and the subscription tiers continue at 2.8 billion, 2.9 billion and 3 billion, the last one annotated with the note that yes, it is true.

The traffic milestones tell the same story from the other side, running from 500TB through 925TB, 1PB, 1.3PB, 1.5PB, 5.7PB and 8.1PB of what the README calls wasted Internet traffic.

The free version of the data is the GitHub repository, and subscribers have access to the raw data at dataset.domainsproject.org, along with some other available features. Those are a TLD only list, a WebSocket for new domains, and DNS JSON with historical data.

```bash
wget -m https://dataset.domainsproject.org
```

The `-m` flag is the mirror flag, which is the right call for a directory listing that changes over time.

## The crawler is rate limited and it tells you how to block it

A dataset of this kind is only as good as its collection method, and the README documents that method in unusual detail, including how to opt out of it.

All data in the dataset is gathered using the Scrapy and Colly frameworks. Starting with version 1.0.7 the crawler has partial `robots.txt` support and rate limiting, and the README asks users to open an issue if they experience problems, with a request to include their domain.

The two crawlers are separate projects. The DNS checks client is called Freya, is in early stages, is used by select few, and the author says he is working on making it stable and good enough for general public use. The HTTP crawler is being rewritten as well, and is called Idun.

The user agent is identifiable, and there are two forms because the version changed. The typical one looks like this.

```
Mozilla/5.0 (compatible; Domains Project/1.0.8; +https://domainsproject.org)
```

And some older versions set it to the GitHub repo.

```
Mozilla/5.0 (compatible; Domains Project/1.0.4; +https://github.com/tb0hdan/domains)
```

Then there is the section on disabling the bot's access to your website, which is the part that tells you the project is run in good faith. You add a `User-agent: domainsproject.org` with a `Disallow:/` line to your robots.txt, or the variant with `User-agent: Domains Project`, and the bot checks for both.

That the README documents the opt-out prominently is a meaningful signal, and it comes with the caveat that the support is partial. Partial robots.txt support means some sites with a disallowed path are still crawled, which is worth knowing if you operate a server you would rather not see in the list.

The passive DNS route is the one the README asks for help with. Running a Passive DNS sensor is listed as one of the ways to support the project, with a separate repository for the sensor, and it is a better data source for the purposes of this dataset than crawling, since it observes what is actually being resolved rather than guessing what exists.

## The crawl budget tells you the scale, and the hardware tells you the cost

The random facts section of the README is where the engineering of the project becomes concrete, and it is worth reading as a set of ratios.

More than 1TB of Internet traffic is just 3 Mbytes of compressed data, and 1 million domains is just 5 Mbytes compressed. More than 5.7PB of Internet traffic is necessary to crawl 1.7 billion domains, which the README puts as 3.4TB per million.

Then the throughput numbers, which are the ones to look at if you are running a sensor or thinking about what this costs. A 1Gbit fully saturated link is good for about 2 million new domains every day, and so is an 8 core, 16 thread machine with 64 Gbytes of RAM. And two ISC Bind9 instances, more than 400 Mbytes RSS each, are required to get 2 million new domains every day.

Those three facts together describe the shape of the operation. The DNS resolution is the bottleneck rather than the HTTP crawl, which is why Bind9 is called out specifically. The disk side is comparatively cheap, since 4.6Gb holds the whole compressed result. And the whole thing is a matter of running continuously, because a million domains is nothing compared to the registration rate.

The scale of the collection is also the reason the support asks are what they are. The project wants other domain datasets attached, and it wants passive DNS sensors, because those are the two ways to make the list grow without the project crawling more of the web itself.

The primary language in the metadata is JavaScript, and the topics list includes the crawlers and search engines it works alongside: colly, dataset, internet-domains, scrapy, search-engines and yacy. Yacy gets its own section in the README, described as a great open source search engine, with a link to a post on the Yacy forum about using a domain list for easier search bootstrapping, which is a concrete use case for a list this large.

## What the licence and the data shape mean for a real user

The licence is BSD-3-Clause, and the repository root carries the LICENSE file. That is a permissive grant with no copyleft, and for a data repository it is a simple answer to the question of what you may do with it.

The practical caveats are not legal ones. The first is the size and the tooling. A 49 gigabyte text output from a 4.6 gigabyte download, delivered through Git LFS, means your environment needs the Git LFS client, the XZ tool, and enough disk to hold the expanded form. A CI pipeline that clones this repository casually will not enjoy it.

The second is freshness. The list is the output of a crawl that is running continuously, so the public repository's contents are a snapshot of some moment, and the update cadence of the GitHub copy is separate from the crawl rate. The `NEWS.md` file and the `STATS.md` statistics are where you look for the current state, and the subscriber features, the WebSocket for new domains and the DNS JSON with historical data, are the parts that solve the freshness problem properly.

The third is the shape of the data by country. Because the files are organised by country code, a study about a specific population is straightforward if that country is well covered and incomplete if it is not. The STATS.md file with the GitHub dataset statistics, and the graphical reports in REPORTS.md, exist so you can check coverage before you rely on it.

And the fourth is what a domain list enables that has nothing to do with measurement. A list of every registered domain is a list of every organisation that has a web presence, which makes it useful for research, for dataset bootstrapping, for brand monitoring and for feeding a search engine. It also makes it a target list, and the fact that the crawler identifies itself and honours robots.txt, and asks for opt-outs, is the reason the project is usable at all.

## Conclusion

Adopt this dataset if you need a large, current list of registered domain names for research, for seed generation or for coverage measurement, and you can work with Git LFS and compressed files. Do not adopt it if you need a commercial data feed with a support commitment, because the unfiltered version is a subscription and the free version is a Git repository of compressed files. Verify first that the country code breakdown in the data directory matches the population you need, that the clone and unpack procedure works in your environment, and that the terms under which you use the list permit what you intend to do with it.

## FAQ

### How do I download the domains dataset?

The repository uses Git LFS, so clone it, run git lfs install and then run ./unpack.sh in the repository root. The files are XZ compressed, and the README asks users to use the freely available XZ tool to unpack them.

### What is the format of the domain list files?

After unpacking, the domain lists are plain text files, about 49Gb at 1.7 billion domains, with one domain per line, organised per country in the data directory. The filename pattern such as domain2multi-af.txt indicates the file contains second level domains and their subdomains.

### How much space does the full dataset need?

Only 4.6Gb of disk space is required to store 1.7 billion domains in compressed form, according to the README, and the uncompressed text is around 49Gb.

### Can I block the Domains Project crawler from my site?

Yes. Add a User-agent line with domainsproject.org or Domains Project and a Disallow:/ line to your robots.txt, and the bot checks for both. The crawler has had partial robots.txt support and rate limiting since version 1.0.7.

### What is the difference between the free and the subscribed data?

Subscribers have access to the raw unfiltered data at dataset.domainsproject.org, plus features such as a TLD only list, a WebSocket for new domains and DNS JSON with historical data. The GitHub repository holds the compressed public list.

## Sources

- [Issues](https://github.com/tb0hdan/domains/issues)
- [License: BSD-3-Clause](https://github.com/tb0hdan/domains/blob/master/LICENSE)
- [Project website](https://domainsproject.org)
- [README](https://github.com/tb0hdan/domains/blob/master/README.md)
- [tb0hdan/domains on GitHub](https://github.com/tb0hdan/domains)

---

Hysen Labs editorial analysis, written from the project's own repository and release notes. Cite the canonical page: https://hysenlabs.com/projects/tb0hdan-domains