tldts: pulling the registrable domain out of a URL in a fraction of a microsecond

What the library answers, and why the question is harder than it looks

tldts is a JavaScript library to extract hostnames, domains, public suffixes, top-level domains and subdomains from URLs. That list of five things is the whole scope, and the fact that it needs a library at all is the interesting part.

Given http://www.writethedocs.org/conf/eu/2017/, the naive answer for the domain is writethedocs.org. But that answer only holds because org is a public suffix, and the same question applied to spark-public.s3.amazonaws.com gives two defensible answers depending on whether you treat the private section of the public suffix list as authoritative. The registrable domain is s3.amazonaws.com under the private section, or amazonaws.com if you ignore it.

The public suffix list is the mechanism that resolves this. It is a list of all the suffixes under which the public registry operates, and a library that wants to give you the registrable domain has to know that co.uk is a suffix and uk is not, and that s3.amazonaws.com is a private suffix. Keeping that list current is a chore that most projects do badly, and tldts ships a continuously updated version of it.

The feature list names the other things the library handles: URLs and bare hostnames, full Unicode and IDNA support, parsing email addresses, and detecting IPv4 and IPv6 addresses. The IP detection matters because the suffix list has nothing to say about 192.168.1.1, and a parser that tries to find a public suffix in an IP address will happily return something nonsensical.

It is written in TypeScript, ships umd, esm and cjs bundles plus type definitions, and describes itself as battle tested with full test coverage and production use.

The API surface: one parse function and a set of narrower ones

The programmatic entry point is a parse function that takes a URL or a hostname and returns a flat object of everything it found:

const { parse } = require('tldts');

// Retrieving hostname related informations of a given URL
parse('http://www.writethedocs.org/conf/eu/2017/');
// { domain: 'writethedocs.org',
//   domainWithoutSuffix: 'writethedocs',
//   hostname: 'www.writethedocs.org',
//   isIcann: true,
//   isIp: false,
//   isPrivate: false,
//   publicSuffix: 'org',
//   subdomain: 'www' }

ES module imports work the same way, and there is a runkit link so you can try it in a browser without installing anything.

The result object carries more than the three parts you probably want. domain is the registrable domain, domainWithoutSuffix strips the public suffix from it, hostname is the full host, publicSuffix is the suffix itself, and subdomain is everything to the left. Then there are three booleans: isIcann, isPrivate and isIp.

The booleans are the part that makes the library trustworthy rather than merely convenient. isIcann tells you the suffix came from the ICANN section of the list, isPrivate tells you it came from the private section, and isIp tells you the input was an address rather than a name. A classification system that only wants the registrable domain can ignore all three, but a system that decides whether a host is a public website or an internal service needs them.

The migration table also points at narrower functions, getDomain, getPublicSuffix and others, with a note that using specific functions like getDomain are more efficient than relying on parse. That is a real and common optimisation: if you only need the registrable domain, parsing everything else is wasted work.

The default that will surprise you: private suffixes are ignored

The migration section is the most important part of the README, and it exists because tldts does not behave like psl by default.

The biggest difference between tldts default behavior and some other libraries like psl has to do with which suffixes are considered by default. The default for tldts is to only consider the ICANN section and ignore the Private section.

The example makes the consequence visible. With the unmaintained psl library, parsing a URL under spark-public.s3.amazonaws.com gives a public suffix of s3.amazonaws.com, which is from the Private section, and a domain of spark-public.s3.amazonaws.com. With tldts and no options, the same input gives a domain of amazonaws.com.

So the same hostname, parsed by two libraries, produces two different registrable domains. If you are grouping hosts by domain for analytics, for cookie scoping, or for a security policy, this difference is the whole ballgame, and it will not show up as an error. It shows up as a metric that quietly changes.

The fix is an option, and the tables make the equivalent call explicit for each psl function. For a hostname, tldts.parse('spark-public.s3.amazonaws.com', { allowPrivateDomains: true }). For a URL, the same option, and the note adds that there is no need to extract hostnames from URLs because tldts can do that. For the domain, tldts.getDomain with the same flag. For the public suffix, tldts.getPublicSuffix with the same flag.

The note in the hostname table is the operational one to remember: make sure to include allowPrivateDomains: true to consider private suffixes.

Reserved names are a separate, opt-in detection rather than a default

The special-use domain handling is a good example of a library making a deliberate performance-versus-completeness trade and documenting it clearly.

Reserved special-use names such as localhost, *.test, *.invalid, example(.com/.net/.org), *.local, *.onion, *.alt, and home.arpa are not identified by isIcann or isPrivate. Most are not in the Public Suffix List, and the few that are, such as onion and home.arpa, appear there as ordinary ICANN suffixes.

So without special handling, parse('http://printer.local/') tells you a public suffix of local and an isIcann of true, which is not very useful information about a printer on a local network.

The solution is opt-in:

const { parse } = require('tldts');

parse('http://printer.local/', { detectSpecialUse: true });
// { ...
//   isSpecialUse: true,
//   publicSuffix: 'local',
//   subdomain: '' }

The key point is what the field is otherwise. The isSpecialUse result field is null by default, so the default path does no extra work. The library is not silently returning false, which would be a lie, and it is not always paying for a check nobody asked for. It returns null, and null is the absence of an answer rather than a negative one.

The list tracks the IANA Special-Use Domain Names registry, which is RFC 6761 and the later RFCs that extend it. That is the right source to track, since it is the registry that actually defines these reservations, rather than a hand-maintained list inside the library.

A command line interface that takes a URL or a stream of them

The library ships a CLI, and it is genuinely useful for spot checks and for shell pipelines.

Point it at a single URL and it prints the parsed result as JSON:

$ npx tldts 'http://www.writethedocs.org/conf/eu/2017/'
{
  "domain": "writethedocs.org",
  "domainWithoutSuffix": "writethedocs",
  "hostname": "www.writethedocs.org",
  "isIcann": true,
  "isIp": false,
  "isPrivate": false,
  "publicSuffix": "org",
  "subdomain": "www"
}

And it reads from standard input, so you can pipe a list of URLs through it and get one JSON object per line. That is the shape of a tool you can use in a shell one-liner to check a batch of hosts without writing a script.

The CLI is also the fastest way to check the behaviour difference described earlier, because you can run the same hostname through the library and through psl and see the two different domains side by side, which is a more convincing argument than a table in a README.

Installation for the library itself is a single command, npm install --save tldts, and the package is published to npm as tldts.

The performance target is stated as an order of 0.1 to 1 microsecond per input, and the README carries a chart comparing getDomain throughput on a real-world URL corpus against psl, tldjs, parse-domain, tld-extract and haraka-tld, benchmarked on Node. The comparison directory at the root of the repository holds the harness, so the claim is reproducible rather than asserted.

A monorepo with its own suffix list generator and a benchmarking setup

The repository is a monorepo, and its structure explains where the data comes from.

There is a packages/ directory holding the published packages, with the main library at packages/tldts and a separate package for the utilities, referenced in the root package.json by a generate-data script that runs a TypeScript script against the tldts-utils package. There is a publicsuffix/ directory, which is where the suffix list data lives, and a .gitmodules file, so the list is likely fetched as a submodule from the canonical upstream rather than copied in.

The tooling is a mix of several tools, which is common in a project that predates some of them. Yarn 4.18.0 is the package manager, lerna handles the multi-package orchestration, and nx.json is present for task running. The build is tsc --build over a project tsconfig followed by lerna run build, and the bundle step runs through lerna with a concurrency of 2.

The scripts are worth reading for what they imply. There is a bench script that runs make -C bench, and the comparison/ directory holds a second benchmark against other public suffix libraries, separate from the internal one. There is a mocha configuration in .mocharc.cjs and a metalint configuration in .metalint.json, which is a linter for package metadata consistency across the monorepo.

Test and lint run per package through lerna with a concurrency of 2, and the lint command is eslint with a flat config in eslint.config.mjs. Formatting is prettier with its own .prettierrc, and .tool-versions pins the tool versions.

The licence is MIT, the author is Rémi Berson, and the homepage for the package is the npmjs.com/tldts listing.

Editorial conclusion

Adopt tldts if you need to group hosts by registrable domain at scale, in TypeScript, and you want the public suffix list kept current without running your own update job. Do not adopt it if you only need a one-off parse of a handful of URLs, because the platform URL class will do, and do not adopt it expecting psl-compatible results, since the default ignores the private section of the suffix list. Before you migrate, read the migration tables and decide whether you need allowPrivateDomains, and if you are doing bulk classification, set detectSpecialUse only if you actually consume that field, since the default path is written to do no extra work.

Frequently asked questions

Official sources

1. License: MIT
2. Project website
3. README
4. Releases
5. remusao/tldts on GitHub