broken-link-checker and the mechanics of a pauseable HTML link crawl

A crawler that only ever looks at HTML documents

broken-link-checker does one job: parse HTML, collect the URLs it references, and report which of them no longer resolve. It suits people auditing a site they control, either from a terminal before a deploy or from a Node script that has to keep the results. Its published topics are html5, http, link-checker, links, nodejs, seo, urls and whatwg, which puts the scope firmly inside markup rather than a general protocol sweep.

The unit of work is the page. `HtmlChecker` scans a single document you hand it, `HtmlUrlChecker` fetches every queued URL and scans whatever HTML comes back, and `SiteChecker` is named as the class to reach for first. Only those first two carry a written method and event list, so SiteChecker's surface has to be read out of lib/ before anything gets scripted against it.

npm, Node 14 or newer, and a first blc scan of a URL or a file

Node.js `>= 14` is the stated floor. Two installation paths exist, and the global one puts a blc binary on your PATH:

npm install broken-link-checker -g

The next line of the README points you at the help output instead of listing the flags, so the quick examples never explain what each switch does:

blc --help

A first run takes either a site root or a local file, and the `-ro` switch appears in both examples:

blc http://yoursite.com -ro
# or
blc path/to/index.html -ro

For script use, drop the `-g` flag and require the package from the project instead:

npm install broken-link-checker

Every page the crawl discovers becomes new work, so start with the file run: it is bounded, and it takes the same flags as the site-wide run.

Relative URLs resolve or fail on one argument

Every scan needs to know where it is. `.scan(html, baseURL)` takes the document and a URL object, and without that second argument any relative URL is reported broken with the reason `BLC_INVALID`, unless the document carries an absolute `<base href>`. The failure mode is quiet in a nasty way: those links are not dead, they were never anchored to an origin, so a crawl of a local file that forgets the base URL marks every stylesheet and image as broken.

The `'html'` event fires once the document is fully parsed and hands over the parse5 tree, which lets you read attributes the URL extraction does not care about. Relative URLs under `<base href>` are the one case where the missing base URL argument is recoverable. Parsing itself is claimed to be WHATWG compliant, so browser and crawler disagree about a malformed document less often than with a regex extractor.

Pausing the queue leaves requests already in flight running

The queue is public surface. `.pause()` stops new links from being dequeued, `.isPaused` reports the state, and `.numQueuedLinks` with `.numActiveLinks` show how much of each remains, while `.resume()` starts the flow again. What the README states plainly is that pausing does not pause any active requests, so a pause stops new traffic rather than halting traffic. `.clearCache()` drops cached URL responses, which matters when a host starts answering differently and you re-run.

`HtmlUrlChecker` adds page-level control on top. `.enqueue(pageURL, customData)` returns a queue ID, `.has(id)` asks whether that ID is still active or queued, and `.dequeue(id)` returns `true` on success or `false` on failure. The optional `customData` of any type rides along on the `'page'`, `'link'` and `'junk'` events, so your own crawl context reaches the callback without a side lookup table.

Handler exceptions are absorbed instead of failing the scan

All of these classes extend EventEmitter, and the `'error'` event carries an unusual contract: it is emitted when an error occurs inside one of your own event handlers, and it prevents the current scan from failing. A callback that throws therefore does not stop the crawl, and it does not raise anything you see either, unless you read the error argument in a handler of your own.

Result reporting comes in two shapes. `HtmlChecker` emits `'link'`, `'junk'` and `'complete'`, where `'complete'` arrives after the last result or after zero results. `HtmlUrlChecker` emits `'link'`, `'junk'`, `'page'` and `'end'` instead, and its `'page'` event carries the fetch error for a page that never arrived. Both hand you a `Link` object per URL, broken or not, and that object is the unit you would aggregate into a report of your own.

Four dependencies install from git with no version range

package.json declares version 0.8.0 under the MIT licence, the repository keeps a changelog.md, and GitHub hosts no releases for the project. Four dependencies are bare GitHub references with no semver range at all: `limited-request-queue`, `robot-directives`, `url-relation` and `urlcache`, each one a stevenvachon repository. npm resolves a reference shaped like that by fetching source rather than a published tarball, so those four track whatever their default branch holds at install time, and the top level of the repository holds no lockfile to pin them.

The rest of the list is pinned the usual loose way, from `chalk` at ^4.1.0 through got at ^10.0.2 and `parse5` at ^6.0.1, with `lodash` at ^4.17.21 and `optionator` at ~0.9.1 covering output and argument parsing, plus `parse-srcset` at ^1.0.2, `strip-ansi` at ^6.0.0 and `node-notifier` at ^9.0.0. Packaging has its own oddity: the entry point is lib-cjs, a directory absent from the repository's top level, which lists .eslintrc, .github/, .gitignore, README.md, bin/, lib/, package.json, scripts/ and test/. A build step under scripts/ has to produce that directory before an install can load the library. The last push was on 2026-09-07 and the repository is not archived.

The alternative is a parse5 and got pipeline you maintain

Both `parse5` and got sit in the ordinary dependency list here, so the same job can be assembled by hand: parse markup with parse5, fetch with got, keep your own frontier and your own concurrency counter. That route gives up the pieces this project already solved. Here the crawler already deduplicates through a cache, bounds concurrency through a request queue, follows redirects, handles compression and basic authentication, and optionally honours robot exclusions from robots.txt, headers and rel attributes, with simple wildcard filtering to skip URL keywords you never want checked.

Where the hand-rolled route wins is network shape and policy. HTTP proxies are not directly supported, and the README's own workaround is a container with proxy settings, so a corporate or egress-filtered network means wrapping the tool rather than configuring it. Keyword filtering is wildcard matching rather than regular expressions, and pause and resume exist here so a long crawl can be held open across a maintenance window instead of restarted.

Documentation links point at master while the default branch is main

The README hands out deep links to lib/internal/Link.js on a master branch, while the default branch of this repository is main. Every result object a callback receives is documented through those links, so anyone following them is asking a hosting site to redirect or guess. The same drift shows up in the binaries: package.json registers two command names, blc and `broken-link-checker`, and both point at bin/blc, yet every documented example uses blc on its own.

None of that breaks the tool. It does mean the prose is a weaker reference than the code, so treat the class, method and event names as the stable part and read lib/ for anything a path points at. One more gap belongs here: `SiteChecker` gets a recommendation and no documentation at all, while the two classes around it carry full method and event lists, which is why the class you are told to use is the one you have the least to go on.

Editorial conclusion

Adopt broken-link-checker when the audit runs against a directly reachable site and you want one result object per link inside your own script rather than a report file. Leave it alone for networks that need a proxy, or for anything outside HTML markup. Before wiring it up, read SiteChecker's surface out of lib/, since the README names that class and documents neither its methods nor its events, and pin the four git dependencies yourself, because npm fetches them from source at whatever their default branch holds. A first run against a local file will report every relative link as broken unless you pass a baseURL.

Frequently asked questions

Official sources

1. Issues
2. License: MIT
3. README
4. stevenvachon/broken-link-checker on GitHub