LinkChecker: a link checker that also crawls, and can hand you SQL

A link checker that crawls while it checks

LinkChecker is described as checking links in web documents or full websites, and the feature list explains the difference between a checker and a crawler in one line: recursive and multithreaded checking and site crawling.

That combination is the design. A document checker validates the links in one page, and a crawler follows the links it finds and validates those as well, and the recursion is what turns a spot check into a site-wide audit. The multithreading is what makes a site-wide audit finish in a reasonable time, since a serial crawl of a few thousand pages is dominated by waiting for other people's servers to answer.

The protocol support is broad for a tool of this age. It handles HTTP/1.1, HTTPS, FTP, `mailto:` and local file links, and it has HTML5 support. The local file support is the one that matters most in practice for a documentation site or a static site, because the internal links are the ones that break most often and the ones nobody notices.

The rest of the feature list is about control. You can restrict link checking with regular expression filters for URLs, so a run ignores a whole pattern of paths, which is how a check stays fast on a large site. There is proxy support, username and password authorization for HTTP and FTP, and cookie support, so a check can reach a page that requires a session. And it honors the robots.txt exclusion protocol, which is a matter of being a good citizen towards the servers being crawled rather than a feature for its own sake.

There is a command line and a web interface, and various check plugins are available.

The project is GPL-2.0, the `pyproject.toml` states GPL-2.0-or-later with the licence file being `COPYING`, the primary language is Python, the default branch is `master`, and the last push was on 2026-09-22. The recent releases are v10.6.0 from 2025-07-28, v10.5.0 from 2024-09-03 and v10.4.0 from 2023-12-11, so the release cadence is a version every year or two on a tool that is clearly used in production, since the package classifiers say so.

Output formats that are meant to be piped somewhere

The second line of the feature list is the one that decides whether a link checker is a tool or part of a pipeline.

Output in colored or normal text, HTML, SQL, CSV, XML or a sitemap graph in different formats. Six destinations, and the two interesting ones are the SQL and the sitemap graph.

A SQL output is a report in a form a database can ingest, which means a link check can be stored next to the other quality data of a project, compared between runs, and turned into a dashboard without any parsing in between. A CSV output is the same idea without the database. The HTML output is for a human who wants a report they can read or publish, and the sitemap graph is for a check that wants to show the structure of the site rather than a list of failures, which is a different question and a different artefact.

The coloured versus normal text distinction is a small thing that shows the tool's age and its care. A tool that knows it is being read by a person in a terminal will colour its output, and a tool that knows it is being read by a pipeline will not, and the flag to switch between them is usually one character.

Together with the check plugins and the per-URL filters, this is a tool that was designed to sit inside an automated quality process rather than to be run by hand, which is the strongest argument for it and also the reason to read its configuration rather than its options.

Installation, and the warning about the packaged version

Installation is one command, with a caveat that is worth reading before you run it.

pipx install linkchecker

The README recommends pipx rather than pip, which for a command line tool with its own configuration directory is the right default, since pipx installs it into its own environment and leaves the system Python alone.

Python 3.10 or later is needed, and the package metadata says the same, with classifiers for 3.10 through 3.15 and `requires-python` of `>=3.10`. The three runtime dependencies are `requests >= 2.20`, `dnspython >= 2.0` and `beautifulsoup4 >= 4.8.1`, and the optional ones are `argcomplete >= 1.8.1` for shell completion and `pdfminer.six >= 20181108` for PDF documents.

The caveat is this: the version in the Python Package Index may be old. To find out how to get the latest code, plus platform-specific information and other advice, the README points at `doc/install.txt` in the source code archive. Given the release list, with v10.6.0 published in July 2025 and commits continuing into September 2026, the packaged version can lag behind the branch by a year, and for a tool that tracks changes in how servers respond, that lag is worth knowing about.

The Docker route is the other answer to the same problem, and it is described separately. Running the tool from a container means the latest code is not an issue, and it is also the way to run it without installing anything at all.

Docker, with the two flags that make it usable

The container image is published on GitHub Packages, and the README gives two examples that between them show the two things the image is for.

For an external site:

docker run --rm -it -u $(id -u):$(id -g) ghcr.io/linkchecker/linkchecker:latest --verbose https://www.example.com

For a local HTML file:

docker run --rm -it -u $(id -u):$(id -g) -v "$PWD":/mnt ghcr.io/linkchecker/linkchecker:latest --verbose index.html

The `-u $(id -u):$(id -g)` is there so that the files the container writes belong to the user rather than to root, which matters because a crawl that saves a report or a cache would otherwise leave root-owned files in the working directory. The `-v "$PWD":/mnt` mounts the current directory so that a local file can be checked, and the `VOLUME /mnt` and `WORKDIR /mnt` lines in the `Dockerfile` are the matching configuration.

The `Dockerfile` itself explains the two details that would otherwise be mysterious. The base image is `python:3.13-slim`, with the comment that it uses the maximum Python version tested. And `ENV HOME=/tmp` is set because the tool creates `$XDG_CONFIG_HOME/linkchecker/` with mode 700 containing the `linkcheckerrc` and other files, and in a container that configuration directory would otherwise be written into the mounted home directory of whatever user the image happens to run as.

The image also installs the dependencies separately from the project itself, with the comment that dependencies change on their own schedule, and it installs the project from its git repository rather than from the package index, which is the same answer to the stale-package problem that the `doc/install.txt` note gives for source installs. The entrypoint is `linkchecker`, so the container takes the same arguments as the installed command.

Beyond the rolling `latest` tag, the README notes that uniquely tagged images are on the packages page, which matters for a pipeline that needs to pin the version it tested with.

The configuration file, the man pages and the plugins

The command itself is one line, and everything else lives in the documentation.

Executing `linkchecker https://www.example.com` is the whole of the usage section, and for the other options the README points at `linkchecker --help`, the manual pages `linkchecker(1)` and `linkcheckerrc(5)`, and the site.

The presence of a `linkcheckerrc(5)` manual page is the signal that this is a tool with a real configuration file, and the man page names are themselves informative: one for the command and one for the configuration file, which is the convention for a tool that expects to be configured.

The configuration lives in `$XDG_CONFIG_HOME/linkchecker/`, which the tool creates with mode 700 and which holds the `linkcheckerrc` and other files. That location is the modern one, and it is the same location the container image is arranged around.

The check plugins are the other half of the extensibility story. The feature list names various check plugins without listing them, so a reader has to look them up, and the honest way to describe them is as the extension point where the tool decides what counts as a valid link beyond the protocols it knows, which is where a site-specific check, a login flow or a custom status policy would go.

The combination of regular expression filters for URLs, the proxy setting, the HTTP and FTP authentication, the cookie support and the robots.txt handling is what makes the tool usable against a real site rather than a small static one, and all of it is configuration rather than code.

The build: hatchling, a custom hook, and a check that greps the changelog

The repository is a mature Python project, and its build configuration says how the package is put together.

The build backend is hatchling, with `hatch-vcs` and `setuptools-scm >= 7.1.0` alongside it, and the version is dynamic, so it comes from the repository state rather than from a number written in the file. There is a custom build hook at `tools/hatch_build.py`, and the hatch configuration names two artifacts, `linkcheck/_release.py` and `linkcheck/data/locale`, which are the generated release metadata and the translation data.

The wheel is configured to include only the `linkcheck` package, and the shared data section installs the manual pages, `doc/man/en/linkchecker.1` among them, so the documentation ships with the package rather than only in the source tree. The command itself is one console script, `linkchecker`, pointing at `linkcheck.command.linkchecker:linkchecker`.

The `Makefile` is for developers, and it says so in its first comment. The default target tells you to read `doc/install.txt` rather than pretending the Makefile installs anything. The real targets are `test`, which runs `tox -e py`, `dist`, which runs `hatchling build`, `upload`, which runs twine, `homepage`, which builds the documentation, and `check`, which runs flake8, yamllint on the GitHub workflows, and the documentation and translation checks.

The `releasecheck` target is the one worth reading:

releasecheck: check
	@if egrep -i "xx\.|xxxx|\.xx|^[[:digit:]]+\.x" doc/changelog.txt > /dev/null; then \
	  echo "Could not release: edit doc/changelog.txt release date"; false; \
	fi

It greps the changelog for placeholder version patterns and refuses to release while they are there, which is a guard against shipping a changelog whose top entry still says that the date is to be filled in. Small, and the kind of check that only exists in a project that has released many times.

The rest of the tree shows the scope of the project: a `po/` directory for translations, a `cgi-bin/` directory behind the web interface, `config/` for the check plugins, `doc/` for the manual and the documentation, `tests/`, `tools/` and `scripts/`, a `pyoxidizer.bzl` for a standalone binary build, and a `robots.txt` for the site itself.

Where LinkChecker is the wrong tool

The limits are the ones every HTTP-based link checker has, and they are worth knowing before the tool is put in a pipeline.

The first is that the tool reads documents, it does not run them. The HTML parsing is done with Beautiful Soup, so the links that a check finds are the links that are in the markup the server returned. A site whose navigation is built by JavaScript has those links in the document and the check can find them, and a site whose content is fetched and injected at runtime does not, and the check will report the page as having no links at all rather than reporting an error. A renderer-based check would find them, and a renderer is a much heavier thing to run in a pipeline.

The second is that a broken link and an unfriendly link look the same to a crawler. A page that answers 403 to anything that does not look like a browser, or that requires a session, is reported alongside a page that answers 404. The authentication, cookie and proxy options exist to handle that, which means the operator of the check has to know which parts of the site need credentials, and the tool will otherwise generate findings that are not findings.

The third is that a crawl is a request to somebody else's server, and the tool says so by honouring robots.txt. That is the right default and it is also a limitation, because a site that disallows crawling in its robots.txt will not be checked thoroughly, and the operator has to decide whether to respect that or not.

The alternative approach depends on what the check is for. For a documentation or content site, where the links are in the markup and the failures are real, this tool is the right shape, and the alternative is a link check inside a headless browser, which finds the JavaScript-built links and costs a browser per page. For a site that is a single page application, the difference in approach is the whole question: one parser sees the shell, and the other sees the application, and the second is the only one that can tell you whether a link in the interface works.

There is a third option for a large site, which is to check the links in the build output rather than on the live site, since a build artefact is stable and does not need the origin to be up while the check runs. That is cheaper and more repeatable, and it is what a continuous integration pipeline is usually for.

Editorial conclusion

LinkChecker fits a team that wants broken link detection in continuous integration rather than by hand, that needs the output in a machine-readable form it can act on, and that accepts a crawler which parses HTML rather than renders it. It is the wrong tool for a site whose links only exist after JavaScript runs, since the parser reads the document it is given rather than a rendered page, and it is the wrong choice for a check that has to distinguish a link that is broken from a link that refuses bots. Before wiring it into a pipeline, read `linkcheckerrc(5)` for the configuration file, because the defaults are conservative and the crawl scope is set there, and remember that it visits other people's servers, so the proxy, the authentication and the robots.txt handling are operational decisions and not only conveniences.

Frequently asked questions

Official sources

1. License: GPL-2.0
2. linkchecker/linkchecker on GitHub
3. Project website
4. README
5. Releases