html5lib builds the tree a browser would build, in pure Python

The tree is the one a browser builds, not the one XML would

The README states the goal in one sentence. html5lib is a pure-python library for parsing HTML, designed to conform to the WHATWG HTML specification, as is implemented by all major web browsers.

That is a narrower and more useful claim than it first appears. HTML has two histories, and parsers disagree about which one to follow. There is the tidy XML lineage, where a void element has no closing tag and the grammar is regular enough to reason about, and there is the messy reality of the web, where the parser has to recover from unclosed tags, implied end tags, misnested formatting elements and a long list of error-handling rules so that every browser ends up with the same document.

html5lib implements the second set of rules. That is why it is a pure-Python library and not a wrapper around a C parser, and it is why it is the reference for the tree construction algorithm rather than an approximation of it. If your tool compares a parsed document against what a browser renders, or produces a document that a browser must not reparse differently, matching the spec is the whole point.

The library covers both documents and fragments, parsing and serializing, and the interface is deliberately small.

import html5lib
with open("mydocument.html", "rb") as f:
    document = html5lib.parse(f)

The same call takes a string, so html5lib.parse("<p>Hello World!") is equally valid. Everything hangs off that one entry point unless you need more control, in which case you instantiate the parser yourself.

The tree builder decides what object you get back

By default the document will be an xml.etree element instance, and whenever possible html5lib chooses the accelerated ElementTree implementation, which the README notes is xml.etree.cElementTree on Python 2.x. So the default output is a standard library tree, not a bespoke one, which matters if you intend to walk it with code you already have.

Two other tree types are supported, xml.dom.minidom and lxml.etree, and you select one by name.

import html5lib
with open("mydocument.html", "rb") as f:
    lxml_etree_document = html5lib.parse(f, treebuilder="lxml")

The same choice is available when you build the parser explicitly, through the tree keyword argument.

import html5lib
parser = html5lib.HTMLParser(tree=html5lib.getTreeBuilder("dom"))
minidom_document = parser.parse("<p>Hello World!")

This is a genuinely useful seam rather than an implementation detail. The tree construction algorithm, which is where the spec lives, is separate from the representation of the result. If you want lxml's tree because your code already knows how to query it, or because lxml's XPath support is what you need, you can have it while keeping the WHATWG behaviour.

There is one optional dependency with a treewalker but no tree builder, genshi, and the README is precise about that asymmetry. lxml is supported for both building and walking, under CPython, but not on PyPy where it is known to cause segfaults. That last point is not a footnote, because a segfault in a parser is not something you want to discover in production.

Encoding is the detail that breaks crawlers, and it is handled explicitly

HTML pages arrive with their encoding declared in three different places, and they disagree with each other. The encoding can be in a meta tag, it can be in the HTTP Content-Type header, and it can be in the absence of both, in which case the spec has rules for guessing.

The README shows the correct way to handle the header case, and it is the part most hand-rolled parsers get wrong.

from urllib.request import urlopen
import html5lib

with urlopen("http://example.com/") as f:
    document = html5lib.parse(f, transport_encoding=f.info().get_content_charset())

The transport_encoding argument is how the charset from HTTP gets into the parser, and there is an equivalent example for Python 2's urllib2 using getparam("charset").

For pages with no usable declaration, chardet can be used as a fallback when character encoding cannot be determined, and the core requirements list webencodings alongside six, which is the package that maps encoding labels to the actual codecs. The dependency list is short on purpose: six for Python 2 and 3 compatibility, webencodings for the charset table, and everything else optional.

A second, quieter benefit of the transport_encoding route is that the parse is done in one pass over the bytes, without decoding the whole document first and hoping. When a page is large or when a scraper runs against untrusted input, that difference matters.

strict=True turns silent recovery into loud failure

By default, a parser recovers. That is the point of the HTML parsing algorithm, and it is why html5lib matches browsers. The cost is that a malformed document produces a tree without telling you anything was wrong, which is exactly the wrong behaviour when you are writing a validator or a linter.

The library gives you the opposite behaviour on request. To have more control over the parser, create a parser object explicitly, and for instance, to make the parser raise exceptions on parse errors.

import html5lib
with open("mydocument.html", "rb") as f:
    parser = html5lib.HTMLParser(strict=True)
    document = parser.parse(f)

The same object is what you use when you need to choose a different tree format, so the two customisations live in one place. The parser object is the API for everything beyond the simple parse call, and both of the things you are likely to want to change, the tree format and the error strictness, are arguments to it.

For a tool that validates markup, strict mode is the feature that matters. For a tool that must process whatever the web serves, the default recovery behaviour is the feature that matters. Which one you want depends entirely on whether you are the browser or the critic, and html5lib lets you be either without forking the library.

The test suite is the spec suite, pulled in as a submodule

How a project proves it implements a specification is a question about the tests, and html5lib's answer is unusually direct. Test data are contained in a separate html5lib-tests repository and included as a submodule, so for git checkouts they must be initialized.

$ git submodule init
$ git submodule update

That means the conformance data is not a snapshot frozen inside this repository. It lives in its own project, and a change to the shared test data is a change to what html5lib is measured against, which is a much better arrangement for a library whose entire value proposition is conformance.

Unit tests require the pytest and mock libraries and can be run using the pytest command in the root directory. If you have all compatible Python implementations available on your system, you can run tests on all of them using the tox utility, which can be found on PyPI.

The rest of the tree supports the same story. There is a benchmarks/ directory, a utils/ directory, a doc/ directory, an AUTHORS.rst, a CHANGES.rst and a CONTRIBUTING.rst, plus a set of requirements files, requirements.txt with the runtime dependencies, requirements-test.txt, requirements-optional.txt and requirements-oldest.txt. That last one is worth noting: the goal is to support a non-strict superset of the versions that pip supports, and the oldest-supported set is a separate file so the minimum can be tested without pinning the whole environment.

The static analysis setup is old-school and visible: a .pylintrc, a .prospector.yaml, a flake8-run.sh and a .coveragerc, along with an .appveyor.yml for Windows CI and a GitHub Actions workflow for python-tox. The project predates the current tooling conventions, and it still carries them.

Pure Python is the trade, and lxml is only part of the answer

The limitation is the one the README implies rather than states. A pure-Python parser that implements the full WHATWG tree construction algorithm does more work per document than a compiled parser that handles the common cases. If your workload is parsing thousands of pages an hour and you only need a few values from each, this is the wrong tool, and the right tool is a compiled parser that tolerates the difference in the tree.

The escape hatch is the lxml tree builder, which moves the tree representation to a compiled library while keeping the WHATWG algorithm in Python. That helps with the representation, not with the parsing, so the win is real but partial, and it comes with the PyPy caveat about segfaults.

The alternative approach is a compiled parser that does its own error recovery, and the difference is exactly what html5lib is for. A compiled parser will generally be faster and will produce a tree that is nearly right; html5lib produces the tree that is right. Which one you need depends on what you do with the result. If the result is fed to something that will be re-parsed by a browser, or compared with a browser's own output, matching the spec is worth the time. If the result is a title tag, it is not.

The second limitation is Python 2. The README says html5lib works on CPython 2.7+, CPython 3.5+ and PyPy, and several examples in the usage section are Python 2 code, including the urllib2 variant and the cElementTree note. The library is maintained for both, and the six dependency is there to make that work. For a project starting today that is a consideration rather than a blocker, but it is a sign of the codebase's age.

The third is that the visible documentation is thin. The README points to the docs site for more, and the repository itself carries doc/ and the CHANGES file, but the API surface is documented on readthedocs rather than in the repository.

MIT licensed, community support on Discussions, and a long tail of maintenance

The licence is MIT, and the LICENSE file is at the repository root. You can use, modify and redistribute the library, including in commercial and closed-source products, provided the copyright notice and licence text are retained. This is a licensing fact rather than legal advice, and for a parsing library that many projects depend on indirectly, it removes an entire category of legal review.

The support model is community-run. Please report any bugs on the issue tracker, and the questions section points at the docs, then at GitHub Discussions, and then at the archives of the html5lib-discuss mailing list. The order is telling: the documentation is the first answer, discussions are the second, and the mailing list archives are the last resort.

The activity record needs reading with care. The repository is not archived, the default branch is master, and the last push was on 2026-04-21. There are no GitHub releases, so there is no tagged version list to check against, and the changelog lives in CHANGES.rst rather than in a release feed.

The version support statement in the README is the most useful thing for a new adopter. html5lib works on CPython 2.7+, CPython 3.5+ and PyPy, and the goal is to support a non-strict superset of the versions that pip supports. In practice, that means the library is unlikely to be the reason a modern environment fails to install, and it means the oldest-interpreter testing is a deliberate part of the process rather than an accident.

The overall picture is a mature, spec-first library with a slow commit rhythm and a small footprint, which is exactly what you want underneath a tool that has to agree with a browser.

Editorial conclusion

html5lib fits a Python project that must produce the same tree a browser would, such as a test harness, a linter, an accessibility checker or a scraper whose output is compared against a reference, and it fits a team that cannot install a C extension. It does not fit a high-throughput scraping pipeline where parsing time dominates, and it does not fit a project that only needs to pull a few values out of a well-formed page. Before adopting it, measure the parse time on your own documents, decide whether the WHATWG tree matters more than the speed, and if you use lxml, test on PyPy, where the README says lxml is known to cause segfaults.

Frequently asked questions

Official sources

1. html5lib/html5lib-python on GitHub
2. Issues
3. License: MIT
4. README