extruct: one call to pull every embedded metadata format out of HTML

Six syntaxes parsed in one call

extruct is a library for extracting embedded metadata from HTML markup. The supported list is what defines it. It handles W3C HTML Microdata, embedded JSON-LD, Microformats through the `mf2py` library, Facebook's Open Graph, Dublin Core metadata, and RDFa through `rdflib`, the last of which the readme marks as experimental.

Six syntaxes that usually arrive in the same page, from different authors and different plugins, and one call that returns all of them. That is the point of the library. A page with a schema.org JSON-LD block, a set of `og:` meta tags, a handful of microformat class names and Dublin Core elements is a normal page, and reading it with a selector means writing one parser per syntax and remembering to run all of them.

The repository is a small Python package with a `setup.py`, `setup.cfg`, `pyproject.toml`, `pytest.ini`, `tox.ini` and a `tests/` directory, plus `HISTORY.rst` and an `AUTHORS` file. The author and maintainer are Scrapinghub, which matches the project description in the readme as a library built around the problem of extracting schema.org data from pages.

The microdata implementation has its own pedigree. The readme says the microdata algorithm is a revisit of a Scrapinghub blog post showing how to use EXSLT extensions, so the parsing strategy for one of the six formats came from an experiment with XPath extension functions rather than from a plain element scan.

One extract call returns a dict keyed by syntax

The simplest usage the readme describes is a single call.

import extruct
extruct.extract(htmlstring, base_url=base_url)

The optional base URL matters more than it looks. Metadata values routinely contain relative URLs, and without a base to resolve them against, the extracted values stay relative or are dropped. The `w3lib` library is a dependency for exactly this reason, and the worked example uses `get_base_url` to derive the base from the fetched document rather than assuming one.

The return value is a dictionary whose keys are the syntax names. In the readme example the result has `dublincore`, `json-ld`, `microdata`, `microformat` and `opengraph` keys. Each value is a list of objects, and the shape inside each list is whatever that syntax produces. Open Graph comes back as objects with a `namespace` dictionary and a `properties` list of name and value pairs. JSON-LD comes back as plain JSON-LD dictionaries with `@context`, `@type` and the original fields. Dublin Core comes back with `elements`, `namespaces` and `terms`.

That last point is the key design decision. extruct does not merge the six formats into one schema. A consumer that wants a title, a description and an image has to look in `opengraph`, then `json-ld`, then `microdata`, and decide which wins. In exchange you get exactly what the page said, without a lossy normalisation layer in between.

Installing and running the first extraction

Installation is a single pip command, and the readme gives it plainly.

pip install extruct

The library itself pulls in `lxml`, `lxml-html-clean`, `rdflib`, `pyrdfa3`, `mf2py`, `w3lib`, `html-text` and `jstyleson`. The `requirements.txt` file lists the same set and notes that it is installed with `pip install -r requirements.txt`.

The worked example fetches a page and hands the response body to extruct, deriving the base URL from the response text and final URL.

>>> import extruct
>>> import requests
>>> import pprint
>>> from w3lib.html import get_base_url
>>>
>>> pp = pprint.PrettyPrinter(indent=2)
>>> r = requests.get('https://www.optimizesmart.com/how-to-use-open-graph-protocol/')
>>> base_url = get_base_url(r.text, r.url)
>>> data = extruct.extract(r.text, base_url=base_url)
>>>
>>> pp.pprint(data)

The result is printed as one dictionary containing the five syntaxes found in that page. The `requests` call is part of the example rather than a dependency of the library, and it is installed separately: setup.py declares an extra named `cli` that adds `requests`.

There is also a console entry point, `extruct = extruct.tool:main`, declared in setup.py. The readme does not document how that command is meant to be used, so treat the command line interface as something to check in the source rather than a documented feature.

Each format is delegated to the library that already knows it

The dependency list explains how the extraction is split. Microformats go through `mf2py`, RDFa through `pyrdfa3` with `rdflib` underneath for the graph handling, and JSON-LD through `jstyleson`, a relaxed JSON parser that tolerates the trailing commas and unquoted keys that appear in the wild. HTML parsing itself is `lxml`, and `lxml-html-clean` is there for sanitising.

So extruct is an orchestrator rather than a from scratch parser. Each syntax is handled by a library built for it, which is why the package can cover six formats without carrying six parsing engines of its own.

The type checking configuration shows where the seams are. The mypy settings in pyproject.toml are strict, with `disallow_untyped_defs`, `disallow_incomplete_defs`, `warn_return_any` and `strict_equality` enabled, but there is an override list that sets `ignore_missing_imports` for `mf2py`, `pyRdfa`, `rdflib`, `jstyleson`, `urlparse` and `html_text`. Those are exactly the modules with no type information of their own, so the boundary of the library is well defined: the strict typing applies to extruct's own code, and it relaxes at the edges where third party parsers take over.

There is a practical consequence for anyone extending the library. The format specific work happens in delegated packages, so a change in `mf2py` or `jstyleson` can alter extruct's output without a change in extruct itself.

RDFa is the format that is not finished

The readme marks RDFa as experimental, and it is the one entry in the list with that qualifier. RDFa support is described as coming via `rdflib`, which is also listed as an experimental dependency, and the package name in the dependency list is `pyrdfa3`.

That distinction matters for anyone building a pipeline on top of extruct. A consumer that treats all six keys as equally reliable will be wrong about RDFa specifically, since the underlying parsing may not handle the full range of RDFa attributes and profiles. The microformats path has the same shape of risk from the other direction: `mf2py` is a separate project with its own release cycle, and the readme points at it rather than vendoring its behaviour.

The other formats are the well trodden ones. JSON-LD, Open Graph and Dublin Core are simple attribute and script scans for a library with an lxml tree in hand, and Microdata is the one the project has an explicit algorithmic history with.

A useful way to think about the output: the more you depend on a specific key, the more you depend on a specific third party parser's behaviour, and extruct does not add a normalisation or validation layer on top that would hide changes there.

Where extruct is the wrong tool

The obvious case is a page with no embedded metadata. extruct parses what is in the markup. If a site has no JSON-LD, no Open Graph tags and no microformat classes, the extraction returns empty lists and no amount of library helps. Pages that render their metadata client side are the same problem in a different form, since extruct works on HTML markup rather than on a live DOM.

The second case is when you need a merged, normalised record. The output preserves each syntax separately, which is faithful and predictable but means the consumer has to write the precedence rules. For a small number of pages that is a handful of lines. For a large corpus with six formats in play, the merging logic becomes your code, and it will need to change when you find another site that uses a field you had not seen.

The third is operational. The package is a library with no GitHub releases, and its version is read from an `extruct/VERSION` file inside the package directory, with setup.py loading it at build time and shipping it as package data. That means there is no release page to pin against, and version tracking depends on what PyPI shows you. The classifiers list Python 3.8 through 3.10 and setup.py declares `python_requires` set to 3.8 or newer, so the stated support range is narrower than the loose lower bound would suggest.

There is also a licensing point to note. The package is BSD licensed, with the licence file at the repository root, so it can be vendored into a commercial product, but the dependencies carry their own licences and a redistribution should account for all of them.

How this differs from writing your own selectors

The alternative is the obvious one: parse the page yourself with lxml and pull the metadata you care about with XPath or CSS selectors.

The difference in approach is breadth against control. Hand written selectors give you exactly the fields you want, in the shape you want, with no dependency on a third party parser and no ambiguity about precedence. The cost is that every syntax becomes your code, and microformats in particular are not a simple attribute scan, so the hand written version of that feature is where the real work sits.

extruct inverts that. It gives you everything the page carries, per syntax, from parsers that were written for those formats, in one pass over the document. The cost is the dependency set, the loose shape of the output, and the fact that a change upstream can move values around.

There is a middle path that many projects end up taking, and it is worth naming explicitly because it is probably the right one for most adopters. Use extruct to get everything, then write a thin layer that pulls the few fields you need out of the structure, with your own precedence rules. You get the parsing work for six formats handled, and you keep control of the part that your application actually depends on.

For the microdata case specifically, the readme points back to the original blog post about using EXSLT extensions. If you are evaluating whether to keep a hand written microdata parser, that post is the reference point for what the alternative implementation looks like.

Editorial conclusion

extruct suits anyone who needs the structured data a page already carries, especially teams doing schema analysis or building a corpus of JSON-LD, and it is the wrong tool for pages that expose no metadata at all or for applications that need the values merged into one normalised record. Before adopting it, check that the syntax you need is fully supported, since RDFa is still marked experimental, pass a `base_url` so relative URLs resolve, and pin your own version, since the package reads its version from an `extruct/VERSION` file and the repository publishes no GitHub releases.

Frequently asked questions

Official sources

1. Issues
2. License: BSD-3-Clause
3. README
4. scrapinghub/extruct on GitHub