# jusText: stripping boilerplate from HTML while keeping whole sentences

> jusText is a Python library and command line tool that classifies HTML paragraphs as boilerplate or content so that the remaining text still reads as full sentences, which is what makes it usable for building web corpora. It is a maintained fork of code that has been dormant on Google Code since the original project died.

**miso-belica/jusText** — Heuristic based boilerplate removal tool

- Repository: https://github.com/miso-belica/jusText
- Website: https://pypi.python.org/pypi/jusText
- Stars: 828 · Forks: 90
- Language: Python
- License: BSD-2-Clause
- Published: 2026-10-20 · Updated: 2026-10-20 · Language: en
- Canonical page: https://hysenlabs.com/projects/miso-belica-justext

## Boilerplate removal aimed at corpus building

jusText removes boilerplate content, such as navigation links, headers and footers, from HTML pages. The interesting part is not that it removes them, several libraries do that, but the stated goal of what should remain.

The tool is designed to preserve mainly text containing full sentences, and it is therefore well suited for creating linguistic resources such as web corpora. That framing explains the design. A readability extractor typically returns the biggest contiguous block of prose, which is fine for display and lossy for research. jusText works paragraph by paragraph and tries to keep the ones that look like sentences, so the output reads as a sequence of complete statements rather than as fragments torn out of a layout.

The audience is therefore people building text datasets: corpus creators, topic modellers, people training or evaluating classifiers on web text. The package carries a Development Status of Production/Stable classifier and is published on PyPI, with the homepage pointing at the PyPI project page.

The project is a fork. The original code was hosted on Google Code and is described as currently unmaintained, and the fork exists to keep the algorithm available and packaged for modern Python. The academic origin is documented too, with the acknowledgement of the Natural Language Processing Centre at Masaryk University in Brno, PRESEMT, Lexical Computing Ltd, and PhD research by Jan Pomikálek, who is also the listed author of the package.

## A stoplist decides what counts as boilerplate

The mechanism is a heuristic that works against a stoplist for the language of the page. The API makes that explicit: a stoplist is fetched for a language name, and the classification of each paragraph then depends on it.

```python
import requests
import justext

response = requests.get("http://planet.python.org/")
paragraphs = justext.justext(response.content, justext.get_stoplist("English"))
for paragraph in paragraphs:
  if not paragraph.is_boilerplate:
    print paragraph.text
```

Two things stand out in that snippet. The classification is exposed per paragraph as a boolean, is_boilerplate, so a caller can keep whatever it disagrees with, and the language is an explicit argument rather than something detected. There is no automatic language identification step in the documented usage, which means the stoplist you pass is the stoplist you get.

The stoplists themselves are shipped with the package. The setup file lists the package data as stoplists/*.txt inside the justext package, which is why a plain pip install gives you the language resources rather than an empty shell.

The consequence for a user is specific rather than general: the quality of the output for a given page depends on the quality of the stoplist for that language. A stoplist built for one kind of text, news say, may not classify a forum thread or a product page the way you want. The project documents English and Czech stoplists by example, and the algorithm description itself lives in a separate document in the repository under doc/algorithm.rst.

## Installing with pip and one parser dependency

Installation is a pip install, and the documentation is written for a period when that still needed a sudo:

```bash
$ [sudo] pip install justext
```

The square brackets are optional and the sudo is marked optional too. The stated prerequisites are Python 2.7 or 3.5 and later, plus pip, with separate notes for Windows and Linux installers. The single dependency is lxml, whose version depends on the Python version, and the setup file pins it more precisely as lxml[html_clean] with a minimum of 4.4.2. The html_clean extra matters more than it looks, because the library parses HTML with lxml and the cleaning helpers were moved out of lxml itself.

There is one dependency that exists for the oldest interpreters, a backport of functools lru_cache for Python versions below 3.2. That line is a leftover from the Python 2 era and tells you something about the age of the codebase.

Running the test suite is documented as a chain of pytest invocations across interpreters:

```bash
$ py.test-2.7 && py.test-3.5 && py.test-3.6 && py.test-3.7 && py.test-3.8 && py.test-3.9
```

The packaging is old in the same way. The setup file still imports from distutils when setuptools is unavailable, the classifiers list Python 2 and 2.7 alongside 3.5 through 3.9, and the repository has both setup.py and setup.cfg with a tasks.py alongside them. The fork has modernised the distribution, but the skeleton underneath is a project that started in the Python 2 era.

## Command line extraction with a language and an output file

The command line interface is the fastest way to see what the tool does, and the documented examples are a URL and a local file:

```bash
$ python -m justext -s Czech -o text.txt http://www.zdrojak.cz/clanky/automaticke-zabezpeceni/
$ python -m justext -s English -o plain_text.txt english_page.html
$ python -m justext --help # for more info
```

The shape is straightforward. The language is given with -s, the output file with -o, and the input is either a URL or a local HTML file. Running the module with the help flag is documented as the way to see the rest of the options.

The interesting part is that the same stoplist mechanism that the library uses is exposed on the command line, which means the language flag is not cosmetic. Passing a stoplist that does not match the page language produces a page where paragraphs have been classified against the wrong signal, and the failure is quiet: you get output, it is just the wrong output.

For a one-off page this is a small tool. For a corpus pipeline the Python API is the more realistic entry point, since it lets you keep the original HTML alongside the classification, filter by is_boilerplate yourself, and store the paragraph objects with their labels rather than a flattened text file.

## A fork, three ports and a list of alternatives

The state of the ecosystem around jusText is documented in the project itself, and it is worth reading before you choose it.

First, the origin. This is a fork of the original, currently unmaintained, code hosted on Google Code. The algorithm itself therefore predates the fork, and what the fork has maintained is the packaging, the tests and the compatibility.

Second, the same algorithm has been adapted to other languages: a C++ implementation, a Go implementation and a Java implementation. For a team that needs identical classification but cannot run Python, one of those ports is a more faithful choice than reimplementing the heuristic, and the algorithm document in the repository is the specification they would follow.

Third, the project lists the libraries that use jusText, including chirp, lazynlp, the off-topic memento toolkit, the PeARS orchard project, a readability calculator and sky. That is evidence of the algorithm being useful rather than of it being the best choice for a new project.

Finally, and most usefully for a decision, the project lists alternatives it describes as maintained as of January 2020: dragnet, html2text, inscriptis, newspaper, python-readability and trafilatura. The date matters, because that list was written years ago, and several of those projects have their own trajectories since. The reason jusText still holds a place in that list is its stated design goal, keeping full sentences and staying suitable for corpus construction, which is not what the general purpose extractors are optimising for.

## Release cadence and a permissive licence

The fork's own release record is the third data point. Version 3.0.0 was published on 2021-10-21, v3.0.1 on 2024-05-09 and v3.0.2 on 2025-02-25, and the last push to the repository was on 2026-08-18. The pattern is a modern major version followed by infrequent patches, which fits a project whose algorithm is stable and whose remaining work is compatibility and packaging rather than new features.

The licence is BSD 2-Clause, with the text in LICENSE.rst and the setup file describing it as The BSD 2-Clause License. That is a permissive licence with a short notice requirement, and it is a straightforward choice for a library that other projects build on. Your own counsel should confirm how it fits your distribution, but there is nothing unusual in the terms.

The package identity is worth noting as well. The distribution is named jusText, the maintainer is Michal Belica, the original author is Jan Pomikálek, and the project ships a web demo hosted by the NLP centre in Brno, which is the quickest way to see the classification on a real page before installing anything.

Taken together, the picture is a stable algorithm with a maintained distribution and a small, specific purpose. If that purpose is corpus building, it does that job well. If you need something else, the alternatives list is the faster route.

## Conclusion

jusText suits researchers and data engineers who need clean page text for corpora, linguistic resources or topic modelling, and who want a stoplist driven heuristic whose output is inspectable paragraph by paragraph rather than a blob of extracted text. It is the wrong choice if you just want readable text out of a page for a human, since several alternatives listed by the project do that more directly, and it is the wrong choice for a stack that cannot run Python, unless you use one of the C++, Go or Java ports of the same algorithm. Verify three things before you depend on it: that a stoplist exists for the language of the pages you are processing, that the paragraph classification matches what you consider boilerplate on a sample of your own pages, and that your Python version is covered by the packaging, which still advertises Python 2.7 and 3.5 to 3.9 in its classifiers.

## FAQ

### What is jusText used for?

It removes boilerplate such as navigation links, headers and footers from HTML pages while preserving text that contains full sentences, so the remaining output is suitable for creating linguistic resources such as web corpora.

### How do I install jusText?

The documented install is pip install justext, with Python 2.7 or 3.5 and later plus pip required. Its only dependency is lxml, which the setup file pins as lxml[html_clean] with a minimum version of 4.4.2.

### How do I run jusText from the command line?

Use the module form, for example python -m justext -s English -o plain_text.txt english_page.html, where -s selects the language stoplist, -o names the output file and the input can be a URL or a local HTML file. python -m justext --help lists the remaining options.

### How do I use jusText from Python and filter the result?

Call justext.justext with the page content and a stoplist obtained from justext.get_stoplist for a language, then iterate the returned paragraphs and keep the ones whose is_boilerplate attribute is false. Each paragraph carries the classification, so you can override any decision.

### Which stoplists does jusText ship?

The stoplists are packaged with the library as text files under the stoplists directory inside the justext package, and they are selected by language name through get_stoplist. The documentation examples use English and Czech.

### Is jusText still maintained?

The Python project is a fork of the original, currently unmaintained, Google Code code. Releases include v3.0.0 in October 2021, v3.0.1 in May 2024 and v3.0.2 in February 2025, and the last push was on 2026-08-18.

## Sources

- [License: BSD-2-Clause](https://github.com/miso-belica/jusText/blob/main/LICENSE)
- [miso-belica/jusText on GitHub](https://github.com/miso-belica/jusText)
- [Project website](https://pypi.python.org/pypi/jusText)
- [README](https://github.com/miso-belica/jusText/blob/main/README.md)
- [Releases](https://github.com/miso-belica/jusText/releases)

---

Hysen Labs editorial analysis, written from the project's own repository and release notes. Cite the canonical page: https://hysenlabs.com/projects/miso-belica-justext