clean-text: a Python preprocessing library for scraped and user-generated text

The problem clean-text targets: dirty scraped text

Web and social media text arrives with encoding artifacts, escaped unicode, mixed scripts, HTML entities and URLs embedded in the middle of sentences. The README frames the package around exactly that case: "Preprocess your scraped data with clean-text to create a normalized text representation." The example it gives turns a string containing escaped quotes, a markdown link and a mangled accented phrase into a lowercased, URL-tokenized version.

The audience is narrow and specific. This is for engineers who already have a corpus of user-generated content and need a deterministic normalization step before tokenization or embedding. It is not a general text utility for writers, and it is not a document converter. The package sits in the same slot as a preprocessing function you would otherwise write yourself, except that the unicode repair and the regex rules are already written and tested.

How clean-text works: ftfy, unidecode and hand-crafted regex

The README states the mechanism plainly: clean-text uses ftfy, unidecode and "numerous hand-crafted rules, i.e., RegEx." There is no model, no training step and no configuration file. The clean() function takes a string plus roughly two dozen keyword arguments, and each argument toggles one transformation.

The transformations fall into groups. ftfy handles unicode errors when fix_unicode is on. Transliteration to ASCII is controlled by to_ascii, and its quality depends on whether unidecode is installed. Lowercasing, line-break stripping and punctuation removal are simple string operations. The replace_with_* arguments substitute classes of content with tokens: URLs become <URL>, emails become <EMAIL>, numbers become <NUMBER>, and so on. The README lists the defaults in one code block, and those defaults matter: no_urls, no_emails, no_phone_numbers and the rest all default to False, so a bare clean("some input") mostly fixes unicode, transliterates and lowercases.

Because the order of operations is fixed inside the library, you cannot reorder the passes. If you need punctuation removed before URL detection, or the reverse, the package does not expose that control.

Installing clean-text and running a first clean

The package installs from PyPI and is named clean-text, not cleantext. The README calls this out explicitly because the import name differs from the distribution name.

pip install clean-text

The plain install omits unidecode, which is GPL-licensed. To get the better transliteration mapping, install the gpl extra instead.

pip install clean-text[gpl]

With the package installed, the smallest useful call passes the text and the flags you actually want. The README's own example enables URL replacement and lowercasing.

from cleantext import clean

clean("some input",
    fix_unicode=True,
    to_ascii=True,
    lower=True,
    no_urls=True,
    lang="en"
)

The lang argument is set to "en" in the README's default block, and the README notes that setting it to "de" enables German special handling. If you are processing a batch rather than one string, the package exposes clean_texts(), which accepts the same keyword arguments and an n_jobs parameter for multiprocessing.

from cleantext import clean_texts

clean_texts(["text one", "text two", "text three"], n_jobs=-1, no_urls=True)

Per the README, n_jobs=1 or None runs sequentially, -1 uses all available CPU cores, -2 uses all but one, and 0 raises ValueError. For scikit-learn pipelines there is a separate extra and a transformer class.

pip install clean-text[sklearn]

from cleantext.sklearn import CleanTransformer

cleaner = CleanTransformer(no_punct=False, lower=False)
cleaner.transform(['Happily clean your text!', 'Another Input'])

The unidecode split is a real fork in output, not a detail

The README is unusually direct about this: "There are inconsistencies between processing text with or without unidecode." That sentence should be read as a warning, because it means the same input can produce different output depending on which extra was installed. If unidecode is absent, clean-text falls back to Python's unicodedata.normalize for transliteration. The README's own assessment is that unidecode's mapping is superior while unicodedata's are sufficient.

The practical consequence is that a corpus normalized on one machine may not match a corpus normalized on another. If you build an index or train a model on clean-text output, the presence or absence of a GPL dependency becomes part of your data provenance. The README also suggests disabling transliteration altogether depending on your data and use case, which is the safer choice when downstream consumers can handle non-ASCII text.

The second limitation is language coverage. The README states that only English and German are fully supported, and that the package "should work for the majority of western languages." For anything else, the special handling is simply not there, and you would be relying on the generic regex passes.

Preserving patterns with the exceptions argument

One of the more interesting additions in recent versions is the exceptions parameter, which takes a list of regex pattern strings and preserves every match verbatim. The README is precise about what verbatim means here: matches are "not lowered, not transliterated, exactly as they appeared in the input." That is a stronger guarantee than a simple skip, and it makes the feature useful for domain terms that would otherwise be mangled.

The README's examples show the pattern.

from cleantext import clean

clean("drive-thru and text---cleaning", no_punct=True, exceptions=["drive-thru"])
# => 'drive-thru and textcleaning'

clean("drive-thru and pick-up", no_punct=True, exceptions=[r"\w+-\w+"])
# => 'drive-thru and pick-up'

Because exceptions are regexes, they compose. The README combines a hyphenated-word pattern with a currency pattern in one call. The trade-off is that you now own those patterns. A loose regex will protect text you wanted cleaned, and the library gives no report of what it skipped.

Where clean-text is the wrong tool

clean-text is a rule engine, so it does not learn from your data. If your normalization problem is context-dependent, for example deciding whether a token is a product name or a typo, hand-written regexes will not get you there. There is no probabilistic component and no way to train one from the package itself.

It is also not a scraper, a parser or a language detector. It assumes you already have text in a Python string. The README's framing is preprocessing, and the package stays inside that boundary.

Finally, the ordering constraint matters in practice. Every pass runs in a fixed sequence, and the flags can interact. Removing punctuation while also replacing currency symbols, for instance, means the currency pattern has to survive whichever pass runs first. The README's exceptions example handles this by protecting the currency pattern explicitly, which is a workaround rather than a configuration option. If your pipeline needs fine-grained control over pass order, you will end up writing your own cleaning function and using clean-text only for the ftfy portion.

Alternatives and how they differ in approach

The most direct comparison is ftfy alone. clean-text depends on ftfy for unicode repair, so if unicode fixing is your only problem, installing ftfy directly gives you the same repair step without the regex layer, the token replacement flags or the language handling. The difference is scope: ftfy targets mojibake and encoding damage specifically, while clean-text wraps that plus transliteration, lowercasing and content-class replacement.

A second comparison is writing the preprocessing yourself. Most of what clean-text exposes as flags is a handful of regex substitutions and a lowercase call. The value the package adds is that the rules are written, tested and documented in one place, and that the token conventions (<URL>, <EMAIL>, <NUMBER>) are consistent. If you need a token convention that differs, or a pass order that differs, the package's fixed pipeline becomes friction rather than help.

For scikit-learn users, the CleanTransformer class is the integration point, and it accepts the same parameters as clean(). That is a genuine convenience over wrapping a function in FunctionTransformer yourself, though the underlying behavior is the same.

Maintenance, licensing and upgrade cost

The repository is not archived, and the last push was on 2026-05-15. The most recent release listed is v0.7.1, published on 2026-01-28, alongside v0.7.0 and 0.6.0 on the same date. The project is developed with Poetry, and the README points contributors to RELEASING.md for the publish process. There is a CHANGELOG.md at the repository root, which is where version-to-version behavior changes would be recorded.

On licensing: the package itself is Apache-2.0, per pyproject.toml and the classifier list. The optional unidecode dependency is described in the README as GPL-licensed, and the gpl extra exists specifically to let you opt in or out. The sklearn extra pulls in pandas, scikit-learn and scipy. If you install clean-text[gpl], you are combining an Apache-2.0 package with a GPL dependency, and whether that is acceptable depends on how you distribute your own software. This is a factual description of the dependency graph, not legal advice; if the distinction matters to your organization, that is a question for whoever handles licensing.

Upgrade cost is low in the ordinary case, because the public surface is a function with keyword arguments and a transformer class. The risk sits in behavior changes to the regex rules, which would alter output without changing your call site. The CHANGELOG is the place to check before bumping the version.

Editorial conclusion

Adopt clean-text if you are preprocessing English or German user-generated text and want a small set of documented flags rather than a pipeline framework. Do not adopt it for languages outside that range or for tasks where you need a learned normalizer. Before committing, verify three things: whether your project can accept the GPL-licensed unidecode extra or must run without it, whether the ASCII transliteration it performs is acceptable for your downstream model, and whether the exception regexes you need actually survive the no_punct pass.

Frequently asked questions

Official sources

1. Issues
2. jfilter/clean-text on GitHub
3. README
4. Releases