# charset-normalizer: detecting the encoding of bytes you did not choose > charset-normalizer is a pure Python detector for unknown text encodings, built as an alternative to chardet, supporting every IANA character set name for which Python ships a codec, plus custom codecs you register yourself. It is MIT licensed, roughly a quarter of a megabyte installed, and it can be compiled with Cython when you want the speed. **jawah/charset_normalizer** — Truly universal encoding detector in pure Python. - Repository: https://github.com/jawah/charset_normalizer - Website: https://charset-normalizer.readthedocs.io/en/latest/ - Stars: 804 · Forks: 79 - Language: Python - License: MIT - Published: 2026-10-20 · Updated: 2026-10-20 · Language: en - Canonical page: https://hysenlabs.com/projects/jawah-charset-normalizer ## Reading text from an unknown encoding charset-normalizer is a library that helps you read text from an unknown charset encoding, and the project description calls it a truly universal encoding detector in pure Python. The project is motivated by chardet, and the README says the author is trying to resolve the issue by taking a new approach. The scope is precise. It detects the encoding of bytes, so that you can decode them, and it does so using the codecs that already exist rather than by shipping a separate statistical model for every encoding. The README states that all IANA character set names for which the Python core library provides codecs are supported, and that you can also register your own set of codecs, and that it would work as-is. That last point is the one that separates this from a typical detector. A detector that only recognises encodings it has been taught to recognise is a closed list, and any input outside that list gets a guess or an error. Here, the set of encodings is the set of codecs in your Python installation, plus whatever you add. An unusual legacy encoding that some other package registers as a codec can be registered here too. The distribution is on PyPI under the name charset-normalizer, and the documentation is hosted on readthedocs. The README carries no install command, and the project is a package rather than an application, so the package index and the documentation site are where the project sends you to start. The package is MIT licensed, and the metadata describes it as the real first universal charset detector and an open, modern, actively maintained alternative to chardet, with a development status of Production/Stable. ## The comparison table, and the footnote it carries The README puts the project next to its alternatives in a table, and the table is more informative than the marketing copy because it marks the areas where the project does not claim an advantage. Chardet and cChardet are listed as fast, and charset-normalizer is listed as fast too. On the specific question of speed on large content, chardet and cChardet are marked as failing and charset-normalizer as passing, which is the uncapped claim in the table. Universality is marked the same way, present only for charset-normalizer. Reliability is marked as present for all three, both without distinguishable standards and with them, which is an admission that the detector has the same fundamental difficulty as its competitors when the bytes carry no signal about their own encoding. Licence and packaging are the other differentiators in the table. Chardet is listed with a 0BSD licence marked as disputed, charset-normalizer with MIT, and cChardet with MPL-1.1 marked as restrictive. On wheel size, charset-normalizer is listed at roughly 250 kB against roughly 1200 kB for chardet and roughly 200 kB for cChardet, and cChardet is marked as not native Python, which matters for wheels and for free threaded builds. Supported encodings are listed as 99 for both chardet and charset-normalizer, and 40 for cChardet, and only charset-normalizer is marked as able to register a custom encoding. The footnote attached to the disputed licence marker is the most pointed text in the README. It states that Chardet 7 replaced the historical LGPL licensed implementation with an AI assisted rewrite, initially distributed under MIT and later under 0BSD, and that the original author contests that the rewrite was sufficiently independent to permit relicensing, while Chardet's maintainer holds that it is a new, non derivative implementation. A separate discussion raises questions about copyright ownership and licensing of substantially AI generated code, and the footnote says explicitly that neither unresolved question is presented as settled law. The ethical part of that footnote goes further than licensing. It argues that independent implementations are essential to open source competition, and that responsible use of AI in open source requires more than producing text that differs from the historical source: transparent provenance, respect for project lineage, meaningful attribution, accountable human review, and consideration for the social agreement under which earlier contributors participated. That is a position statement from one project, not a ruling, and it is worth reading as the maintainer's stated reason for taking the approach this library takes. ## How the approach differs from a statistical detector The footnote explains the difference in one sentence, and it is worth quoting: chardet is clearly using specific code for a specific encoding even if covering most of them. That is the design distinction. A statistical detector works out what the bytes probably are and then hands you an encoding name, and the decoding step afterwards is your problem, and if the guess is wrong you get a UnicodeDecodeError partway through a document. charset-normalizer instead leans on the decoders themselves: it can try to decode with the codecs it has, and it lists UnicodeDecodeError safety as a feature, which is the property that matters when a single mis-detected byte turns a document into an exception rather than into a wrong character. The README also lists the ability to detect spoken language among the features, alongside the encoding work, which is a different problem that the same machinery of character distribution analysis can serve. What the README does not do is document the function names, the module layout, or the calling convention. There is no usage example in the file, and the API documentation lives on the readthedocs site rather than in the repository. If you are evaluating this library against a specific call pattern in your own code, that documentation is the place to look, because the repository itself will not tell you what the entry point is called. The package layout does hint at the structure. The source tree has a charset_normalizer package, and the build configuration compiles two Cython modules from .pyx sources, named md and cd, which suggests a content detector and a character distribution detector as the two pieces of the implementation, with the pure Python versions as the fallback when the extension is not built. ## Cython acceleration is opt-in through two environment variables The pure Python claim is the default, and the optional speed-up is controlled by environment variables read at build time in setup.py. ```python USE_CYTHON = os.getenv("CHARSET_NORMALIZER_USE_CYTHON") == "1" LIMITED_API = os.getenv("CHARSET_NORMALIZER_CYTHON_ABI3") == "1" ``` Setting the first variable to 1 makes the build attempt to cythonize the two extension modules, and the code handles the case where Cython is not installed by falling back to a None value and carrying on without extensions, so a build without a compiler still works. The second variable is the more interesting one, because it selects the limited API. When it is enabled, the extension is compiled against Py_LIMITED_API 0x03070000 with the PEP 489 multi phase initialisation macro defined, and the wheel is built with the py_limited_api option set to cp37. On non Windows platforms an extra compiler flag of -g0 is added to strip debug information. The result of that combination is a single wheel that works across CPython versions, which is the point of the limited API: a project that supports Python 3.7 and 3.15 does not want to ship a separate binary wheel per version, and an abi3 wheel is the standard way out of that. The declared supported range in the metadata starts at 3.7 and the classifiers list CPython and PyPy, a free threading resilient marker, and WebAssembly with Emscripten, so the build configuration is doing real work to keep that matrix honest. The compiler directives passed to cythonize turn off bounds checking, cdivision, and the other safety checks, which is the usual trade for a numeric style module, and it tells you that the maintainer treats the extension as a performance path rather than as a separate implementation. ## Packaging that a plain setuptools project would not do The build configuration is unusual enough to be worth reading, because it is the part of the project that most affects whether you can install it at all in your environment. The pyproject file declares a setuptools build system with a required version of setuptools 68 or later, and instead of the usual build-backend setting it names a backend of backend with a backend-path of _build_hook, pointing at a directory in the repository. That is a custom in-tree build backend, and the _build_hook directory at the top level is where it lives. ```toml [build-system] requires = ["setuptools>=68,<84.1"] build-backend = "backend" backend-path = ["_build_hook"] ``` The reason for the custom hook is not documented in the files available here, so the honest description is that the project ships its own build backend and that you should expect the build to be more than a stock setuptools run. The project metadata is otherwise thorough. The name is charset-normalizer, the licence is MIT, the keywords cover encoding, charset, detector, normalization, unicode, and chardet, and the author and maintainer are the same person. The supported versions run from Python 3.7 to 3.15, the requires-python is 3.7 or newer, and the version and readme are declared as dynamic, which is the usual arrangement when they are filled in at build time rather than written by hand. There is an optional dependency group called unicode_backport with nothing in it, which is the sort of placeholder that exists so that a downstream requirement can be expressed without changing the package. The development tooling is nox based, with the CI dependency group pinning different nox versions per Python version, and the repository also carries a noxfile.py, a pre-commit configuration, a readthedocs configuration, an osv-scanner configuration for vulnerability scanning of dependencies, and separate requirement files for CI and development. ## A steady release line and an active repository The project is not archived and the last push was on 2026-10-20, so the code is being worked on daily. The release line moves at a moderate pace, with 3.5.0 published on 2026-08-12, 3.5.1 on 2026-08-15, and 3.5.2 on 2026-09-29, and the version numbers follow the standard three part scheme with a patch that arrives days after a minor. The repository carries the files you would expect from a project with that level of process: a CHANGELOG.md, a CONTRIBUTING.md, a CODE_OF_CONDUCT.md, a SECURITY.md, a MANIFEST.in, coverage configuration, and a data directory that presumably holds the data the detector uses. On the licence question, the answer is clean for this project. charset-normalizer is MIT, with a LICENSE file at the top of the repository, and the pyproject metadata says the same. The licence dispute the README raises is about a different project, and it is worth being clear about that: nothing in this repository is the subject of the unresolved questions the footnote describes, and the note is there as a statement of the maintainer's position rather than as a claim about this package. What a user should take from the README is a comparison and a warning. The comparison says this detector is universal in the sense that it can use any codec Python provides, fast on large uncapped content, and smaller than chardet while being native Python, and that it is equally reliable in the cases where reliability is even possible. The warning says the difference in approach, using the decoders instead of a per encoding statistical model, is a deliberate stance in a field where a competing project had recently changed its licence and its provenance, and that stance is part of why this project exists. ## Conclusion charset-normalizer fits a Python service that ingests text from sources it does not control, such as uploaded files, scraped pages, or feeds in mixed encodings, and that wants a detector which can also use the decoders Python already ships instead of maintaining its own statistical models per encoding. It does not fit a project that has to distinguish encodings with no distinguishing standards, where the README's comparison table is candid that reliability is comparable rather than better, and it does not fit a workflow that depends on a stable, versioned API of the chardet kind. Verify first that the encoding you expect is one of the supported IANA names, that the raw bytes you pass in are genuinely undecodable by your own guess before you trust the detector, and that your build path works, since the Cython acceleration is opt-in through environment variables and is not the default. ## FAQ ### What is charset-normalizer and how is it different from chardet? It is a pure Python detector for the encoding of unknown text, an alternative to chardet that takes a different approach, using the codecs Python already provides rather than a separate statistical implementation per encoding. The README's comparison table marks it as universal, fast on large uncapped content, and able to register custom encodings. ### Which encodings does charset-normalizer support? All IANA character set names for which the Python core library provides codecs, which the comparison table counts as 99, the same number as chardet. You can also register your own set of codecs, and the README says that works as-is. ### How do I build charset-normalizer with the Cython extension? The setup script reads two environment variables at build time, CHARSET_NORMALIZER_USE_CYTHON and CHARSET_NORMALIZER_CYTHON_ABI3. Setting the first to 1 builds the md and cd extension modules, and the second selects the limited API so a single abi3 wheel covers the supported Python versions, with the pure Python implementation as the default when the extension is not built. ### What Python versions does charset-normalizer support? The metadata declares requires-python of 3.7 and newer, and the classifiers list Python 3.7 through 3.15, with CPython and PyPy, a free threading resilient marker, and WebAssembly with Emscripten. The project status is listed as Production/Stable. ### Is charset-normalizer actively maintained? The repository is not archived, the last push was on 2026-10-20, and releases 3.5.0, 3.5.1, and 3.5.2 were published between 2026-08-12 and 2026-09-29. The package is MIT licensed and the project description calls it an actively maintained alternative to chardet. ## Sources - [jawah/charset_normalizer on GitHub](https://github.com/jawah/charset_normalizer) - [License: MIT](https://github.com/jawah/charset_normalizer/blob/master/LICENSE) - [Project website](https://charset-normalizer.readthedocs.io/en/latest/) - [README](https://github.com/jawah/charset_normalizer/blob/master/README.md) - [Releases](https://github.com/jawah/charset_normalizer/releases) --- Hysen Labs editorial analysis, written from the project's own repository and release notes. Cite the canonical page: https://hysenlabs.com/projects/jawah-charset-normalizer