PyThaiNLP: Thai NLP in Python, with corpora that download themselves

A Thai focused counterpart to NLTK

The README describes PyThaiNLP as designed to be a Thai-focused counterpart to NLTK, providing standard tools for linguistic analysis under an Apache-2.0 license, with its data and models covered by CC0-1.0 and CC-BY-4.0.

That three-way licensing statement is worth pausing on, because it is a pattern more projects use than they should. The code is Apache-2.0, the data is public domain under CC0, and the models carry attribution under CC-BY-4.0. A project that ships trained models has to say what may be done with them, and PyThaiNLP separates the three cases so that a commercial user can check which part applies to what they are actually using.

The feature list is broad for a language specific library. Linguistic units come first, with sentence, word and subword segmentation exposed as sent_tokenize, word_tokenize and subword_tokenize. Then tagging with pos_tag, transliteration and IPA conversion, spelling suggestion and correction through spell and correct, and a group of utilities that are specific to the language and the country: Soundex, collation, number to text through bahttext, Thai date formatting through thai_strftime, and keyboard layout correction, which is the kind of utility that only makes sense for users typing on a keyboard laid out for a different script.

The data section lists built-in Thai character sets, word lists and stop words, and there is a command line interface called thainlp, with commands for listing datasets and showing usage.

The package supports Python 3.9 and later, the classifiers list 3.9 through 3.14, and the project status badge says Active. The current release is 5.4.0, published on 2026-10-09, with 5.3.8 in September and 5.3.6 in August, and the last push was on 2026-10-15, so the release cadence and the commit activity agree with each other.

The extras decide what you actually install

The installation section is short, and then it gets interesting, because the same library serves very different needs depending on which optional dependencies you pull in.

pip install pythainlp

Or with extras:

pip install "pythainlp[extra1,extra2,...]"

The extras listed are compact, described as installing a stable and small subset of dependencies and marked recommended. translate is machine translation support, wordnet is WordNet support, and full installs all optional dependencies, with the warning that it may introduce conflicts.

That warning is the interesting part, and it is the reason compact is the recommended option. A library that offers translation, WordNet, tokenization, tagging, transliteration and correction as one installable package has a wide dependency surface, and the full extra is where the conflicts would appear. The README points at the full list of extras in the documentation and at the project.optional-dependencies section of pyproject.toml for the specific libraries in each one, which is the right place to look if you need to know what a given extra drags in.

The build backend is hatchling, and the package metadata lists nine authors and two maintainers, with the project description of Thai Natural Language Processing library.

The development environment mirrors the extras choice. The Dockerfile builds on python:3.12, installs build-essential, libicu-dev, python3-pip, python3-venv and pkg-config, creates a virtual environment at /opt/venv, and then installs the package in editable mode with the full extra, pip install -e ".[full]". The compose file mounts the working directory into the container, sets the working directory to it, and points the data directory at a path inside it, PYTHAINLP_DATA=/workspace/pythainlp-data, so the downloaded corpora survive the container.

The corpus download is the part that surprises people

The data model is the thing to understand before you install anything, because it means the library writes to your disk and reaches the network on first use.

PyThaiNLP downloads data to ~/pythainlp-data by default, and the catalogue of what is available is a data catalog called db.json, kept in a separate repository, PyThaiNLP/pythainlp-corpus. The location is overridable with the PYTHAINLP_DATA environment variable.

The environment variable table in the README is the reference for all of this, and it is unusually well annotated, with a status column that says Current or Deprecated for each one.

PYTHAINLP_DATA is the current variable for the data directory, with a default of ~/pythainlp-data. PYTHAINLP_DATA_DIR is a legacy alias for it, which emits a DeprecationWarning, and setting both raises a ValueError. PYTHAINLP_OFFLINE, set to 1, disables automatic corpus downloads. PYTHAINLP_READ_ONLY, set to 1, enables read-only mode, which prevents implicit background writes to the internal data directory, that is corpus downloads, catalog updates and directory creation. PYTHAINLP_READ_MODE is the deprecated legacy alias for that, with the same warning and the same ValueError when both are set.

The two current modes are not the same thing, and the README spells out the difference. Offline mode blocks only automatic downloads triggered by get_corpus_path, so an explicit pythainlp.corpus.download() still works, and when a corpus is not already cached locally a FileNotFoundError is raised instead of a network attempt. Read-only mode is more restrictive and also blocks explicit download() calls, because any download requires writing to the data directory. The stated use for read-only mode is a data directory on a read-only filesystem, such as a read-only Docker volume or a shared cluster mount, and that is exactly the situation the docker-compose.yml in the repository sets up.

The read only mode exists for containers and cluster mounts

The two modes together are the part of the design that shows the library has been used in production environments rather than only on a laptop.

import pythainlp
print(pythainlp.is_offline_mode())  # True if PYTHAINLP_OFFLINE=1

The offline check is a public function, so an application can assert on its state rather than guessing, and the same pattern applies to the read-only mode.

The motivation for read-only mode is stated as a set of side effects the user may not be aware of. The library blocks corpus downloads, catalog updates and automatic data directory creation, and the operations where the user explicitly specifies an output path are unaffected, with model.save("path"), tagger.train(..., save_loc="path") and the misspelling corrector's save all named in the README as examples.

That distinction is the design. A read-only mode that blocked explicit saves would be useless, since saving a trained model is the whole point of training one. What it blocks is the background writing, the part that happens as a side effect of calling a function that looks like a pure query.

For distributed computing the README gives a specific instruction. When using PyThaiNLP in distributed computing environments, for example Apache Spark, set the PYTHAINLP_DATA environment variable inside the function that will be distributed to worker nodes. That is a detail which bites people, because a variable set on the driver is not inherited by the workers, and a worker that silently downloads its own corpora at task time is a performance and reproducibility problem. The repository carries an example for exactly this, examples/distributed_pyspark.py.

There is also a Dockerfile and a compose file, a Makefile whose targets run tox for linting across ruff, flake8 and mypy, a core test run with python -m unittest tests.core, a markdown lint through npx markdownlint-cli2, and a coverage target. The repository also has fuzz/ and .clusterfuzzlite/ directories, which is a level of testing investment worth noticing in a library that parses user supplied text.

Where the tool stops and your own code starts

The honest limits here are about scope and about Thai itself.

The first is that this is a Thai language resource library. The tokenizers, dictionaries, stop words, transliteration tables and the bahttext and thai_strftime utilities are all specific to Thai and to Thailand. For English or multilingual text, the general purpose toolkits do the job, and this repository says as much by positioning itself as a counterpart to NLTK rather than a replacement. The difference in approach is that PyThaiNLP ships the linguistic resources that a general toolkit leaves to you, which is exactly what you need for Thai and exactly what you do not need otherwise.

The second is the dependency breadth of the optional features. Machine translation support in particular pulls in a different world, and the README's own warning that the full extra may introduce conflicts is the reason to start with compact and add extras one at a time.

The third is the corpus model itself. Data is fetched from the catalogue at runtime, which means a first run needs network access, a machine with a writable home directory, and a data directory that is shared or pre-populated if you have many workers. The environment variables exist to make that controllable, but the control is opt-in, and a deployment that forgets them will attempt downloads from inside a container that has no route out.

The fourth is model provenance. The data and models are licensed separately from the code, and the project keeps model cards in its wiki, which is the place to look before you ship a trained tagger or corrector into a product.

The last is the release branch structure. There is a 5.4.0 release and a dev branch, each with its own changelog and its own documentation tree, pythainlp.org/docs and pythainlp.org/dev-docs, so which branch you install decides which documentation you should be reading.

Editorial conclusion

Adopt PyThaiNLP if your text is Thai and you want tokenization, tagging, transliteration and correction in Python without assembling a Thai specific pipeline yourself. Do not adopt it for multilingual or English text, where the resources and dictionaries are not aimed at you, and do not adopt it in an environment without a writable data directory unless you set the read only variable and pre-provision the corpora. Verify first that the corpora you need are in the data catalog, that the extras you install match the features you call, and that the data directory is either writable or pre-populated before your workers start.

Frequently asked questions

Official sources

1. License: Apache-2.0
2. Project website
3. PyThaiNLP/pythainlp on GitHub
4. README
5. Releases