# WeTextProcessing converts written Chinese and English into speakable form, and back

> WeTextProcessing is the text normalization and inverse text normalization toolkit from the WeNet project: it turns written forms such as 2.5平方电线 into spoken ones, and spoken forms such as 二点五平方电线 back into digits, using weighted finite state machines built on Pynini. The Python package is the practical entry point, the C++ runtime in the repository is the deployment form, and the two are not interchangeable.

**wenet-e2e/WeTextProcessing** — Text Normalization & Inverse Text Normalization

- Repository: https://github.com/wenet-e2e/WeTextProcessing
- Stars: 832 · Forks: 117
- Language: Python
- License: Apache-2.0
- Published: 2026-10-20 · Updated: 2026-10-20 · Language: en
- Canonical page: https://hysenlabs.com/projects/wenet-e2e-wetextprocessing

## Two directions of the same problem

WeTextProcessing is a toolkit for two related jobs, and it is worth separating them before looking at the code, because they are used in different places in a speech system.

Text normalization, which the project abbreviates to TN, is the direction from written text to spoken form. The README's own example is a length of electrical cable: 2.5平方电线 becomes 2.5 square metres of wire in words, because a number written with a decimal point is not how a person says it.

Inverse text normalization, ITN, is the opposite direction, from spoken form back to written form: 二点五平方电线 becomes 2.5平方电线, so a number spoken aloud can be displayed as a number.

The implementation underneath is finite state machines. The README acknowledges the foundational libraries, OpenFst and Pynini, and the Python package depends on pynini at 2.1.6 or newer, which is the grammar and FST layer that does the actual matching. The rule data itself is shipped as TSV files, and the pyproject lists them for Chinese, English and Japanese under tn and itn.

The project is from the WeNet organisation, the same group behind the WeNet speech toolkits, and the README links the Chinese-language introduction article as a must read document, with a separate WeChat account and group for Chinese users, and GitHub issues for everyone else.

The licence is Apache-2.0, the last push to the master branch was on 2026-07-29, and the published releases are 1.0.4 in August 2024, 1.0.5 in February 2026 and 1.2.0 in June 2026.

## Installing it and running the two console commands

The install is a single pip command:

```bash
# install
pip install WeTextProcessing
```

Two console scripts are declared in the pyproject, wetn pointing at tn.main:main and weitn at itn.main:main, and they are the fastest way to see what the toolkit does:

```bash
wetn --text "2.5平方电线"
weitn --text "二点五平方电线"
```

The first verbalises the written form, the second converts the spoken form back. The same two pipelines are available as modules, python -m tn and python -m itn, which is what the advanced usage section uses.

Both commands accept --file PATH, and if neither --text nor --file is given they read UTF-8 text from standard input. The README is specific about what that input mode produces: each input line produces two output lines, the tagged representation followed by the verbalized result, and only the input line ending is removed, so leading, trailing and internal spaces are preserved.

That last detail is the kind of thing that matters in a pipeline. Text normalisation is usually one stage in a larger flow, and a stage that silently trims your input changes what the next stage sees.

The Python API mirrors the console commands, with a module per language and direction:

```py
from itn.chinese.inverse_normalizer import InverseNormalizer
from itn.english.inverse_normalizer import InverseNormalizer as EnInverseNormalizer
from tn.chinese.normalizer import Normalizer as ZhNormalizer
from tn.english.normalizer import Normalizer as EnNormalizer
```

And the same call twice, once per direction, with the option that changes the behaviour:

```py
zh_tn_model = ZhNormalizer(remove_erhua=True)
zh_itn_model = InverseNormalizer(enable_0_to_9=False)
```

remove_erhua controls whether the Chinese erhua, the 儿 sound, is stripped, and enable_0_to_9 controls whether standalone small numbers are converted.

## The FST cache, and why overwrite_cache exists

The most operationally interesting thing in the library is the cache, and the README explains it in a sentence that saves a lot of confusion.

FST graphs are cached automatically outside the source tree. The cache records the build parameters and a fingerprint of the rule data, and when the configuration or the rules change the graphs are rebuilt automatically. For daily use you do not need overwrite_cache, and you should only set it to True when you want an unconditional rebuild.

That is a well-behaved cache design. The fingerprint means a rule edit is noticed, and the cache living outside the source tree means the compiled artefacts do not dirty a working copy.

The cache location is configurable. cache_dir set to a path chooses another directory, cache_dir=False disables persistent caching entirely, and overwrite_cache=True forces a rebuild:

```py
# tn usage
>>> from tn.chinese.normalizer import Normalizer
>>> normalizer = Normalizer(cache_dir="/path/to/wetext-cache")
>>> normalizer.normalize("2.5平方电线")
# itn usage
>>> from itn.chinese.inverse_normalizer import InverseNormalizer
>>> invnormalizer = InverseNormalizer(cache_dir="/path/to/wetext-cache")
>>> invnormalizer.normalize("二点五平方电线")
# Disable persistent caching:
>>> uncached = Normalizer(cache_dir=False)
```

The rebuild path matters most for contributors, because it is what turns the library into something you can fix. The advanced section says that if you want modifications and adapt tn or itn rules to fix bad cases, you should clone the repository, install the requirements, and install the pre-commit hooks, and then run the commands with overwrite_cache so the rules are rebuilt according to your modifications:

```bash
python -m tn --text "2.5平方电线" --overwrite_cache
python -m itn --text "二点五平方电线" --overwrite_cache
```

So the workflow for a team with domain vocabulary is: find the bad case, edit the rule file under tn/chinese/rules, rebuild, and ship the corrected rules. The version number is derived by setuptools_scm from the tags and written into tn/_version.py, so a fork that carries its own rules gets its own version.

## normalize_with_mapping and the spans that changed

The most useful API in the library is not the one that returns a string, it is the one that tells you what changed.

Calling normalize on a string gives you the output text. Calling normalize_with_mapping gives you the output plus a list of the spans that were rewritten:

```py
result = zh_tn_model.normalize_with_mapping("今天中午12点")
print(result.output_text)
# 今天中午十二点

for mapping in result.mappings:
    print(mapping.token_type, mapping.input_text, "=>", mapping.output_text)
# math 12 => 十二
```

The mapping carries a token_type, which identifies the matched grammar rule, so a caller can tell a number conversion from a unit expansion. The as_dict form gives the same information in a serialisable shape:

```py
print(result.as_dict())
# {
#   "input": "今天中午12点",
#   "output": "今天中午十二点",
#   "mappings": [{
#     "kind": "replace",
#     "token_type": "math",
#     "input": {"start": 4, "end": 6, "text": "12"},
#     "output": {"start": 4, "end": 6, "text": "十二"}
#   }]
# }
```

Two details in that output matter for anyone building on it. Offsets are Unicode character offsets, not byte offsets, which is the correct choice for text and a common source of off-by-one bugs in mixed Chinese and Latin content. And the include_identity option adds unchanged tokens to the mapping list, which is what you want when you need a full token alignment rather than only the edits.

The span data is what makes the toolkit debuggable. Without it, a bad normalization is a string difference you have to read and guess at; with it, you get the rule that fired and the exact characters it touched.

## N-best output and per language options

Both APIs support n-best output, and the distinction between the two is worth knowing before you write a caller:

```py
outputs = zh_tn_model.normalize("输入文本", nbest=3)
results = zh_tn_model.normalize_with_mapping("输入文本", nbest=3)
```

nbest=1 returns a single result, and nbest greater than one returns a list. So the same call returns a different type depending on the argument, which is convenient in a script and a trap in a typed codebase, since a caller that ignores the argument has to handle both shapes.

The console commands expose a different set of options, and the README is explicit that the available options are limited to the selected direction and language. The examples show that in practice:

```bash
python -m tn --language zh --no-remove-erhua --text "这儿"
python -m tn --language ja --transliterate --text "WeNet"
python -m itn --language zh --enable-0-to-9 --text "一二三"
python -m itn --language ja --full-to-half True --text "１２時"
```

Two details there are practical. The Japanese pipeline can transliterate, and it can convert full width characters to half width, which is the kind of normalisation a Japanese ASR front end needs. And boolean options accept both the switch style, --option and --no-option, and the legacy form with an explicit True or False value, so a script written for an older version of the package keeps working.

To see what a given pipeline supports, the README points to python -m tn --language LANG --help and python -m itn --language LANG --help, which show only the options for that pipeline. The installed wetn and weitn commands expose the same interfaces.

## The Python package and the C++ runtime are not the same artefact

The advanced usage section pairs two ideas that are easy to run together and important to keep apart: DIY your own rules, and deploy WeTextProcessing with the cpp runtime.

The repository has a runtime directory next to tn and itn, and the README is direct about the boundary: Python cache files are not C++ runtime exports. Rule contributors are pointed to the Python rule architecture guide in docs/python-rule-architecture.md.

That sentence decides an architecture question for a team deploying this. The Python package is a convenient way to run and to debug rules, with FST graphs cached locally and rebuilt when the rules change. The C++ runtime is the form you put in a service. If your production process is C++, the Python caches you get by experimenting locally are not what you ship, and the rule data has to be exported into the format the runtime expects.

The rule data itself is shipped as TSV files, and the pyproject lists the exact paths that end up in the package: for tn the Chinese, English and Japanese data directories with TSV files, and for itn the same three languages, with the English set also including .far files. The .far extension is worth noting, because it is not TSV, which tells you the rule format is not a single uniform table.

The pipeline documentation lives in tn/README.md and itn/README.md, and the package is built with setuptools and setuptools_scm, requires Python 3.7 or newer, and depends on pynini and importlib_resources.

## Conclusion

Adopt WeTextProcessing when your speech stack needs written text converted to a speakable form, or spoken text converted back to digits, in Chinese, English or Japanese, and you can live with rule files rather than a learned model. Do not adopt it as a general text cleaner, because the transformations are grammar driven and the rule set is the product. Before you build on it, run both directions on your own domain vocabulary, since the rules live in TSV files under tn and itn and bad cases are fixed by editing them, and read docs/python-rule-architecture.md, because the Python caches are not the same artefact as the C++ runtime exports.

## FAQ

### How do I install WeTextProcessing?

Run pip install WeTextProcessing. The package requires Python 3.7 or newer and depends on pynini 2.1.6 or newer and importlib_resources, and it registers the wetn and weitn console commands.

### What is the difference between wetn and weitn?

wetn is the text normalization entry point and weitn is the inverse one, so the first turns written text into a speakable form and the second turns spoken form back into written form. Both also accept --file PATH, and with neither --text nor --file they read UTF-8 text from standard input.

### How do I see which parts of a string WeTextProcessing changed?

Call normalize_with_mapping instead of normalize. It returns the output text, a list of mappings carrying token_type, input_text and output_text, and an as_dict form with the input, the output and the character spans; include_identity=True adds unchanged tokens, and the offsets are Unicode character offsets.

### How do I rebuild the FST cache after editing a rule?

Pass overwrite_cache=True to the normalizer, or run python -m tn --text ... --overwrite_cache. The cache is stored outside the source tree, cache_dir chooses its location and cache_dir=False disables persistent caching, and the cache records a fingerprint of the rules so it rebuilds automatically when they change.

## Sources

- [Issues](https://github.com/wenet-e2e/WeTextProcessing/issues)
- [License: Apache-2.0](https://github.com/wenet-e2e/WeTextProcessing/blob/master/LICENSE)
- [README](https://github.com/wenet-e2e/WeTextProcessing/blob/master/README.md)
- [Releases](https://github.com/wenet-e2e/WeTextProcessing/releases)
- [wenet-e2e/WeTextProcessing on GitHub](https://github.com/wenet-e2e/WeTextProcessing)

---

Hysen Labs editorial analysis, written from the project's own repository and release notes. Cite the canonical page: https://hysenlabs.com/projects/wenet-e2e-wetextprocessing