Sacremoses: the Moses tokenizer, truecaser and normalizer as a Python package

What Sacremoses is for, and who actually needs it

Sacremoses is a Python port of the Moses tokenizer, truecaser and normalizer. The people who need it are the ones already inside the Moses ecosystem: teams with parallel corpora that were tokenized with the Perl scripts in mosesdecoder, teams running a statistical or neural MT system whose preprocessing is defined by those rules, and anyone who has to reproduce an older pipeline without installing Perl and a checkout of mosesdecoder. The README presents the package as a drop-in replacement for those scripts, and the repository layout backs that up: the Unicode character classes and nonbreaking prefixes live in generated Python modules rather than being read from text files at run time.

The package is not a tokenizer in the spaCy or NLTK sense. It does not produce a tokenization designed for a particular downstream model, and it makes no claim about linguistic correctness. It reproduces a specific, historically important set of rules, and that is the whole value proposition. If your data never touched Moses, the rules are an arbitrary constraint rather than a compatibility requirement.

The MIT licence removes most of the friction. The setup.py carries an OS Independent classifier and Python 3.9 through 3.14 classifiers, so there is no platform-specific build step to plan around.

How the tokenizer, truecaser and normalizer are put together

The package exposes four classes: MosesTokenizer, MosesDetokenizer, MosesTruecaser and MosesPunctNormalizer. The tokenizer takes a lang argument and applies language-specific rules; the README's example uses lang='en'. When return_str is true, tokenize returns a single string, and when it is false it returns a list of tokens.

The tokenizer escapes XML-special characters by default. In the README's second example, the sentence contains a pipe, square brackets, angle brackets and an ampersand, and the expected token list contains '|', '[', ']', '<', '>' and '&' in their place. Apostrophes inside words are split off as ''t' and ''s'. This is the behaviour that makes detokenization possible: MosesDetokenizer reverses the escaping, and the README shows the detokenized string coming back identical to the input.

The truecaser is trained, not pretrained. You feed it tokenized documents with train, or a file path with train_from_file, and it learns which words are capitalized in your corpus. The README's Sherlock Holmes example shows the difference between the default behaviour and use_known=True: without it, "THE ADVENTURES OF SHERLOCK HOLMES" becomes 'the adventures of Sherlock Holmes'; with it, only 'Sherlock' and 'Holmes' are restored and the rest stays uppercase. That flag is the switch between a full recasing and a partial one, and the README does not explain when each is appropriate, which is a gap for a first-time user.

The normalizer is the smallest piece. The README's only example passes an all-caps sentence with "AS-IS." through normalize and gets the same string back, which tells you the default configuration changes nothing in that case. The CLI help lists the options that do change things, including --normalize-quote-commas and --normalize-numbers, plus a --re option whose description is cut off in the README.

Installing Sacremoses with pip and running a first tokenization

The README gives one install command. It upgrades an existing installation if one is present.

pip install -U sacremoses

The README states that Sacremoses only supports Python 3 now, that sacremoses>=0.0.41 is the boundary, and that the current release requires Python 3.8 or newer. If you are on Python 2, the last possible version is sacremoses==0.0.40. The four runtime dependencies are listed in requirements.txt with floors: click>=8.0, joblib>=1.2.0, tqdm>=4.66.3 and regex>=2021.8.3.

A first real use is the tokenizer with return_str, which is the shortest path to seeing whether the rules match your expectations.

from sacremoses import MosesTokenizer
mt = MosesTokenizer(lang='en')
text = 'This, is a sentence with weird\xbb symbols\u2026 appearing everywhere\xbf'
print(mt.tokenize(text, return_str=True))

The README's expected output for that input is 'This , is a sentence with weird \xbb symbols \u2026 appearing everywhere \xbf'. Punctuation is separated by a space, the non-ASCII symbols are left in place, and the sentence is not otherwise altered. If you get that string, the install is working.

For a file-sized job, the CLI is the entry point. Since version 0.0.42 there is a pipeline feature, and the global options must come before the subcommand. The README's chained example pipes a file through normalize, tokenize and truecase in one pass.

cat big.txt | sacremoses -l en -j 4 \
    normalize -c tokenize -a truecase -a -m big.truemodel \
    > big.txt.norm.tok.true

The -l flag sets the language, -j sets the number of processes, normalize -c removes control characters, tokenize -a enables aggressive dash split rules, and truecase -a marks the model as being for ASR. The README states that if big.truemodel exists it is loaded with -m, and otherwise a model is trained and saved to that path. The available subcommands are detokenize, detruecase, normalize, tokenize, train-truecase and truecase.

Where the Moses rules get in the way

The default XML escaping is the first surprise. If you tokenize a sentence containing an ampersand and then feed the tokens to something that expects literal text, you get '&' rather than '&'. The detokenizer exists precisely to undo this, but any consumer sitting between the two steps has to know the convention. The README's own example makes the escaping visible rather than hiding it, which is honest, but it is also a design decision that leaks into everything downstream.

The truecaser has no pretrained model. Every deployment has to train one on domain text, and the quality of the result depends entirely on that text. The README shows train_from_file taking a plain text file and tokenizing it internally, but it does not say how much data is enough, what happens with a tiny corpus, or how the model behaves on words it never saw. For a small project without a large in-domain corpus, the truecaser is the part most likely to disappoint.

The version history is worth reading before you commit. The recent releases list shows 0.1.0 dated 2023-10-30, and the README references behaviour introduced in 0.0.42. The last push to the default branch was on 2026-09-07, so the repository has seen work recently even though the release cadence is slow. That combination, active commits with infrequent releases, means features can land on master well before they reach PyPI.

Finally, the CLI pipeline is genuinely sequential. The README's example chains three commands with pipes, which is efficient for throughput, but there is no documented way to inspect intermediate output without breaking the chain and running the commands separately. The README does not document rollback or a dry-run mode.

Sacremoses against a general-purpose tokenizer

The obvious alternative is a tokenizer built for modern NLP work, such as the rule-based tokenizers shipped with NLTK or the subword tokenizers used by transformer models. The difference is not quality, it is purpose. NLTK's tokenizers aim at linguistically plausible splits across many languages and are used for analysis and teaching. Sacremoses aims at bit-for-bit agreement with a specific Perl script, and its output is designed to be detokenized back to the original string.

That distinction matters in practice. If you are preparing data for a model that was trained on Moses-tokenized text, a general tokenizer will produce a different token stream and the model will see input it was not trained on. If you are building a search index or doing linguistic annotation, Moses rules will give you escaped entities and split contractions that you did not ask for. The README's round-trip example, where detokenize reproduces the input exactly, is the property that a subword tokenizer does not offer in the same form.

There is also a tooling difference. Sacremoses is a pure Python package with four dependencies and a click CLI, so it installs with pip and runs anywhere Python runs. The original Moses scripts require Perl and a checkout of mosesdecoder, which is the friction this port exists to remove.

Maintenance, licensing and the cost of upgrading

The licence is MIT, stated in the README and in the setup.py classifier. That permits commercial use and modification, but it also means the package carries no warranty, and the README offers no support commitment. If you need a guarantee, this is not the project that provides one. This is a description of the licence terms, not legal advice; check the LICENSE file and your own obligations.

Maintenance signals are mixed and worth stating plainly. The default branch last received a push on 2026-09-07, and the repository is not archived. The most recent release, 0.1.0, is dated 2023-10-30. So the code moves but the published artifact does not move with it. If you pin to a PyPI version, you are pinning to something that may be behind master, and the README's own version references, such as the 0.0.42 pipeline feature and the 0.0.41 Python 3 boundary, show that the project has used version numbers to mark real behavioural changes before.

The upgrade cost is mostly the dependency floors. The setup.py comments explain that the requirements are floors rather than bare names so a resolver cannot land joblib 1.1.0 or tqdm 4.66.2, both of which have associated CVEs. That is a deliberate choice, and it means installing Sacremoses can force an upgrade of joblib and tqdm in an environment that pinned them lower. On a shared environment, that is the change most likely to break something else. The package_data comment in setup.py also explains that the Unicode character classes and nonbreaking prefixes ship as generated Python modules rather than text files, which keeps the installed footprint smaller but means the data is baked into the release rather than swappable at run time.

Editorial conclusion

Adopt Sacremoses if your corpus or your model was prepared with the Moses scripts and you need the same tokenization at inference time without a Perl toolchain. Do not adopt it as a general-purpose NLP tokenizer: it splits on Moses rules, escapes XML entities by default, and its truecaser is a case model trained on your own text, not a pretrained one. Before wiring it into a pipeline, verify two things: that Python 3.8 or newer is available, since the README states the current release requires it, and that the aggressive dash split flag matches whatever produced your training data, because the README shows the tokenizer and detokenizer round trip only for the examples it prints.

Frequently asked questions

Official sources

1. hplt-project/sacremoses on GitHub
2. Issues
3. License: MIT
4. README
5. Releases