chinese-ime-lm, language models and an evaluation set for Chinese input methods

Two routes, one job: choosing between pinyin readings

Chinese input method editors work by turning pinyin keystrokes into candidate readings and then choosing between them. The hard part is that one keystroke sequence maps to many characters, and the choice depends on context. The repository exists to attack that choice and, just as importantly, to measure whether an attack worked.

The README describes two parallel routes serving the same purpose. The n-gram route runs corpus cleaning, KenLM statistics and Viterbi decoding, living under corpus/clean/ and ngram/. The neural route fetches corpora, trains a character level Transformer and reranks the candidates assembled by the decoder, spread across corpus/fetch.py, neural/ and reference/.

What makes the project unusual is that both routes are judged by one evaluation set, which the README argues is more valuable than either model. The eval/ directory holds 25,119 word level cases plus 60 whole-sentence cases. The README's claim is that this turns the question of whether a change is an improvement into a number rather than an opinion, and that it freezes two measurement traps into explicit test cases. The traps are documented in docs/measurements.md.

The projects are staged differently, though. The directory table lists reference/ as a Rust inference reference implementation with no unsafe code and no dependencies beyond serde, licensed Apache-2.0, and docs/format.md as a model file format specification that claims a loader in any language needs only that one document.

The current status line in the README is blunt: both neural routes are in use, while the n-gram route has never produced anything outside this repository.

What the measurements say about model size and ranking quality

The README reports a comparison against four Qwen models on the same candidate pool. The observation is that none of them beat the project's own 25.5 MB model, that a 0.6B and an 0.8B model tie with the 4.5 MB one, and that the size difference of 140 to 380 times comes with an order of magnitude slower single decisions.

But the conclusion drawn is more interesting than the ranking. The README states that all of the rank differences in ranking quality fall inside the noise. Across 52 cases, pairwise disagreement is only 2 to 6 cases, McNemar's test finds no significant pair, and detecting 1 to 2 percentage points would require somewhere between a thousand and nearly ten thousand cases.

So the stated reason to train your own weights is the two deterministic measurements, size and latency, not the ranking position. The same measurement run also found 12 whole-sentence cases where a pronoun is not decidable from pinyin at all, so those cases should not have been collected in the first place. All of this is recorded in docs/measurements.md.

There is a second measured caveat that matters more for a decoder designer. The README says the model is not universally beneficial: an exact dictionary hit on the whole key already carries the corpus word frequency, and covering those under any set of weights loses points. The model's value sits where the decoder assembles candidates without frequency evidence.

This is a genuinely useful framing for anyone considering a reranker. A general purpose model is not automatically better, and a benchmark with a known sample size tells you what you can and cannot conclude from it.

Corpus licences, the aggregation trap and why data/ stays empty

The repository distributes only processing scripts. The data/, model/ and kenlm_bin/ directories are gitignored apart from .gitkeep, and nothing inside them is tracked, so you prepare the corpora yourself.

The README explains why in detail, and the explanation is the part to read before anything else. The upstream aggregation repository brightmart/nlp_chinese_corpus is published under MIT, but that covers the aggregator's own work rather than the third-party texts it packages. The wiki2019zh derivative comes from Chinese Wikipedia and is CC BY-SA 3.0, a licence that requires attribution and is viral in the sense that anything substantially derived inherits those terms. The news2016zh and baike2018qa sets are crawled news and Baidu Baike content, and neither carries a redistribution licence.

The practical consequence the README spells out is that nobody downstream of Wikipedia can relicense Wikipedia text as MIT, so the aggregation repository's MIT badge only covers its scripts and packaging work. It also notes the ambiguous boundary: statistics computed from a corpus, such as n-gram counts and probabilities, usually do not equal the corpus itself, but the line between a statistic and a derived work is unclear, and a model able to reproduce source sentences is closer to a derived work than a bigram frequency table.

The result is that no product built from this pipeline has ever shipped. The README names MSIME-Engine specifically and says it contains no n-gram or KenLM model and no code path that would load one. Before any model from this pipeline reaches an input method, the licence of the specific subset used has to be settled, either by restricting training to cleanly redistributable sources or by meeting the CC BY-SA attribution and viral terms for the Wikipedia-derived part, and the conclusion written into the Engine NOTICE.md alongside other dictionary sources.

The workaround is corpus/fetch.py, which only fetches sources with clear licences, streaming the C4 set under ODC-BY and the LCCC set under MIT. The published neural weights are trained only on those.

Fetching clean corpora and training the character level model

The fetch script takes a source name and streams it, normalising as it goes without writing the raw archives to disk:

python corpus/fetch.py c4   --out data/c4.txt --max-chars 1_000_000_000
python corpus/fetch.py lccc --out data/lccc.txt --split large

The relationship between fetch.py and corpus/clean/ is that the latter handles corpora you prepared yourself while the former acquires ones with clear licences, and both use the same Chinese normalisation algorithm, which keeps connected runs of Han characters and treats everything else as a separator.

The neural route then trains and exports a model. The export step writes an int8 safetensors file:

pip install -r neural/requirements.txt

python neural/train.py --corpus data/c4.txt data/lccc.txt --out runs/desktop --preset desktop
python neural/export.py --run runs/desktop --out dist/model.safetensors --precision int8

Inference reference implementations live in reference/, and the file format is specified in docs/format.md, which the README presents as the single document a loader in any other language needs.

The weights themselves are published separately on Hugging Face as pinyin-ime-reranker-4M at 4.5 MB, which is the one to use on the keystroke path, and pinyin-ime-reranker-25M at 25.5 MB, which is described as topping out on accuracy. The same files are attached to a model-v1 release on the repository. The README explains the split in terms of the MSIME host API, which loads them through resource locks, with the small model serving the keystroke path and the large one taking over when the user pauses typing.

The n-gram route on Windows, and why it is a reference not a product

The n-gram side is documented in more operational detail, mostly for Windows with PowerShell 7.0 or later, and it is the older path. The README asks for Python 3.12, a virtual environment, pip install -r requirements.txt and a pip install of the kenlm package from source. Before decoding you need a pinyin to single character candidate table, generated by corpus/pinyin/make_single_pinyin_table.py, and then the Viterbi decoder runs from ngram/viterbi.py.

The statistics themselves are computed with KenLM rather than by hand, using the compiled binaries copied into kenlm_bin. The corpus format is character spaced, one line per sentence, which is what a character level n-gram needs since tokenisation is just splitting characters apart.

The reason to read all of this is not to adopt it. The README's status line says the n-gram route has never produced anything that left this repository, and that even the neural route, when used to decode directly by syllable rather than to rerank candidates, still performs worse than the engine it was meant to improve. Reranking helped; direct decoding did not reach the mark.

That admission is the most useful sentence in the README for an outside reader. The project has separated the two jobs, found that the learned model helps with candidate selection and does not yet help with generation, and has published both the measurements and the reason.

Directory licensing, the NOTICE obligation and how this compares to other IME work

Licensing in this repository is per directory, which is unusual and deliberate. The repository as a whole is GPL-3.0, covering the scripts. The Rust code under reference/ has a single copyright holder and is additionally offered under Apache-2.0, with the reason given plainly: many open source input methods are MIT, Apache or BSD and cannot link GPL code, so third parties can take just that directory. The format specification, the evaluation set and the published weights are not constrained by the code licence at all. The weights are Apache-2.0 for the same linking reason, and the README adds the observation that a public resource nobody can adopt is not a public resource. That licence is written into the weights' own __metadata__, so it travels with the files, and a separate NOTICE file carries the attribution obligations that come from the corpora rather than from the code, with the README noting those two kinds of obligation cannot substitute for each other.

Compared with the alternative, this is a different approach from commercial input method engines that ship their own language models as closed artefacts, and from open source input methods that embed a large statistical dictionary without publishing the evaluation method. Here the evaluation set is a first class artefact, 25,119 word level cases plus 60 whole-sentence cases, and docs/measurements.md publishes the conclusions including the ones that are unfavourable.

For a team deciding whether to use the published weights, the practical questions are narrow. The 4.5 MB and 25.5 MB files carry an Apache-2.0 licence that will not block linking into an MIT or BSD input method, and the corpus attribution obligations live in the NOTICE file. What you still have to check is whether a reranker helps in your decoder at all, which is the question the project's own measurements answer for the engines it tested, and only for those.

Editorial conclusion

Adopt chinese-ime-lm if you are building a Chinese input method and want a shared benchmark plus two Apache-2.0 model files you can load without inheriting GPL obligations. Do not adopt the n-gram route, since the README states no n-gram or KenLM model has ever shipped in a product and the neural route is the only one in use. Verify first by reading docs/measurements.md, running the evaluation set against your own decoder, and confirming that the corpora you feed the scripts are licensed for redistribution, because the corpus licence question is the one that has to be settled before any of this reaches users.

Frequently asked questions

Official sources

1. License: GPL-3.0
2. metasequoiaime/chinese-ime-lm on GitHub
3. Project website
4. README
5. Releases