nagisa: Japanese word segmentation in four lines, and a train method nobody expects

Two models, split by task, and the split is in the citations

Nagisa is a Python module for Japanese word segmentation and part-of-speech tagging, and it is built on recurrent neural networks. The two tasks are handled by two different models, and the documentation attributes each to separate prior work rather than to one paper.

The word segmentation model uses character-level and word-level features, credited to a 2017 conference paper. The part-of-speech tagging model uses tag dictionary information, credited to a paper in the ACL anthology. That separation is the reason the package is not just a character n-gram segmenter with a lookup table attached: the boundary decision and the tag decision are learned separately, and each is built on a feature type that suits its problem.

The output is a list of token and tag pairs. Given the sentence Pythonで簡単に使えるツールです, the tagging call returns Python paired with a noun tag, で with a particle tag, 簡単 with an adjective, に with a verb ending, 使える with a verb, ツール with a noun, and です with a verb ending. Two accessors sit on the result: one for the words alone and one for the tags alone, so the common cases do not require you to unpack tuples.

One detail is stated without emphasis and matters for anyone comparing output strings: the tokens are normalized using Unicode NFKC normalization. Half-width and full-width variants collapse before you see them.

The post-processing is a filter and an extract, over a tag set that reaches URL

Once you have tokens and tags, the package offers two functions and they are not the same thing. `filter` removes the tokens whose tags are in a list you pass, and `extract` keeps only those tokens. Filtering particles and verb endings out of a sentence leaves the content words behind; extracting nouns leaves only the nouns.

The tag set is available at runtime rather than being documentation you have to look up, which is the practical way to discover what you can filter on. The printed list starts at an auxiliary symbol class and a noun class and runs through many more, and the last entry shown is a URL tag. That last one is worth a second look, because it tells you the tagger treats URLs as a first class token type rather than as three separate words split on punctuation.

The examples use a compound-script sentence, so the list you would pass is given in Japanese: a list of the particle and verb-ending tags, then a list of the noun tag. Both calls take the raw text rather than a previously tagged result, so each one runs the tagger again.

For anything beyond Japanese, the tag set is a Universal Dependencies style set, which is what makes the training section possible to pick up. The sample dataset shown is tagged with NOUN, ADP, ADV, ADJ, AUX, PUNCT, and EOS, and the trained model on a Fukuoka tourism sentence returns PROPN, SYM, ADP, NOUN, and NOUN.

The stopword list is a plain set, so filtering it is a list comprehension

Nagisa ships a built-in Japanese stopword list, and the way you use it is deliberately not a package function. The example filters the tagged words with a Python list comprehension, testing membership against `nagisa.stopwords`.

That is a design decision with a practical upside. A function that removed stopwords for you would be convenient, and it would also hide what got removed. With the set exposed, you can see the list, and you can invert the test, or restrict it to a subset of the tags, or intersect the two rules yourself, without the package deciding which of those you wanted.

The example that demonstrates it is instructive precisely because the segmentation is aggressive. The sentence about using Japanese stopwords easily tokenises with 日本 and 語 as separate words, and ストップ and ワード as separate words, which is what a character and word level segmenter does with an unlisted compound. Removing the stopwords therefore removes the two particles but leaves both halves of both compounds.

If that matters for your use, the user dictionary is the answer, and it is a one argument change rather than a code edit. The `Tagger` class accepts a `single_word_list`, and adding a multi-word entry makes the tagger treat it as one token instead of splitting it. The documentation demonstrates this with a title containing digits and kanji, where the default tagger splits it and the custom tagger keeps it whole.

Training produces three files and takes a TSV with EOS between sentences

The training interface is a single function, and the data format is the first thing to understand. Train, dev, and test files are TSV, one token per line as `word` then a tab then `tag`, and a line consisting of `EOS` separates sentences. That is the whole contract, and the sample files show it: a line of `唯一` tagged NOUN, then `の` tagged ADP, through to a `EOS`, then the next sentence beginning.

The function call names all three files and a model name:

nagisa.fit(train_file="sample.train", dev_file="sample.dev", test_file="sample.test", model_name="sample")

When it finishes it writes three model files, with the extensions given as a vocabulary file, a parameters file, and a hyperparameter file. You then build a tagger from them by passing the three paths:

sample_tagger = nagisa.Tagger(vocabs='sample.vocabs', params='sample.params', hp='sample.hp')

The scope of the model is described as joint word segmentation and sequence labelling, with part-of-speech tagging and named entity recognition given as the examples of the latter. That is the important line in this section: the trainer is not specific to tagging, so the same call produces a NER model if your tags say so.

Sample datasets ship inside the package under a sample datasets directory, and there is a tutorial for training on Universal Dependencies. So the fastest way to understand the format is to train on the shipped sample and then point it at your own.

Installation compiles a Cython extension, and the compiler flags are chosen per platform

The install is one command:

pip install nagisa

What it does behind that is worth knowing before you hit a failure on a machine without a build toolchain. The build requirements list setuptools, Cython, and numpy, and the setup script compiles a Cython source file into an extension module called `nagisa_utils`, built against the numpy include directory. The extension file itself is listed in the package data, so the Cython source is shipped rather than only the compiled artefact.

The compiler flags are platform dependent and the reason is given in a comment. On Windows the extra flag is `/O2`; on everything else it is `-O3`, because the inference kernels in the module rely on the compiler auto-vectorising their inner loops. So the project is explicitly asking for optimisation rather than accepting a debug build, and the difference between the two platforms is a syntax difference in the flag, not a difference in intent.

The supported platform matrix is asymmetric. Linux is listed from Python 3.6 through 3.14, while macOS and Windows start at 3.9. The runtime dependencies are few: six, numpy, and DyNet, with the DyNet package name itself switching at Python 3.8, where the modern package applies and the older one is used below that.

The classifier list in the package metadata is the part that has not kept up. It still advertises Python 2.7 and 3.5 through 3.14, and it does advertise 3.14 alongside the Linux floor, so the metadata is broader than the documented matrix in one direction and stale in another.

The project is a person's research made installable, and the release cadence says so

The authorship is a single named developer, Taishi Ikeda, and the supporting material is a mix of formats that suggests a research project packaged for reuse rather than a product with its own documentation site. There is a readthedocs build, a Japanese article hosted elsewhere, a slide deck from a Python conference in 2022, a demo space on a model hosting platform, and a package index listing. Four different venues, no single canonical one.

The release history explains part of the shape. Version 0.2.11 shipped in January 2024, version 0.2.12 in February 2026, and version 0.3.0 in July 2026. So there was a long gap of more than two years between two patch-level releases, then a minor bump, and the last push to the repository is dated 2026-09-22, which is after the newest tag.

The zero major version is doing real work here. At 0.3.0, conventional expectations would say the API is not stable, and nothing in the documentation promises otherwise. The public surface is small enough that a breaking change would be obvious immediately, since most programs import the module and call one function.

For an evaluator, the practical questions are whether the POS tag set covers what you need and whether the segmentation granularity matches your text. Japanese tokenisation is not a detail you can defer, and the stopword and user dictionary mechanisms exist precisely because the default segmentation is opinionated.

Editorial conclusion

This suits someone who needs Japanese tokens and part-of-speech tags quickly, in a few lines, without a tokenizers library and a model download, and who is comfortable with a package that compiles a Cython extension at install time. It is a poor fit if you need named entity recognition out of the box, since the tagging model does not ship one, and a poor fit on a machine without a build toolchain, since the wheel is not self-contained. Before you install, check three things: that your Python is in the supported range, which is 3.9 to 3.14 on macOS and Windows and 3.6 to 3.14 on Linux; that you have a compiler available, because the inference kernels are built from a Cython source file at install time rather than shipped prebuilt; and whether you need anything beyond nouns, since the tag set goes further but the examples in the documentation mostly filter down to a handful of categories. The newest release is 0.3.0 from 2026-07-06 and the last push was 2026-09-22.

Frequently asked questions

Official sources

1. License: MIT
2. Project website
3. README
4. Releases
5. taishi-i/nagisa on GitHub