Sudachi: a Japanese morphological analyzer with three segmentation modes and a plugin system

What a morphological analyzer does, and why the three tasks are one operation

Sudachi is a Japanese morphological analyzer, and the README states that morphological analysis consists mainly of three tasks: segmentation, part-of-speech tagging and normalization.

That grouping is the design decision. A tokenizer that only splits text into words has to leave the decision of where the boundaries belong to a dictionary lookup, and for Japanese that lookup is genuinely hard, because a compound can be written as one word, two words or a phrase and the correct answer depends on context. Splitting, tagging and normalising separately means doing the hard part three times over or losing information between the steps. Doing them together means the tagger and the segmenter can inform each other.

The primary language is Java, the homepage is nlp.worksap.co.jp, and the topic tags name the domain precisely: morphological-analysis, nlp-library, pos-tagging and segmentation. The default branch is develop, which is worth noting alongside the stability warning later on. The repository is not archived, and the last push was on 2026-09-18, the same day as the v0.8.2 release tag.

The feature list adds four things beyond the three core tasks. Multiple-length segmentation, where you can change the mode of segmentation, and which can extract morphemes and named entities at once. A large lexicon based on UniDic and NEologd. Plugins, where you can change the behaviour of the processings. And work closely with the synonym dictionary, with the note that the synonym dictionary will be released at a later date, which is a promise rather than a shipped feature.

The lexicon point is worth pausing on, because UniDic and NEologd are the reason Sudachi is usable on modern Japanese at all. Both are named as the basis of the lexicon, and the Full dictionary size exists to carry the miscellaneous proper noun vocabulary.

The project describes itself as a Japanese tokenizer for business, which is a claim about the target use rather than about the technology, and the sentence in the file header is more direct: Sudachi is a Japanese morphological analyzer.

The three dictionaries, and the V1 binary format that broke upgrades

There are three dictionary sizes, and the difference between them is what they include. Small includes only the vocabulary of UniDic. Core includes basic vocabulary and is the default. Full includes miscellaneous proper nouns.

That is a simple and useful axis, because the choice is a size versus coverage trade in a direction people can predict. A name-heavy corpus wants Full, a normal document is fine on Core, and a constrained deployment or a task where the tail of proper nouns does not matter wants Small.

The dictionaries are published pre-built at a dedicated distribution URL under the v1 path, and the SudachiDict repository holds the details of how they are built.

Now the important operational detail, and it is the first thing in the README under an IMPORTANT marker. Version 0.8.2 introduces a new dictionary binary format, called V1. When upgrading from v0.8.1, you need to update the system dictionary and rebuild all user dictionaries against the exact system dictionary that will be used at runtime. The URL of the dictionary distribution page is also changed, and there is a migration guide.

Rebuild all user dictionaries is a strong requirement and it is the kind of thing that catches people out. A user dictionary is a compiled artefact tied to a particular system dictionary, and if the system dictionary changes format the user dictionary has to be regenerated against it. For a team with a handful of domain terms, that is an afternoon of work. For a team with many user dictionaries across many services, it is a migration with a checklist.

The reason to do it anyway is that the two dictionaries have to agree exactly at runtime. The requirement is not to rebuild against the newest system dictionary but against the exact one that will be used, which means the system dictionary version and the user dictionaries are a matched pair that has to be deployed together.

And then there is the CAUTION marker right below, which is the more important of the two for anyone pinning versions. Release v0.8.* is unstable. It may include breaking changes even between patch versions, so please pin the exact version and review release notes carefully before upgrading.

So the project is telling you two things at once, and they are not in tension. The 0.8 line is where the new dictionary format lives, so it is where the work is, and it is explicitly not a stable line. The releases confirm the pace: v0.8.2 on 2026-09-18, v0.8.1 on 2026-09-07 and v0.8.0 on 2026-05-26, so three releases in four months, and a patch release that changes the dictionary format is exactly the kind of thing the caution is warning about.

One more inconsistency worth noting, because it would send you to the wrong artefact. The Maven dependency example in the README shows version 0.5.3, which is well behind the 0.8 line. If you copy that snippet you get a version from the 0.5 series, with the old dictionary format and none of the 0.8 behaviour.

The three segmentation modes, and why you would choose each

The segmentation modes are the part of Sudachi that you will spend the most time choosing between, and the README gives a worked example for each.

In A mode, texts are divided into the shortest units equivalent to the UniDic short unit. In C mode, it extracts named entities. In B mode, into the middle units.

The examples in the core dictionary show the difference. For 選挙管理委員会, A gives 選挙/管理/委員/会, B gives 選挙/管理/委員会, and C gives 選挙管理委員会. For 客室乗務員, A gives 客室/乗務/員, B gives 客室/乗務員, and C gives 客室乗務員. For 労働者協同組合, A gives 労働/者/協同/組合, B gives 労働者/協同/組合, and C gives 労働者協同組合. For 機能性食品, A gives 機能/性/食品, B gives 機能性/食品, and C gives 機能性食品.

The full dictionary examples are longer and more realistic. For 医薬品安全管理責任者, A gives 医薬/品/安全/管理/責任/者 while C gives 医薬品安全管理責任者. And for さっぽろテレビ塔, A gives さっぽろ/テレビ/塔.

The pattern is worth understanding rather than memorising. A mode breaks at the morphological level, so it produces 東京 and 都 separately, and 選挙管理委員会 becomes four pieces. C mode groups the whole span into one named entity, which is why it gives you the entire compound as a single unit. B mode is the middle, giving you the compound broken at a coarser level than A but not as a single entity.

And the advice is explicit about which to use for a given task. In full-text searching, to use A and B can improve precision and recall. The reasoning is straightforward once you see the examples. A search index built on A mode has fine-grained postings, so a query for 選挙 matches 選挙管理委員会, and C mode would not produce that token at all. Precision comes from not indexing a fragment that happens to be a common word, and recall comes from the compound still containing the query term as a sub-unit.

So the mode is a choice about what a token means to your application, not a quality setting. A search index wants A or B, a named entity recogniser wants C, and a downstream task that consumes morphemes wants whichever gives the boundaries your model was trained on.

The plugin hooks, and why they exist

The plugin system is the other half of Sudachi's flexibility, and the four plugin types are defined by where in the analysis they intervene.

Modify the Inputs, with character normalization as the example. Make OOVs, with considering script styles as the example. Connect Words, with inhibition and overwrite costs as the examples. And Modify the Path, with fixing person names and equalization of splitting as the examples.

The naming is a description of the analysis pipeline. Inputs are the characters before anything else happens, so a plugin here changes what the analyzer sees. Out-of-vocabulary handling is where unknown words are dealt with, and a plugin here decides how a token the dictionary has never seen is treated. Word connection is the step that joins adjacent morphemes into a unit, and the cost model there, inhibition and overwrite, is what determines whether two neighbours merge. And the path is the search the analyzer does over its lattice of candidate segmentations, so a plugin that modifies the path changes the outcome of the search itself.

The prepared plugins are listed by the same categories. Under Modify the Inputs there is character normalization covering full and half width, cases and variants, and normalization of prolong symbols, which normalizes the tilde and the long vowel mark, and removal of yomigana, which removes the reading in parentheses.

The reason the character normalization plugin matters in practice is that Japanese text arriving from the web is full of half width katakana, mixed case latin and the wave dash and the tilde used interchangeably, and none of that is in a dictionary. Normalising it before lookup is the difference between matching a word and missing it.

And on the OOV side, the script style plugin is about the case where a word is not in the dictionary at all, which for Japanese means deciding whether an unknown katakana sequence is a loanword or a name, and that decision changes the token and therefore the tags.

The plugin tutorial is a separate document from the main tutorial, and there is a development page for building Sudachi from source. The project is a Java library with a Gradle build, since the repository has a build.gradle, settings.gradle, a gradlew wrapper and a gradle.properties, plus a src/ directory, a benchmark/ directory and a scripts/ directory. The Maven coordinates are com.worksap.nlp with the artifact sudachi, and the API is documented in the Javadoc.

Running it from the command line

The command line interface is complete enough to be useful on its own, and the option list is worth reading because several of the flags solve problems you would otherwise solve with a script.

$ java -jar sudachi-XX.jar [-r conf] [-s json] [-p directory] [-m mode] [-o output] [-t|-ts] [-a] [--print-reading] [-f] [-d] [--systemDict file] [--userDict file] [--format class] [file...]

The settings flags have a precedence order that is not obvious. -r specifies the setting file and overrides -s, and -s gives additional settings as JSON and overrides -r. The two descriptions claim to override each other, so the documentation does not settle which of -r and -s wins.

-p sets the root directory of resources, which is how you point the tool at a different dictionary set. -m selects the mode of splitting with the A, B and C values. -o writes to a file rather than standard output. And --systemDict and --userDict override the settings for the dictionary paths, with the note that --systemDict will override other settings while --userDict will not override other settings but will add another user dictionary, which is the useful behaviour when you want to layer a second dictionary on top of the configured one.

The output flags are where the tool earns its keep. -t separates words with spaces, and -ts separates words with spaces and breaks the line for each sentence, which is the form you want when the output is going into something that reads line by line. -a outputs the dictionary form, the reading form, the dictionary id, the synonym group id list, and the OOV flag, and it takes precedence over --print-reading, which outputs just the reading form. -f ignores errors, -d dumps the debug outputs, and --format class lets you use a provided class for formatting the output instead of the default configuration.

The examples show the output shape. Running the tool on 東京都へ行く gives 東京都, へ and 行く with their part-of-speech tags, and an EOS marker at the end. Adding -a adds the reading forms, the dictionary ids, the synonym group ids and the OOV flag, so 東京都 gains the reading トウキョウト. Adding -m A splits 東京都 into 東京 and 都, which is the same distinction the mode section describes. And -t gives the flat output 東京都 へ 行く.

For the API, you declare a dependency on the artifact in your Maven project, with the group com.worksap.nlp, the artifact sudachi, and a version, and the details are in the Javadoc.

Editorial conclusion

Sudachi fits a Japanese NLP team that needs accurate segmentation and POS tags with the freedom to adjust the behaviour per domain, and that can live with a project in a pre-stable 0.8 line. It is the wrong choice if you need a tokenizer that never changes under you, since the documentation says the 0.8 release may include breaking changes even between patch versions, and it is the wrong choice if you cannot rebuild user dictionaries when the system dictionary changes. Before upgrading past 0.8.1, rebuild every user dictionary against the exact system dictionary you will run at runtime, and pin the exact version rather than tracking the branch.

Frequently asked questions

Official sources

1. Issues
2. README
3. Releases
4. WorksApplications/Sudachi on GitHub