# Lindera: a Rust morphological analyser with bindings for six languages

> Lindera is a Rust library for morphological analysis and tokenization that grew out of kuromoji-rs and now ships dictionaries for Japanese, Chinese and Korean along with bindings for Python, Node.js, Ruby, PHP and WebAssembly. The workspace is a single Cargo project with twenty members, and the project is about to change its major version.

**lindera/lindera** — A multilingual morphological analysis library.

- Repository: https://github.com/lindera/lindera
- Stars: 696 · Forks: 64
- Language: Rust
- License: MIT
- Published: 2026-10-21 · Updated: 2026-10-21 · Language: en
- Canonical page: https://hysenlabs.com/projects/lindera-lindera

## A kuromoji fork that grew into a workspace of twenty crates

Lindera is a morphological analysis library in Rust, and it starts from a fork of kuromoji-rs, which was itself a Rust port of the Japanese tokenizer. The stated aim is to build a library that is easy to install and provides concise APIs for various Rust applications.

What it has become is a Cargo workspace with twenty members, and the member list in Cargo.toml is the clearest map of the project. Alongside the core lindera crate there are lindera-analysis for the analysis layer, lindera-segmenter, lindera-dictionary, lindera-cli, lindera-crf, lindera-trainer and a binding crate, and then one crate per language dictionary or per binding.

The dictionaries are the point of much of that. There is lindera-ipadic for the standard Japanese dictionary, lindera-ipadic-neologd, which is the neologd variant of IPADIC, lindera-unidic, lindera-ko-dic for Korean, and lindera-cc-cedict for Chinese. There is also lindera-jieba, which is named for the Chinese segmenter and points to a different approach, since Jieba is a statistical segmenter rather than a dictionary lookup.

Two supporting crates are worth naming. lindera-trainer suggests the dictionary formats are trained rather than only curated, and lindera-cli is the command line interface, which the readme uses for the dictionary download command.

The workspace uses the 2024 Rust edition, and it declares a minimum of Rust 1.89. The comment in Cargo.toml explains why that floor is higher than the edition implies: lindera-dictionary uses slice::as_chunks, stable since 1.88, and File::lock, stable since 1.89, and the declaration turns a missing method into a clear cargo message on older toolchains.

## Dictionaries are downloaded, not compiled in

The install story has two parts, and the second one surprises people.

The Rust dependency is ordinary:

```toml
[dependencies]
lindera = "7"
```

But a dictionary is a large data file, and it is not in the crate. Pre-built dictionaries are available from the GitHub Releases page, and you download a dictionary archive, for example one named lindera-ipadic-*.zip, and specify its path when loading. There is a shortcut for the command line: lindera download ipadic downloads and installs a dictionary automatically.

That split between the library and the data is what makes the multi-language design work. A single crate can carry several dictionaries if you ask it to, and a user who only needs Japanese never downloads the Korean one. It also means the first run of a program that needs a dictionary will fail with a missing path unless the dictionary is in place, which is the sort of thing worth catching in a health check rather than in production traffic.

The Makefile shows how the build handles dictionaries for the bindings. The WebAssembly build uses the feature embed-ipadic, which embeds the Japanese dictionary into the artefact, and the Node.js test build uses embed-ipadic,embed-ko-dic. The comment there is precise about the reason: the dictionaries are embedded so that the dictionary-backed tests run instead of being skipped, and the memcheck builds use the same features so that they do not switch the addon to a different build.

Embedding a dictionary into a binary is a real decision about artefact size, and the fact that it is a cargo feature means the choice is per build rather than per distribution.

## Bindings for Python, Node, Ruby, PHP and the browser

The number of language bindings is what distinguishes Lindera from a plain Rust crate, and each one has a different installation story.

Python installs from pip:

```bash
pip install lindera
```

Node.js installs from npm:

```bash
npm install lindera
```

Ruby installs as a gem, and from version 6.1.0 it installs from RubyGems as lindera. The caveat is stated plainly: the gem compiles its native extension during installation, so it needs a Rust toolchain and libclang.

```bash
gem install lindera
```

PHP is a PHP extension rather than a library, and from 6.1.0 it is published on Packagist as lindera/lindera and installed with PIE:

```bash
pie install lindera/lindera
```

WebAssembly is a separate package, lindera-wasm, installed from npm:

```bash
npm install lindera-wasm
```

The WebAssembly build uses wasm-pack's web target and is described as working in browsers natively as well as with modern bundlers such as Vite and Webpack 5, with a demo application in the lindera-wasm directory.

The Ruby and PHP cases are the interesting ones for a decision. Both compile native code at install time, which means end users of a Ruby or PHP application that depends on Lindera need a Rust toolchain on their machines, or need the prebuilt wheels or extensions to be available for their platform. That is a materially heavier deployment story than pip or npm, and it is the kind of constraint that decides a binding choice for a team without Rust in its CI already.

## Ten to twenty megabytes per second on Japanese

The performance claim is specific and qualified, which is a good sign.

Lindera tokenizes Japanese text with IPADIC at roughly 10 to 20 MB per second single-threaded, depending on the hardware and on whether the Viterbi lattice is reused across calls. The mention of lattice reuse is the interesting qualifier, because the Viterbi lattice is the intermediate structure the segmenter builds over candidate segmentations, and building it dominates the cost of a short input. Reusing it across calls is a real optimisation and it changes the numbers substantially for workloads with many small strings.

The benchmark suite lives in lindera/benches/ and you reproduce it with make bench. The readme points to issue 875 for the detailed methodology and comparison numbers, which is the right place to look before quoting a throughput figure, because the number on its own does not say what was tokenized or how the lattice was handled.

The vocabulary used is the right one for this kind of library. Morphological analysis is not the same as splitting text on whitespace, and the dictionary-based approach is what gives you word segmentation with correct handling of Japanese agglutination, plus part of speech and other features attached to the tokens. The cost is the dictionary dependency and the memory it takes to load it.

That trade is the one to be clear about. A dictionary based tokenizer gives better results on the languages it has dictionaries for, and gives you nothing for languages it does not. Lindera's dictionary crates cover Japanese, Chinese and Korean, and the Chinese support includes both a dictionary approach with cc-cedict and a statistical one with jieba, which is the right way to cover a language where both approaches are common.

## A version number that is about to move under you

There is an inconsistency in the version story that anyone adopting this needs to notice.

The Cargo.toml workspace declares version 7.0.0, and the dependency snippet in the readme asks for lindera = "7". The three most recent releases, however, are v6.2.0 from 2026-09-26, v6.1.0 from 2026-09-25, and v6.0.0 from 2026-08-25. The readme also carries a note pointing at a migration guide for anyone upgrading from v6, with the guide file named for the move from version 6 to version 7.

So the workspace has already moved to 7.0.0 on the main branch, the crates.io version you would resolve with "7" is a version whose release tag you may not find in the list above, and the migration documentation exists because the break is real. The Ruby and PHP bindings also changed distribution channel in 6.1.0, which is the kind of change that breaks a build script quietly.

The activity level otherwise is strong. The repository is not archived, the default branch is main, and the last push was on 2026-10-20, the day before the current date, with three releases inside the last two months.

The licence is MIT, and there is an individual contributor licence agreement, Lindera_Individual_CLA.md, at the root, alongside per-language version files such as .python-version, .ruby-version, .node-version and .php-version, which is what a project with this many bindings needs to keep its CI matrix coherent.

One dependency pin is worth reading before you touch it. crawdad is pinned exactly at 0.4.1, and the comment in Cargo.toml explains that the runtime walks crawdad's serialized byte layout in place, which the crate does not document as a stable contract. A round-trip test fails loudly if the layout changes, and the comment asks reviewers to look before moving the pin. That is a well handled risk, and it is also a reminder that this library sits on top of a format it does not own.

## Conclusion

Lindera suits a team that needs dictionary-based morphological analysis of Japanese, Chinese or Korean with a native speed advantage, or a Python or Node developer who wants the same tokenizer without writing one. It does not suit a project that needs a stable major version right now, because the workspace is already at 7.0.0 while the newest tagged release is v6.2.0 and the documentation carries a migration guide from 6 to 7. Before upgrading, read the migration guide, plan the dictionary download, and check the pinned crawdad version, since the runtime walks its byte layout in place.

## FAQ

### What is Lindera and what language is it written in?

It is a morphological analysis library in Rust, forked from kuromoji-rs, intended to be easy to install and to provide concise APIs. The repository is a Cargo workspace with twenty members covering the core library, the analysis and segmenter layers, the dictionaries, the CLI, the trainer and the language bindings.

### How do I install Lindera and its dictionaries?

Add lindera = "7" to your dependencies in Cargo.toml, and separately obtain a dictionary. Pre-built dictionary archives are on the GitHub Releases page, and you specify the path when loading, while the CLI can fetch one with lindera download ipadic.

### Which languages does Lindera have bindings for?

Python via pip, Node.js via npm, Ruby as a gem, PHP as a native extension installed with PIE, and WebAssembly via the lindera-wasm package. The Ruby and PHP ones compile native code at install time and need a Rust toolchain and libclang.

### How fast is Lindera on Japanese text?

The documentation states roughly 10 to 20 MB per second single-threaded with IPADIC, depending on the hardware and on whether the Viterbi lattice is reused across calls. The benchmark suite is in lindera/benches/ and is run with make bench, with methodology and comparisons in issue 875.

### Which dictionaries does Lindera ship?

The workspace includes crates for IPADIC, the neologd variant of IPADIC, UniDic, the Korean ko-dic and the Chinese cc-cedict, plus a jieba crate for the statistical Chinese segmenter. A Cargo feature can embed a dictionary into the binary, as the WebAssembly build does with embed-ipadic.

## Sources

- [Issues](https://github.com/lindera/lindera/issues)
- [License: MIT](https://github.com/lindera/lindera/blob/main/LICENSE)
- [lindera/lindera on GitHub](https://github.com/lindera/lindera)
- [README](https://github.com/lindera/lindera/blob/main/README.md)
- [Releases](https://github.com/lindera/lindera/releases)

---

Hysen Labs editorial analysis, written from the project's own repository and release notes. Cite the canonical page: https://hysenlabs.com/projects/lindera-lindera