Lingua: language detection that holds up on single words
The most accurate natural language detection library for Rust, suitable for short text and mixed-language text
At a glance
- What is it?
- pemistahl/lingua-rs is an Apache-2.0 Rust library that identifies which of 75 languages a piece of text is written in, with Python bindings shipped from the same repository. It uses rule-based and statistical Naive Bayes methods rather than a neural network, and the 1.8.0 release traded disk space for a large drop in memory use.
- Who is it for?
- Use Lingua if you need to tag the language of short text, single words or mixed-language input and you want that decision made locally with no network call, which is the gap the README says existing Rust libraries left open. Use it on the server where the 300 MB of model files on disk is not a problem, and measure the WASM module before shipping it to a browser.
- Can I use it commercially?
- Yes. Apache-2.0 is a permissive licence: you can use, modify and sell software built on it, as long as you keep its copyright and licence notices.
- Is it still maintained?
- Yes. The repository last received commits 1 day ago.
- What is it written in?
- Mainly Rust, according to GitHub's language statistics.
Answers come from the project's GitHub data, last synced on September 18, 2026, and from our analysis. They are not legal advice.
DEEP OPEN-SOURCE ANALYSIS
The single job it does
Lingua answers one question: which language is this text written in. The README frames it as a preprocessing step for other NLP work such as text classification and spell checking, and gives a second use case that is easier to picture, routing e-mails to the right geographically located customer service department based on the language they are written in.
That narrowness is the point. The README argues that language detection is usually bundled into large machine learning frameworks, and that when you do not need the rest of that framework, a small flexible library is the better tool.
The repository is more than a Rust crate. Alongside src/ and Cargo.toml it carries pyproject.toml, requirements.txt, lingua.pyi and a README_PYPI.md, so Python bindings are built from the same source tree, plus language-models/, accuracy-reports/, benches/ and tables/ directories. The Cargo workspace lists language-models/* as members, so each language model is its own crate.
The audience is anyone who needs a language tag before they can do anything else with a piece of text, particularly when that text is short.
Rule-based plus Naive Bayes, no neural network
The README is specific about the method. Lingua draws on rule-based and statistical Naive Bayes methods, and explicitly does not use neural networks or dictionaries of words. It also needs no connection to an external API, so once the library is downloaded it runs completely offline.
Those three denials describe the design more precisely than a feature list would. No neural network means no inference runtime and no GPU, which is why it can be a library you call synchronously inside another program. No dictionary means the model is ngram statistics rather than a word list, which is why it can work on words it has never seen. No API means it is usable in air-gapped or privacy-sensitive pipelines, and it means latency is local.
The README names the two problems it was written to solve in the existing Rust libraries. Detection only worked on fairly long text fragments, so short snippets such as Twitter messages came back wrong, and accuracy fell as more languages joined the decision process. Lingua claims accurate results on both long and short text, even single words and phrases, and states it needs almost no configuration.
The rule-based half is what handles the cases where statistics are thin, such as a text in a script that only one supported language uses.
75 languages, chosen for quality
The README states a policy of quality over quantity, getting detection right for a smaller set of languages before adding more, and the badge in the README says 75 supported languages. The list runs from Afrikaans through Zulu and includes Chinese, Japanese, Korean, Arabic, Hebrew, Hindi, Tamil, Telugu, Thai, Vietnamese and the major European languages, plus a set of African languages including Ganda, Shona, Somali, Sotho, Tsonga, Tswana, Xhosa, Yoruba and Zulu.
Accuracy testing is split into three shapes, which is the right way to test a detector. Each language's test data is a list of single words of at least 5 characters, a list of word pairs of at least 10 characters, and a list of complete grammatical sentences of varying length.
The data comes from the Wortschatz corpora at Leipzig University. Per the README, training used corpora of one million sentences each crawled from news websites, and testing used separate corpora of ten thousand sentences from arbitrarily chosen websites, from which a random unsorted subset of 1000 single words, 1000 word pairs and 1000 sentences was taken. Training and test data coming from different sources is what makes the numbers worth reading at all.
Adding it to a Rust or Python project
The crate is published as lingua. The version in the repository's Cargo.toml is 1.9.0, declared with edition 2024 and a crate-type of both cdylib and rlib, which is what allows the Python binding to load it:
[dependencies]
lingua = "1.9.0"Note that the newest tag in the repository is v1.8.0, published on 2026-03-09, so the version in Cargo.toml is ahead of what has been released. Pin 1.8.0 if you want a tagged build, and check crates.io for what is actually published before you write the line above.
Python users get a separate distribution. The pyproject.toml names it lingua-language-detector at version 2.3.0, and it requires Python 3.12 or newer:
pip install lingua-language-detectorThe build side is visible in requirements.txt, which pins maturin at 1.15.0 and pytest at 9.1.1, so the Python package is compiled from the Rust source with maturin rather than reimplemented.
Beyond the library there is a binary named accuracy_reports gated behind an accuracy-reports feature, declared in Cargo.toml, which is how the comparison numbers in the README are produced.
The 1.8.0 trade: memory down, disk up
Release v1.8.0, published on 2026-03-09, changed how language models are stored, and the release notes are candid about the cost.
The models moved from a hashmap-based format to finite-state transducers. The notes say this reduces memory consumption drastically at the cost of slightly slower runtime performance, and that FSTs can be searched on disk without being read entirely into memory, so loading all languages now needs only a few dozen megabytes where the former approach required at least hundreds of megabytes.
The second change is that model files are no longer compressed with Brotli. Loading is much faster, which the notes say nearly eliminates latency problems in web services, and the FST format helps there too. The stated downside is size on disk: roughly 300 MB altogether, up from 110 MB before, and the WASM module grows as well.
That is a clear trade to evaluate against your deployment. If you run one service with all 75 languages loaded, memory is now cheap and startup is fast. If you ship to a browser or an edge function where download size is the constraint, the larger WASM module is the thing to measure.
The same release also notes that unique and most common ngrams per language improve accuracy slightly when low accuracy mode is enabled.
Reading the accuracy comparison honestly
The README compares Lingua against CLD2, Whatlang and Whichlang over Lingua's 75 languages, and it takes care to point out a trap in its own charts.
The first box plot appears to show Whichlang as the most accurate classifier. The README says that impression is wrong, because Whichlang supports only 16 languages while Lingua supports 75, and a detector choosing among fewer languages has an easier problem. The last two plots therefore restrict the comparison to the common subset of 16 languages supported by all three, and on that subset the README states Lingua is the most accurate.
Languages unsupported by a given library are ignored for that library during detection, which is a fair rule and also means the all-languages plots are not comparing like with like.
The README presents these as bar plots and box plots with medians and middle-50% boxes rather than as a single headline number, and the body of the README available here does not state a specific accuracy percentage. Anyone who needs a figure should read the plots in the repository's accuracy-reports or generate them with the accuracy_reports binary rather than quote a number from memory.
CLD2, Whatlang and Whichlang
The alternatives are named in the README itself, and the difference is what each one was built to handle.
CLD2 and Whatlang are the established Rust-ecosystem options and cover many languages, but the README's criticism of most existing libraries is that they need lengthy text and lose accuracy as more languages participate. Whichlang is the fast one, restricted to 16 languages, which is exactly why it looks strong on Lingua's all-languages chart.
So the choice is shaped by your input and your language set. If your text is long and your languages are common, CLD2 or Whatlang will serve and carry less weight. If your text is a tweet, a search query or a single word, the short-text behaviour is the whole decision, and that is the case Lingua was written for.
The Python side has other options too, though the README does not discuss them, and the binding is the same engine rather than a separate implementation, so results should match between languages calling it.
Licence, versions and upkeep
Lingua is Apache-2.0, stated both in the README badge and in Cargo.toml, with a LICENSE file at the root. Apache-2.0 is permissive and carries a patent grant, so commercial use is straightforward.
The release history shows a project that fixes what it breaks. Version 1.7.1 on 2025-03-21 fixed a build failure when only a subset of language features was selected when declaring the dependency, which is the kind of bug that appears when you use feature flags to trim compile time. Version 1.7.2 on 2025-05-27 fixed low accuracy mode producing random results for certain kinds of text, which is a correctness bug in a mode people enable for speed.
The last push to the repository was on 2026-09-10, and the newest tag is v1.8.0 from 2026-03-09, so roughly six months of commits sit outside a release while Cargo.toml already reads 1.9.0. If you adopt it, decide deliberately whether you are tracking the repo or a tag, and if you enable low accuracy mode, test it on your own text, since that mode has had one correctness bug already.
Editorial conclusion
Use Lingua if you need to tag the language of short text, single words or mixed-language input and you want that decision made locally with no network call, which is the gap the README says existing Rust libraries left open. Use it on the server where the 300 MB of model files on disk is not a problem, and measure the WASM module before shipping it to a browser. If your text is long and your languages are common, CLD2 or Whatlang will do the job with less weight. Before you commit, check what crates.io actually has, because Cargo.toml in the repository reads 1.9.0 while the newest tag is v1.8.0 from 2026-03-09, and run the accuracy_reports binary against your own samples rather than trusting the plots alone.
Frequently asked questions
What are some examples of language detection models?
Lingua uses rule-based and statistical Naive Bayes methods and explicitly avoids neural networks and word dictionaries. The README compares it against CLD2, Whatlang and Whichlang.
How many languages does Lingua detect?
The README badge states 75 supported languages, and the project describes its policy as quality over quantity, getting detection right for a smaller set before adding more.
Does Lingua need an internet connection?
No. The README states it needs no connection to an external API or service and can be used completely offline once downloaded.
Can I use Lingua from Python?
Yes. The repository builds a Python package named lingua-language-detector, version 2.3.0 in pyproject.toml, requiring Python 3.12 or newer and built from the Rust source with maturin 1.15.0.
Why did the model files get larger in Lingua 1.8.0?
Release v1.8.0 switched models to finite-state transducers and stopped compressing them with Brotli. Memory use dropped from hundreds of megabytes to a few dozen and loading got faster, but on-disk size grew from 110 MB to roughly 300 MB.
Is Lingua accurate on very short text?
The README says it yields accurate results on long and short text, even single words and phrases, and its test data is split into single words of at least 5 characters, word pairs of at least 10 characters and full sentences.
Community notes