whatlanguage: Ruby language detection rebuilt on trigrams and a 220 KB model

A one-method API for the common case

The library is easy to pick up. It is a pure Ruby natural language detection library for 160+ languages, with no runtime dependencies, requiring Ruby 3.0 or newer, with JRuby and TruffleRuby also described as good options. The minimal call takes a string and returns a symbol.

require 'whatlanguage'

WhatLanguage.language("Que linguagem é essa? É uma pergunta sobre a língua portuguesa.")
# => :portuguese

That is the whole integration for most projects. The fuller API returns a result object with the language, the ISO code, a score and a ranked list of alternatives.

wl = WhatLanguage.new
text = "Die Stadt plant neue Investitionen in den öffentlichen Verkehr"
result = wl.detect(text)
result.language   # => :german
result.iso        # => :de
result.score      # => 79018
result.ranked     # => [[:german, 79018], [:dutch, 77631], ... ]

If you only need the ordering, there is a ranked method and a score hash.

wl.ranked(text)       # => [[:german, 79018], [:dutch, 77631], ... ]
wl.score_hash(text)   # => { german: 79018, dutch: 77631, ... }

The class can also be narrowed to a set of candidate languages, which is the practical move when your application knows the possible set.

wl = WhatLanguage.new(only: [:english, :german, :french])

Detection runs in two stages, script first and trigrams second

The mechanism is described precisely in the README, and the two-stage design explains both the speed and the failure modes. First, the dominant Unicode script is detected from the letters. Scripts used by a single supported language resolve immediately, and the README names Greek, Korean, Thai and Japanese using Hiragana or Katakana as examples. When kana is present, Han, Hiragana and Katakana count together as Japanese, though a dominant unrelated script can still win. For scripts shared by several languages, Latin, Cyrillic, Arabic and Hebrew are the examples given, the text is broken into trigrams which are ranked by frequency and compared against candidate language profiles. Digits, punctuation and emoji do not count toward the minimum text length and act as separators when trigrams are extracted, and combining marks are preserved inside trigrams but do not count as additional letters.

Version 2.0 dropped a Bloom filter and 5 MB of data for one 220 KB file

The README carries an important notice for anyone upgrading. Versions 1.0.6 and earlier, which covers the 2007 to 2025 run of the library, used a Bloom filter technique and shipped 5 MB of binary files to handle around 20 languages. Version 2.0 reimplemented the whole library, has many breaking changes, though the core WhatLanguage.language API remains similar. The new version is described as more accurate, faster, and supporting more languages from a single 220 KB JSON file. That is the shape of the trade: a 23-fold reduction in shipped data and a jump from roughly 20 languages to more than 160, bought by changing the underlying technique from a Bloom filter to ranked trigram profiles. The model itself lives at lib/whatlanguage/trigrams.json. A v2.0.1 release followed in July 2026 adding a command line executable, and v2.1.0 in September 2026 improved Unicode and mixed-script detection and short text accuracy.

Short text gets a five-language bonus that cannot be turned off

The short-text behaviour is where this library is most opinionated, and the README explains the mechanism in unusual detail. English, Chinese (Mandarin), Hindi, Spanish and French receive a built-in preference for short texts, selected using the 2025 total-speaker ranking including second-language speakers. Eligible trigram candidates get the same bonus: 600 ranking points through 10 letters, decreasing linearly to zero at 50 letters, and longer texts keep their original scores. The bonus is not configurable, it respects the only: restriction, and script detection takes priority, so Chinese resolves from its script and needs no statistical bonus. Scores exposed by the API include the bonus and remain ranking values rather than probabilities. That last sentence matters more than it first appears: a score of 79018 is a relative position, not a 79 percent probability, and code that treats it as confidence will be wrong.

Below ten letters, statistical detection returns nil on purpose

Short shared-script fragments are ignored by default because there is not enough signal to rank their languages reliably, and statistical detection returns nil below 10 letters by default. The threshold applies to the trigram stage, while scripts that identify a single supported language, such as Greek, Korean or Thai, can still resolve from shorter text. The threshold can be adjusted if you want to trade accuracy for coverage.

wl = WhatLanguage.new(min_chars: 0)

Setting it to zero is the switch to think about before using in production. The README's own caveat is that short texts can be ambiguous and the built-in preferences help some phrases while biasing others toward the preferred languages. A nil return is a signal the library is asking you to handle, and a biased answer with a rank attached is harder to notice. The default of ten letters is a deliberate refusal to guess, and turning it off means accepting guesses that carry the five-language preference with them.

Known limitations: closely related languages, kana-less Japanese, romanized text

The README lists its limitations directly, which makes evaluation easier. Short texts can be ambiguous. Scores indicate relative rank, not probability or confidence, so you should inspect the ranked alternatives. Similar written languages can be confused, with Norwegian Bokmål against Danish and Hebrew against Yiddish given as the pairs. Japanese without kana is classified as Chinese when Han is the dominant script, which follows directly from the script-first stage. Romanized text is compared with Latin-script profiles, so it may not resolve to its original language, and that is a real limitation for a lot of user-generated content in Latin script. On language codes, the README notes that iso and language_iso return ISO 639-1 symbols with an ISO 639-3 fallback, that selected languages use CLDR macrolanguage aliases such as Swahili swh to sw, Mandarin cmn to zh and Western Farsi pes to fa, and that these are a normalization policy rather than a BCP 47 requirement. Codes are not unique: both :northern_uzbek and :uzbek return :uz, so use result.language when you need to tell them apart.

A command line executable, vendored MIT models, and a licence to read

Installing the gem also installs a whatlanguage executable that reads files given as arguments or from stdin and prints the ISO 639 code, or und when undetermined.

$ whatlanguage README.md
en
$ echo "Wie geht es dir heute?" | whatlanguage
de

The command line uses the same language code mapping as the API, so a script and a service agree on codes. On licensing, the trigram profiles in lib/whatlanguage/trigrams.json are taken from whatlang, an MIT licensed project by Sergey Potapov, which is itself a derivative of Franc by Titus Wormer, also MIT. Those profiles are derived from the public-domain Universal Declaration of Human Rights translations. The repository itself carries a LICENSE.txt file that GitHub cannot classify, so read it directly before you ship it, and the vendored MIT attribution in the README is a separate obligation from whatever the gem's own terms are. The repository is not archived, the last push was on 2026-10-19, and the codebase is small: a gemspec, a Rakefile, a Gemfile, a .rubocop.yml, an exe directory, lib and test directories.

Editorial conclusion

whatlanguage fits a Ruby application that needs to route or tag user text by language, where a nil answer is acceptable and the alternative is a service call, since it has no runtime dependencies and ships its model in the gem. It does not fit code that needs confidence scores, calibrated probabilities, or reliable detection of short strings, because scores are ranks and statistical detection returns nil below ten letters by default. Verify first that the trigram profiles cover the languages you need, read the LICENSE file directly since GitHub cannot classify it, and remember that the profiles are vendored MIT work from whatlang and Franc rather than original to this repository.

Frequently asked questions

Official sources

1. Issues
2. peterc/whatlanguage on GitHub
3. README
4. Releases