soynlp: unsupervised Korean word, noun and token extraction from raw text

soynlp learns from raw text instead of a labelled corpus

The core claim of the project is that Korean analysis can be done without training data, and the description states the intent plainly: it pursues an unsupervised learning approach that finds words present in the data, splits sentences into word sequences, and does part of speech tagging without learning data. The English name given in setup.py is Unsupervised Korean Natural Language Processing Toolkits. That framing has an immediate consequence for how you use it. The WordExtractor and the NounExtractor work from statistical information learned across many documents, so the quality of their output depends on the corpus you feed them. The project is explicit about the failure mode: unsupervised approaches that use statistical patterns to extract words work well on a reasonably large collection of homogeneous documents, such as movie comments or one day of news articles, and heterogeneous document sets gathered together produce poor extraction. The practical reading is that soynlp rewards you for scoping the corpus narrowly, and punishes you for dumping a general web crawl into it.

WordExtractor scores boundaries with cohesion, branching entropy and accessor variety

The word extraction mechanism is where the interesting statistics live. The project explains the motivation through the out of vocabulary problem: entertainment news from October 2016 contains newly popular group names and English loanword phrases that a corpus-trained part of speech tagger or morphological analyzer has never seen, because new words are constantly created. The project defines a word as a frequently occurring sequence of words in a document set, and offers three statistical measures of word boundary: Cohesion score, Branching Entropy and Accessor Variety. Cohesion is the probability that the right side follows the left, branching entropy measures how varied the left and right contexts are, and accessor variety counts distinct left and right neighbours. The API reflects that split, since a WordExtractor takes a min_frequency and separate min_cohesion_forward and min_right_branching_entropy thresholds, and the resulting dictionary maps each extracted word to a Scores namedtuple carrying cohesion_forward, cohesion_backward, left and right branching entropy, left and right accessor variety, and leftside and rightside frequency. The output is therefore a ranking, not a yes or no decision, and you choose the threshold yourself.

The extractor is constructed with the thresholds you care about, trained on a list of sentences, and then asked for its word set.

from soynlp.word import WordExtractor

word_extractor = WordExtractor(min_frequency=100,
    min_cohesion_forward=0.05, 
    min_right_branching_entropy=0.0
)
word_extractor.train(sentences) # list of str or like
words = word_extractor.extract()

NounExtractor adds left-right structure to decide which words are nouns

The word extractor is deliberately limited in one direction: it learns boundary scores and nothing more, so it cannot tell you the part of speech of anything it finds. The noun extractor adds that on top, using a linguistic observation about Korean. The project notes that new words are created most often in nouns, and that specific particle characters frequently follow nouns, for example the endings that mark a subject or object, the quotative ending and the connective ending. If you look at the distribution of characters appearing to the right of a left-positioned substring within a document's space-delimited units, you can judge whether that substring is a noun. Three extractor generations exist, and the project states that v2 shows the best performance, being the version that fixed the accuracy and compound noun recognition of the earlier version and the errors in the output information. The v2 extractor returns a dictionary of namedtuples, and the printed example shows the shape of the data, with the frequency and a score for each noun. It also keeps a compound breakdown, so a long compound noun is recorded with its components, while words that are morphologically compound but behave like single nouns in practice are treated as single nouns.

The v2 extractor is used like this, taking a corpus directory of one document per line and iterating sentences from it.

from soynlp.utils import DoublespaceLineCorpus
from soynlp.noun import LRNounExtractor_v2

corpus_path = '2016-10-20-news'
sents = DoublespaceLineCorpus(corpus_path, iter_sent=True)

noun_extractor = LRNounExtractor_v2(verbose=True)
nouns = noun_extractor.train_extract(sents)

The parameter renaming at 0.0.47 will break code that sets thresholds by name

There is a migration note in the documentation that deserves attention before anything else, because it silently changes an API. Up to soynlp 0.0.46 there was no rule for naming parameters that take a minimum or a maximum, and names such as min_score, minimum_score and l_len_min were used inconsistently. From 0.0.47 onward the names were shortened so that min and max carry the meaning, followed by the item and the threshold type, following a pattern of {min, max}_{noun, word}_{score, threshold}, with the parts omitted when the item is obvious. Frequency related parameters are now called frequency rather than count, index and idx are both written idx, and num and n are unified as num. The project frames this as a deliberate early change to reduce future inconvenience, and it is kind of the maintainer, but the practical effect is that any code that passed thresholds by keyword argument against an older version will raise an error rather than warn you. Read your own call sites before upgrading. The same section notes that soynlp often does substring counting, which is the reason the frequency naming matters for how you interpret the numbers.

The install step is a single pip command, and the documentation gives no other setup path.

$ pip install soynlp

Three tokenizers split a string where the statistics say a word ends

Once you have a set of learned words, the tokenizers apply it. The documentation lists LTokenizer, MaxScoreTokenizer and RegexTokenizer as the three options, and the underlying logic differs in how the split point is chosen. LTokenizer is the model that learns a left and right boundary score for each candidate and looks for a locally optimal split, which is why the noun extractor exposes the same structure through an LRGraph, where get_r and get_l return the observed right and left characters that follow a given space-delimited unit with their counts. The example output for a group name shows the right side distribution, dominated by a few common particles with small counts for rarer ones, which is exactly the signal the noun extractor reasons over. MaxScoreTokenizer picks the split with the maximum score, and RegexTokenizer falls back to pattern based splitting for text the model has not seen. The presence of three options matters practically, because Korean text mixes with English, numbers and punctuation, and no single strategy handles that mixture well.

A POS tagger, a vectorizer and a normalizer round out the package

Beyond extraction and tokenization, the project ships a part of speech tagger, a vectorizer and a normalizer. The tagger is backed by dictionary files that ship inside the package, since setup.py includes pos/dictionary/*.txt and postagger/dictionary/default/*/*.txt in package_data, along with trained_models/*, noun/*.txt and lemmatizer/dictionary/default/*/*.txt, so the models and lexicons are installed alongside the code rather than downloaded at runtime. The vectorizer turns a corpus into document vectors for downstream machine learning work, and the normalizer handles the non spacing conventions of Korean text. The dependency list in setup.py is worth noting: numpy, psutil, scipy and scikit-learn with lower bounds of 1.12.1, 5.0.1, 1.1.0 and 0.20.0 respectively, which tells you the library leans on scientific Python rather than on anything Korean specific.

Licence, releases and the gap between the code and the package

There is an unresolved detail here that you have to check yourself. The repository records the license as a custom licence that GitHub cannot classify, with a LICENSE file at the root, while the setup.py classifiers list the project under the GNU General Public License v3. Those two statements are not the same claim, and the package metadata is the part that downstream tools read when they check what they are allowed to do with the code. Before shipping anything built on soynlp, read the LICENSE file directly rather than trusting either signal. On maintenance, the last push to the repository was on 2026-10-04, so the code is being worked on, and the two published releases, 0.1.0 and 0.1.1, are both dated 2024-04-13, minutes apart. The gap between a repository that receives commits and a package with no release since 2024 is worth noticing, because a pip install gives you whatever was last published rather than the current state of master. The repository also carries notes/, tutorials/, data/, a test directory and a setup.py, so the tests and the written explanations ship with the source even when the packaged wheel does not expose them the same way.

Editorial conclusion

Adopt soynlp if you are working with Korean text, you have a decent corpus of documents from one domain, and you need to discover words that a trained tagger has never seen, which is exactly the out of vocabulary problem the project set out to solve. Do not adopt it for a mixed or tiny corpus, since the project states that heterogeneous documents train poorly and that a single sentence or document is not enough. Before you build on it, read the LICENSE file yourself, because the package metadata classifies the project as GPLv3 while the repository records a custom licence, and check that your numpy, scipy and scikit-learn versions still satisfy the lower bounds listed in setup.py, since those bounds are fairly old lower limits.

Frequently asked questions

Official sources

1. Issues
2. lovit/soynlp on GitHub
3. README
4. Releases