jieba

Four modes in one module: precise, full, search engine, and a paddle mode built on PaddlePaddle. jieba uses a prefix dictionary, dynamic programming, and an HMM with Viterbi for words it has never seen.

Jieba is the Chinese word for to stutter, and the README leans into the pun: it calls the module Chinese text segmentation, built to be the best Python Chinese word segmentation module. The project is Python under the MIT license and has been around since 2012. Four segmentation modes form the core story, and each is explained with its tradeoff.

Precise mode tries to cut a sentence into the most accurate pieces, which suits text analysis. Full mode scans out every possible word, fast but unable to resolve ambiguity. Search engine mode starts from precise mode and re splits long words to raise recall, aimed at search engine segmentation. Paddle mode uses a PaddlePaddle deep learning framework with a bidirectional GRU network for sequence labeling, and supports part of speech tagging, but it needs paddlepaddle-tiny and jieba 0.40 or newer.

Under the hood the README describes a prefix dictionary for efficient word graph scanning, building a directed acyclic graph of all possible word combinations. Dynamic programming finds the most probable path based on word frequency, and for words not in the dictionary an HMM model with the Viterbi algorithm steps in. The README also prints speed figures: 1.5 MB per second in full mode and 400 KB per second in default mode on a 3.4GHz i7 with a Chinese novel as the test corpus.

jieba recognizes new words on its own, but the README encourages custom dictionaries for higher accuracy. You can load a user dictionary in the same format as dict.txt, one word per line with optional frequency and POS tag. add_word, del_word, and suggest_freq let you adjust the dictionary from within a program, and a custom Tokenizer can run multiple dictionaries at once. The prefix dictionary loads lazily, taking one to three seconds the first time.

Beyond cutting, the module does more. Keyword extraction is available through TF-IDF with extract_tags, set_idf_path, and set_stop_words, and through TextRank with a similar interface. Part of speech tagging lives in jieba.posseg using labels compatible with ictclas. Parallel processing splits text across multiple Python processes with enable_parallel, and the README claims a 3.3x speedup on a four core machine, though it does not support Windows. A ChineseAnalyzer bridges jieba into the Whoosh search engine.

• Source 1 · Project repository
• Source 2 · README