HanLP

A production oriented natural language toolkit running on PyTorch and TensorFlow 2.x dual engines, with pretrained models across a dozen plus tasks and corpora the project calls among the largest.

HanLP runs on PyTorch and TensorFlow 2.x dual engines and describes itself as production oriented, with the stated mission of popularizing state of the art NLP technology. The self assessment runs feature complete, accurate, efficient, with up to date corpora, clear architecture, and customization. Python under Apache-2.0, homepage hanlp.com, around 36,500 stars.

HanLP 2.1 supports 10 joint tasks and multiple single tasks across 130 languages, including simplified and traditional Chinese, English, Japanese, Russian, French, and German, drawing on what the project calls one of the largest multilingual corpora in use. Numbers like that are unusual for an open toolkit and are the first thing that sets this one apart.

Word segmentation, part of speech tagging, named entity recognition, dependency parsing, constituency parsing, semantic dependency parsing, and semantic role labeling. Each row lists the available model, the annotation standard, and tutorials for RESTful, multi task, and single task usage, so the table doubles as a getting started map.

The lightweight RESTful API is described as a few kilobytes, suitable for agile development and mobile apps, with fast installation and no GPU environment needed. Native APIs handle larger scale scenarios, interfaces are semantically consistent across languages, and the code is open source all the way down.

Pretrained models cover more than ten tasks and are continuously iterated, per the README. The feature table cites annotation standards including CTB, PKU, and OntoNotes for the Chinese tasks. Attention to standards like that is what makes the models usable in comparative work rather than only in demos.

• Source 1 · Project repository
• Source 2 · README