bnlp bundles Bengali tokenizers, embeddings, POS, NER, and spell checking behind one import

Three tokenizer backends sit behind one import

Tokenization is the entry point of the toolkit and the first place the design shows its hand: there are three of them, not one.

`BasicTokenizer` is built in and needs nothing external. NLTK is offered as an alternative, which means the NLTK dependency is a real runtime requirement rather than an example. Sentencepiece is the third, and it is a subword tokenizer rather than a word splitter.

The documented choice happens in Python by importing the class you want:

from bnlp import BasicTokenizer

In the command line it happens with a flag, `bnlp tokenize` with `--type nltk` to switch backend and `--sentence` for sentence segmentation rather than word segmentation. The same `tokenize` command also takes `--json` when you want output you can pipe into something else.

Supporting all three is a deliberate answer to a real problem in Bengali. Scriptio continua, the absence of spaces between words in running text, means that a whitespace tokenizer is not enough, and each of these approaches fails differently on punctuation, on loanwords written in Latin script, and on compound verbs. Being able to switch backend and compare is more useful than committing to one.

The sample output in the documentation shows the basic tokenizer splitting a Bengali sentence into words plus the final punctuation mark as its own token, which is the kind of detail worth checking against your own expectations before building on it.

The CLI covers every feature, including downloading the models

The command line interface exists so you can inspect behaviour without writing Python, and it is more complete than a demo script usually is.

Eight commands are listed: `tokenize`, `ner`, `pos`, `embedding`, `clean`, `download`, `list-models`, and `corpus`. The first five are the analysis commands, and the last three are the housekeeping ones, which is the unusual part. Most tools leave model acquisition out of the CLI; here you can list what is available and download what is missing.

The options are as documented:

bnlp list-models

`bnlp download all` fetches every model and `bnlp download word2vec` fetches one, `bnlp corpus stopwords` and `bnlp corpus letters` read the bundled corpus data, and `bnlp clean` takes cleaning flags such as removing email addresses from a string.

For embeddings there is a similarity mode, `bnlp embedding` with `--similar`, and the number of neighbours is configurable with `--topn`, which is the flag to reach for when you want to sanity check what a word vector has learned.

The entry point is declared in the packaging as a console script, `bnlp` mapped to `bnlp.cli:main`, so the command name is stable across installs rather than depending on what you invoke by hand.

Models are downloaded, not bundled, and the corpus is a command

The separation between code and data is explicit. The toolkit is a pip package, and the pre-trained models are obtained through the `download` command rather than shipped inside it.

The repository layout reflects that, with a `model/` directory in the tree alongside the `bnlp/` package, a `notebook/` directory for examples, and a `tests/` directory. The published artefact therefore stays small and the weights stay fetchable.

Corpus resources are a separate category again. The documentation lists letters, vowels, punctuations, and stopwords as a corpus class, and the CLI exposes them through `bnlp corpus`, with stopwords and letters given as examples and digits mentioned in the command table. Stopwords in particular are the kind of list you want to inspect rather than accept, because a list built for one corpus can quietly remove words you need for a different one.

The practical sequence for a new machine is therefore: install the package, run `bnlp download all` or fetch a single model, and only then import the class. Calling a component before its model exists is the obvious failure mode of this design, and `bnlp list-models` is how you check for it.

Four embedding flavours, and two of them live in extras

Embeddings are split into word vectors and document vectors, with three word vector sources and one document method.

Word2vec is the default path. Fasttext is second, and it is an optional extra rather than a base dependency. GloVe is third. Doc2vec covers documents rather than words, which is the one to reach for when the unit you want to embed is a paragraph or a page rather than a term.

The extras declare what is genuinely optional:

from bnlp import BengaliNER

The `fasttext` extra pulls `fasttext`, the `spellcheck` extra pulls `symspellpy`, `langdetect` also pulls `fasttext`, and `all` pulls both of the first two. So a language detection install quietly gets the fasttext library, which is worth knowing when you are reasoning about install size or about a build that has to compile native extensions.

The base install list is still substantial: sentencepiece, gensim, numpy, scipy, sklearn-crfsuite for the CRF models, tqdm, ftfy for text repair, emoji, requests, nltk, and symspellpy. The Python floor is 3.8 and the documented tested versions run from 3.8 through 3.13 on Linux, Windows, and Mac.

Spell checking is described as fast and accurate using the SymSpell algorithm, exposed as `BengaliSpellChecker`.

Four ways to load a model, for four different situations

Model loading is the part of the API with the most options, which tells you that loading a Bengali model is slow enough to matter.

`AsyncModelLoader` takes the model class plus callbacks, with an `on_progress` handler receiving a progress object whose fields you format yourself, and an `on_complete` callback. You call `start_loading()` and then do other work, and later call `get_model()`, which blocks until the model is ready.

`LazyModelLoader` is the simpler case: construct it with the class and call `get()`, and the load happens on first access. Nothing is loaded until you actually need it.

`load_model_async` is the one-liner version, returning a loader you can call `get_model()` on when you are ready.

`BatchProcessor` is the fourth, and it is about throughput rather than about the model. You give it a function and a `max_workers` value, then call `process` with a list and `show_progress=True`. The batch helpers `tokenize_batch`, `tag_batch`, and `clean_batch` cover the common cases where you already have a tokeniser or tagger instance and only need the loop.

Between them these cover a CLI one-shot, a notebook exploring interactively, and a server that wants the model warm before the first request.

Pipelines are hand-assembled or pre-built, and can return intermediates

Operations can be chained either way, which is the difference between a library and a collection of functions.

The hand-assembled form takes a list of components in order:

from bnlp import Pipeline, CleanText, BasicTokenizer

`CleanText` takes keyword flags such as removing URLs and punctuation, and `BasicTokenizer` follows it. The pipeline is then callable on a string and returns the final result.

The detail worth knowing is `run` with `return_details=True`, which exposes intermediate outputs. In a two-stage pipeline that means you can see the cleaned string as well as the tokens, which is exactly what you need when a result looks wrong and you cannot tell whether cleaning or tokenisation caused it.

The pre-built factories save the argument list. `create_tokenization_pipeline` takes `clean` and a `tokenizer_type`, `create_ner_pipeline` takes `clean`, and `create_pos_pipeline` takes `clean`, so the common chains are one call each.

The POS and NER models are both CRF based, which is a deliberate choice for a low-resource language: a linear chain CRF with sklearn-crfsuite is easier to train on limited labelled data than a transformer, and the accuracy claim is left to the documentation rather than promised in the interface.

setup.py is still the build path and the floor is Python 3.8

The packaging is plain setuptools, which is worth knowing if you maintain a fork.

`setup.py` calls `setuptools.setup` with the name `bnlp_toolkit`, version 4.4.1, and reads the long description from README.md as UTF-8 markdown. Packages are discovered with `find_packages()`, so the directory layout is the package layout.

The declared Python floor is 3.8, matching the documentation which lists 3.8 through 3.13 as supported, and the classifiers are minimal: Python 3, MIT licence, and operating system independent. So the operating system claim is a classifier rather than a tested matrix, even though the readme names Linux, Windows, and Mac.

Install paths are the usual pair. `pip install bnlp_toolkit` from the package index, `pip install -U bnlp_toolkit` to upgrade, or from source with a clone followed by `python setup.py install`.

The entry point is the one that makes the CLI work after any of them:

bnlp corpus letters

It is declared as a console script pointing at `bnlp.cli:main`, so installing by any route gives you the same `bnlp` command.

Editorial conclusion

bnlp fits someone working on Bengali text who wants pretrained models and a working pipeline in one dependency rather than assembling each component, and the CLI makes it easy to check outputs before writing any code. It does not fit a project that needs the pipeline to stay offline without a download step, since models are fetched rather than bundled. Before adopting it, decide whether you need fasttext, because it is an optional extra, and check that language detection also depends on that extra even though its name does not suggest it.

Frequently asked questions

Official sources

1. License: MIT
2. Project website
3. README
4. Releases
5. sagorbrur/bnlp on GitHub