rank_bm25: Small-Corpus BM25 Ranking in Python

What rank_bm25 solves

rank_bm25 addresses document ranking inside an existing Python application. It takes a collection of documents that have already been divided into tokens, builds a BM25 index from those tokens, and returns relevance scores for a tokenized query. The intended users are developers who need lexical ranking without implementing several BM25 variants themselves, as well as readers who want a direct Python example of an information-retrieval ranking method.

The package supports Okapi BM25, BM25L, and BM25+. BM25-Adpt and BM25T appear in the README as unimplemented items, so the advertised algorithm set is not unlimited. The package does not fetch documents, build a web service, expose a network API, or provide a complete search application. It ranks a corpus already held by the caller.

Its scope is intentionally narrow. The README describes the package as a collection of algorithms for querying documents and returning the most relevant ones. That makes it useful when lexical ranking is one stage inside a larger system, such as a prototype, notebook, internal tool, or educational project. It does not remove the difficult parts of search outside scoring, including ingestion, text normalization, index persistence, distributed execution, and relevance evaluation.

How the token and score flow works

The core data flow is visible in the public API. The caller first supplies a list of lists of strings to a BM25 class. Each inner list is one tokenized document, and the outer list represents the corpus. Constructing `BM25Okapi` performs the indexing work needed for later scoring. The README describes this as creating the document indexes, but it does not document the internal data structures or scoring equations.

A query follows the same token path. The caller splits or otherwise tokenizes the query, then passes that token list to `get_scores`. The result is an array containing one score per document, in the same order as the corpus. This design avoids a hidden tokenizer, which is useful when the application already has domain-specific tokenization rules. It also means that a mismatch between document preprocessing and query preprocessing can produce poor results.

The package exposes two retrieval styles. `get_scores` returns the score array, while `get_top_n` returns the highest-ranked documents according to the supplied corpus and `n` value. The README example uses `get_top_n` with `n=1`, producing a one-item list containing the document about wind in London. The package does not state whether scores can be compared across separately constructed indexes, so scores should normally be interpreted within the index created from the current corpus.

Installation and first rank_bm25 query

The README gives `pip` as the easiest installation route. The package declares `numpy` as its only runtime dependency, so the environment also needs that dependency installed and compatible with the Python version in use. The README does not document Python version support or platform-specific installation behavior.

This command installs the package from the named package index source:

pip install rank_bm25

The README also documents a direct installation from GitHub over SSH:

pip install git+ssh://[email protected]/dorianbrown/rank_bm25.git

The direct form requires access to the repository through the configured SSH environment. The README says it is useful when the newest version is wanted, but it does not provide a command for selecting a particular release.

A first real use follows the README's `BM25Okapi` example. The corpus is split by spaces, and the resulting token lists are passed to the ranker:

from rank_bm25 import BM25Okapi

corpus = [
    "Hello there good man!",
    "It is quite windy in London",
    "How is the weather today?"
]

tokenized_corpus = [doc.split(" ") for doc in corpus]
bm25 = BM25Okapi(tokenized_corpus)

The query must use the same tokenization decisions:

query = "windy London"
tokenized_query = query.split(" ")
doc_scores = bm25.get_scores(tokenized_query)

The README's example shows scores in corpus order, with the London document receiving the nonzero score shown there. To retrieve the best document directly:

bm25.get_top_n(tokenized_query, corpus, n=1)

That call returns the single top document from the original corpus. The result is a basic lexical ranking path, not a search service.

Preprocessing decisions belong to the application

The most important integration constraint is stated directly in the README: rank_bm25 does not perform text preprocessing. Lowercasing, stopword removal, stemming, and related operations must be implemented by the application. The same preprocessing must be applied to documents and queries.

This is a design boundary rather than a minor omission. For example, splitting on spaces leaves punctuation attached to tokens and makes capitalization significant. A case-sensitive index will not treat `London` and `london` as the same term. The caller can normalize text before `doc.split(" ")`, but that decision changes both the index and the query path. The package's examples therefore demonstrate token plumbing, not a complete production text pipeline.

The requirement that the class receive a list of lists of strings is important. Passing raw strings instead of token lists does not match the documented interface. Empty documents, unusual whitespace, punctuation, multilingual tokenization, and domain abbreviations are not discussed in the README. Their behavior should be treated as unverified until covered by application tests.

The package also does not document stemming, stopword lists, phrase matching, synonym expansion, field weighting, or semantic retrieval. Those capabilities would need to be added around it, or supplied by another retrieval system. This keeps the package small and understandable, but transfers responsibility for retrieval quality to the caller.

Scale, maintenance, and upgrade considerations

rank_bm25 is intended for small in-memory collections rather than a full production retrieval service. The README explicitly points large-scale production users toward `retriv`, describing it as a more performant Python retrieval package. That recommendation is the clearest documented limitation and an important architectural difference: rank_bm25 focuses on BM25 scoring classes, while the alternative is presented as a retrieval package designed for larger workloads.

The repository is not archived, and its last push was on 2026-05-02. The README's release history lists `0.2.2` from 2022-02-16 and `0.2.1` from 2021-02-08. These facts indicate a small project with infrequent published releases and a much later repository push than the listed releases. The README does not provide a compatibility policy, migration guide, changelog, or supported-version matrix. Upgrading therefore requires reviewing the package and application tests rather than relying on documented release procedures.

The repository declares the `Apache-2.0` license, and its setup metadata identifies the package license as `Apache2.0`. That designation indicates the project's declared licensing terms, but this review does not provide legal advice. Teams still need to review the repository `LICENSE` file and dependency terms for their own distribution and use. The package depends on `numpy`, so upgrades may also affect that dependency and the application's numerical environment.

When rank_bm25 is the wrong tool

rank_bm25 is the wrong choice when the requirement is a scalable search platform rather than a local ranking function. The README specifically recommends considering `retriv` for large-scale production environments and describes it as more performant. It does not document distributed indexing, persistence, incremental updates, concurrency behavior, or service interfaces in rank_bm25, so none of those capabilities should be assumed from the package name or from the existence of its scoring methods.

It is also a weak fit for applications whose query and document pipelines require built-in normalization. Because preprocessing is explicitly excluded, an application that expects the library to lowercase, stem, remove stopwords, or parse punctuation must supply those operations itself. An application needing semantic similarity, embeddings, synonym expansion, phrase constraints, or relevance tuning beyond BM25 will need another component.

The documented API is also oriented toward a corpus already represented in memory. Large collections may create memory and indexing costs that the README does not quantify. Since no benchmark, memory profile, or throughput guidance is supplied by the repository, an engineer should not use the package name to infer production capacity. The first real check is whether the intended corpus and query volume fit the caller's memory and indexing constraints.

Editorial conclusion

rank_bm25 is appropriate for engineers building a small local search feature, a ranking experiment, or a teaching example around BM25. It is a poor fit for large-scale production retrieval unless another layer handles preprocessing, indexing, and query workloads. Before adoption, verify that your documents can remain in memory, that tokenization is consistent across indexing and querying, and that BM25 scores meet the application's relevance requirements. For the package's documented small-corpus workflow, `pip install rank_bm25` followed by `BM25Okapi(tokenized_corpus)` is the first implementation boundary to confirm.

Frequently asked questions

Official sources

1. dorianbrown/rank_bm25 on GitHub
2. Issues
3. License: Apache-2.0
4. README
5. Releases