Rankify: A Python Toolkit for Retrieval, Re-Ranking and RAG Pipelines

What Rankify is trying to fix in retrieval experiments

Retrieval research has a plumbing problem. A paper compares a new re-ranker against a baseline, but the two runs use different corpora, different candidate depths, different tokenisation and different evaluation scripts. Reproducing the comparison means rebuilding the whole stack. Rankify's answer is to fix the corpus and the candidate set in advance: the README states that the toolkit integrates 40 pre-retrieved benchmark datasets, so the first-stage retrieval output is already on disk and every re-ranker sees the same candidates. That is the design decision the rest of the project hangs on. It removes the retriever from the comparison, which is exactly what you want when the object of study is the re-ranking stage, and exactly what you do not want when the object of study is recall at the retrieval stage. The audience follows from that: researchers and practitioners who need to benchmark retrieval and re-ranking pipelines, as the project description puts it, and who are comfortable with an academic artefact that ships as a pip package.

The pipeline API, the agent, and the server as three entry points

The README's navigation lists four ways in. The one-line pipeline API is marked as recommended and is the shortest path: you construct a pipeline, hand it a query, and get back retrieved and re-ranked documents, with a generator optionally attached at the end. RankifyAgent is described as AI-powered model selection, which suggests an LLM choosing which retriever or re-ranker to apply rather than a fixed configuration; the README does not spell out the selection criteria, so treat that as the least documented surface. Rankify Server deploys the same pipeline as a REST API, and the Web Playground is a Streamlit demo. The demo instructions are concrete: install Streamlit with pip install streamlit, then run streamlit run demo.py from the repository. The architecture that emerges is a chain of swappable stages (retrieve, re-rank, generate) with a fixed dataset layer underneath. The cost of that modularity is that each stage has its own dependency group, which is why the October 2025 release split installation into extras.

Installing Rankify with the right extras and the right PyTorch

The README is specific about the environment. Create a conda environment on Python 3.10, activate it, then install PyTorch 2.5.1, torchvision 0.20.1 and torchaudio matching that version. The badge in the README advertises Python 3.10 and 3.11, so 3.12 is outside what the project states it supports. The installation section recommends the CUDA 12.4 or 12.6 builds of PyTorch because, in the README's words, many of the evaluation metrics are optimized for GPU use; on a CPU-only machine the metrics still run but you should expect that path to be slower. The October 2025 release notes introduce optional extras named retriever, reranking, rag and all, which means a bare pip install rankify does not pull in every model family. If you plan to use the dense retrievers added in the 2026 update, including the LLM-based bi-encoders and the reasoning-augmented models, install the retriever extra rather than assuming they arrive by default. Nothing in the supplied material gives a minimum disk or RAM figure for the full dataset collection, so size your machine against the dataset you actually download.

Indexing via the rankify-index CLI

For corpora that are not already pre-retrieved, the project ships a CLI. The release notes for October 2025 describe a new rankify-index syntax with examples covering BM25, DPR, ANCE, Contriever, ColBERT and BGE, and a June 2025 contribution added indexing for custom datasets. That gives you two working modes: consume one of the 40 pre-retrieved benchmarks, or build your own index with rankify-index and run the same downstream stages over it. The CLI is the part of the project where the documentation has moved fastest, and the README points to the Read the Docs site for the current syntax rather than reproducing every flag inline. If you are building a custom index, read the CLI section on the documentation site before copying any example from an older release, because the syntax changed between versions.

Where Rankify stops being the right tool

The pre-retrieved dataset design is a constraint, not a convenience, once you leave the benchmark setting. If your corpus changes daily, the fixed candidate sets are stale the moment they are written, and you are back to running rankify-index on every update. The project also carries a hard version pin: PyTorch 2.5.1 with Python 3.10 or 3.11. If your environment is already on a newer PyTorch because another model in your stack requires it, you are choosing between two incompatible dependency trees. The licence is a second open question. The README badge points at Apache-2.0 and links to opensource.org, but the repository metadata supplied here lists the licence as unknown, and the README's License section is a heading with no text visible in the material. Until you open the LICENSE file yourself, do not assume the badge is the whole story, and note that the 40 benchmark datasets come from third parties with their own terms. Finally, the project is a research toolkit maintained by contributors at a university group; the release cadence visible in the material is three releases across 2025 and a push in September 2026, which is not the rhythm of a vendor-backed product.

How Rankify differs from building on Haystack or LlamaIndex

Haystack and LlamaIndex both give you retrieval and generation pipelines, and both expect you to bring your own corpus and index it yourself. Rankify's difference is the dataset layer. It ships 40 pre-retrieved benchmark collections on Hugging Face, in a full and a light variant, so a re-ranking experiment starts from a known candidate set rather than from whatever your indexer produced that morning. That makes cross-paper comparison cheaper and makes the toolkit a poor fit for a document question-answering service over private files, where the corpus is the thing you cannot share. The other difference is the re-ranker count: the README claims 24 or more state-of-the-art re-ranking models behind one interface. If your work is specifically about scoring functions over a fixed candidate list, that breadth is the reason to look here. If your work is about chunking strategy or ingestion, the pre-retrieved datasets are irrelevant to you and a general framework is the better starting point.

Maintenance burden and what to verify before you commit

Two costs deserve attention. The first is the dependency pin. Holding PyTorch at 2.5.1 and Python at 3.10 or 3.11 while the rest of your stack moves means periodic conflict resolution, and the extras split means you may be installing retriever, reranking and rag separately, each with its own transitive dependencies. The second is the licence ambiguity described above. Apache-2.0 permits commercial use with attribution and a notice of changes, but that is a statement about the badge, not about the repository, and the supplied metadata does not confirm it. Before you build on Rankify, do three things: open the LICENSE file at the repository root and read it, check the terms attached to the specific Hugging Face dataset you intend to use, and run one end-to-end pass on a small benchmark split to confirm that the pipeline API, the rankify-index CLI and your installed PyTorch version agree with each other. The project's own documentation lives at rankify.readthedocs.io and is the place to check the current CLI flags, since the README's examples have changed between releases.

Editorial conclusion

Rankify fits researchers and graduate students who need to run the same query set through several retrievers and re-rankers and compare the scores, and who can accept a Python 3.10 or 3.11 environment pinned to PyTorch 2.5.1. It does not fit teams that need a supported production search service with an SLA, or anyone unwilling to install the retriever, reranking and rag extras separately. Before adopting it, check the LICENSE file in the repository root against the Apache-2.0 badge, and confirm that the benchmark datasets you need are in the Hugging Face collections the README links to rather than only in the full dataset release.

Official sources

1. DataScienceUIBK/Rankify on GitHub
2. Issues
3. Project website
4. README
5. Releases