# Rankify: a Python toolkit that puts retrieval, reranking and RAG behind one pipeline object

> Rankify bundles 40 pre-retrieved benchmark datasets, several sparse and dense retrievers, more than 20 reranking models and a generator layer behind a single Pipeline call. It is aimed at researchers who need to compare retrieval stacks on the same data, not at teams shipping a production search box.

**DataScienceUIBK/Rankify** — 🔥 Rankify: A Comprehensive Python Toolkit for Retrieval, Re-Ranking, and Retrieval-Augmented Generation 🔥. Our toolkit integrates 40 pre-retrieved benchmark datasets and supports 7+ retrieval techniques, 24+ state-of-the-art Reranking models, and multiple RAG methods.

- Repository: https://github.com/DataScienceUIBK/Rankify
- Website: https://rankify.readthedocs.io/
- Stars: 684 · Forks: 71
- Language: Python
- License: not declared
- Published: 2026-09-10 · Updated: 2026-09-10 · Language: en
- Canonical page: https://hysenlabs.com/projects/datascienceuibk-rankify

## What Rankify is for, and who it is not for

Rankify targets a specific kind of work: comparing retrieval and reranking configurations on shared benchmarks. The README describes it as a modular retrieval, reranking and RAG framework, and the project ships 40 pre-retrieved benchmark datasets plus Hugging Face mirrors named reranking-datasets and reranking-datasets-light. That combination is the point. If you are writing a paper or an internal evaluation where BM25, DPR, ANCE, Contriever, ColBERT and BGE need to be scored on identical inputs, the pre-retrieved datasets remove the step where every comparison starts from a different corpus state.

The audience is narrower than the badge list suggests. Rankify is a research toolkit with an arXiv paper (2502.02464), example scripts under examples/, and a Read the Docs site. Nothing in the README describes an SLA, a migration path between releases, or a hosted service. A team that needs a search endpoint with an on-call rotation is looking at the wrong project. A team that needs to answer the question "does reranker A beat reranker B on this dataset" is looking at the right one.

## The Pipeline API and the module stack underneath it

The README recommends a one-line Pipeline API, and the repository layout backs that up: rankify/ holds the package, examples/ holds per-component scripts such as retreiver.py, reranking.py and generator_examples/, and the CLI entry point is declared in pyproject.toml as rankify-index = "rankify.cli.cli:main". So there are three ways in: the Pipeline object, the individual retriever, reranker, generator and evaluation modules, and the rankify-index command.

The dependency split is the clearest statement of the architecture. Core dependencies are deliberately small: pandas, transformers 4.45.2, datasets 3.2.0, sentence_transformers 3.3.0, onnxruntime, sentencepiece. Everything heavy sits in optional extras. The reranking extra pulls vllm 0.7.0, llm-blender, peft, llama-cpp-python, together and cohere. A retriever extra exists for sparse retrieval, and the release notes for 2025-10-14 describe adding extras named retriever, reranking, rag and all. That means a sparse-only experiment does not have to install vLLM, and a reranking-only experiment does not have to install the retriever stack. It also means the "24+ reranking models" figure is not something you get from a base install.

Reranking itself is not one algorithm. The extra pulls in vLLM for local LLM serving, llm-blender for rank fusion, llama-cpp-python for quantized local inference, and hosted clients for Together and Cohere. Those are four different execution models behind one interface, and the choice between them is mostly a hardware and cost decision rather than a quality one.

## Installing Rankify and running a first retrieval

The README gives a conda-first install and pins Python 3.10. The package metadata is slightly wider, declaring requires-python >=3.10 with classifiers for 3.10, 3.11 and 3.12, so 3.11 and 3.12 are declared but the documented path is 3.10.

```bash
conda create -n rankify python=3.10
conda activate rankify
```

The README recommends PyTorch 2.5.1 and points at the PyTorch previous-versions page for platform-specific commands. It notes that CUDA 12.4 or 12.6 builds are preferable if you have GPUs, because many evaluation metrics are optimized for GPU use. The README's own pip line is truncated mid-package-name in the text available here, so use the PyTorch page rather than copying a partial command.

After PyTorch, install the package with the extras you need. The 2025-10-14 release notes name four: retriever, reranking, rag and all. The optional dependency groups in pyproject.toml are the source for those names.

```bash
pip install streamlit
streamlit run demo.py
```

That is the README's own demo path: install Streamlit, then run demo.py, which starts the local web playground. For indexing, the CLI entry point is rankify-index, and the 2025-10-14 notes list BM25, DPR, ANCE, Contriever, ColBERT and BGE as the retrievers with CLI syntax and examples. The repository also carries examples/indexing.py and examples/indexing_demo.py, plus examples/custom_datasets/ for indexing a custom dataset, which was contributed in June 2025. Start from those files rather than reconstructing flags from the README, because the CLI examples live in the release notes and the docs site.

## The install is heavier than the core dependency list suggests

The honest limitation is dependency weight and version pinning. Core dependencies are pinned exactly (pandas==2.2.3, transformers==4.45.2, datasets==3.2.0, sentence_transformers==3.3.0, onnxruntime==1.19.2, sentencepiece==0.2.0), and the reranking extra pins vllm==0.7.0, peft==0.14.0, llama-cpp-python==0.2.76, together==1.3.3 and cohere==5.14.0. Exact pins make results reproducible, which is the right call for a benchmarking toolkit. They also mean Rankify will not share an environment with a project that needs a different transformers or vLLM version. Expect a dedicated environment, not an addition to an existing application venv.

The retriever extra is the other sharp edge. The package data section of pyproject.toml declares .cpp, .cu and .h files under rankify.utils.retrievers.colbert.colbert.indexing.codecs, which means the ColBERT path carries compiled code and needs a working build toolchain. That is a real constraint on locked-down machines and on Windows setups where CUDA toolchains are already fragile.

A third limitation is scope. Rankify is a benchmark and experimentation layer. The README does not document rollback, version migration, or how to move an index built with one release to the next. If your requirement is a stable search API with a deprecation policy, the absence of that documentation is the signal.

## How Rankify differs from LangChain and LlamaIndex

The obvious comparison is LangChain or LlamaIndex, and the difference is where each starts. Those frameworks start from application composition: you wire together a loader, a splitter, a vector store and a chain, and retrieval is one node in a larger graph. Rankify starts from evaluation. It ships the datasets (40 pre-retrieved benchmarks, plus the two Hugging Face mirrors), it ships the metrics (examples/RAGAS_metrices.py, examples/rag_evaluation.py, examples/rag_methods_evaluation.py), and it ships the rerankers as first-class citizens rather than as one optional node type.

That difference shows up in what each makes easy. In LangChain, swapping a reranker means changing a component in a chain and re-running your own evaluation harness. In Rankify, the reranker is the thing being measured, and the harness is part of the package. The trade-off runs the other way too: Rankify gives you far less help with document loading, chunking strategy, incremental index updates, or serving. If your problem is "we have PDFs and need answers", Rankify is downstream of the hard part. If your problem is "which reranker should we use", Rankify is the shorter path.

## Maintenance, releases and the licence question

The repository is not archived and the last push was on 2026-09-07. Releases are spaced rather than continuous: v0.1.2 in February 2025, v0.1.3 in March 2025, v0.1.4 in October 2025. The news section lists contributions between those releases, including 15+ new dense retrievers in February 2026 (SFR, E5, GritLM, and reasoning-augmented models RaDeR, ReasonIR, ReasonEmbed and BGE-Reasoner), so the codebase moves between tagged versions. Plan upgrades around tags, not around a rolling main branch.

Upgrade cost is dominated by the pins. Moving to a new Rankify release will likely move transformers, vLLM and sentence_transformers together, which means re-validating any cached indexes and any local model weights. The README does not describe a migration procedure for indexes built by an earlier version.

The licence needs a direct look. The README badge links to the Apache 2.0 text, while pyproject.toml declares license = {file = "LICENSE"} and the repository has no SPDX identifier in the metadata shown. Those two signals should agree; confirm which one applies to your use before you depend on it. This is a factual discrepancy to resolve with the maintainers, not a legal opinion.

## Conclusion

Adopt Rankify if you are comparing retrievers or rerankers and want the same datasets, metrics and Pipeline API across every run. Do not adopt it if you need a supported production search service, a documented rollback story, or a licence you have already cleared: the package metadata points at a LICENSE file while the README badge says Apache-2.0, and the README does not document rollback. Verify first that your Python version is 3.10 to 3.12, that you want the extras you are installing (retriever, reranking, rag or all), and that the LICENSE file in the repository matches the badge before you build anything on top of it.

## FAQ

### What Python version does Rankify need?

The README's install path creates a conda environment with Python 3.10, and pyproject.toml declares requires-python >=3.10 with classifiers for 3.10, 3.11 and 3.12. The documented and tested path is 3.10.

### How do I install Rankify with only the parts I need?

The 2025-10-14 release notes describe optional extras named retriever, reranking, rag and all, so you can install the retriever extra without pulling the vLLM-based reranking stack. Core dependencies stay small; the heavy serving libraries sit in the extras.

### What is Rankify's licence?

The README badge links to the Apache 2.0 licence text, while pyproject.toml declares license = {file = "LICENSE"} without an SPDX identifier. The two signals should be reconciled with the maintainers before you rely on either.

### Is Rankify legit?

It is a published research toolkit: the README links an arXiv paper (2502.02464), a Read the Docs site, Hugging Face dataset mirrors and tagged releases up to v0.1.4. That is the evidence available here; the README does not describe any commercial support or hosted service.

## Sources

- [DataScienceUIBK/Rankify on GitHub](https://github.com/DataScienceUIBK/Rankify)
- [Issues](https://github.com/DataScienceUIBK/Rankify/issues)
- [Project website](https://rankify.readthedocs.io/)
- [README](https://github.com/DataScienceUIBK/Rankify/blob/main/README.md)
- [Releases](https://github.com/DataScienceUIBK/Rankify/releases)

---

Hysen Labs editorial analysis, written from the project's own repository and release notes. Cite the canonical page: https://hysenlabs.com/projects/datascienceuibk-rankify