What is Reranking?

Why a second pass is needed

The first stage of semantic search is a bi-encoder. The query goes into one model and comes out as a vector; the document goes into the same model and comes out as a vector; relevance is a dot product or a cosine similarity. Because a document's vector does not depend on the query, it can be computed once, stored, and searched with an approximate nearest neighbour index. That is what makes searching millions of documents cheap.

The weakness is the same property. The query and the document were never in the same context while either was encoded, so the model could only compress each of them independently into a fixed-size vector. Whether two sentences actually agree is a question about their relationship, and a fixed-size vector has to answer it without seeing the other side. That is why a well-tuned bi-encoder leaves relevant documents sitting at positions five to fifty when the user needed position one.

A cross-encoder fixes this by taking the pair. The query and the document are placed in one input, the model attends across both, and a single relevance score comes out. It is markedly better at this, and it is correspondingly expensive: the computation depends on both inputs, so nothing can be precomputed and no index can be built. Running it across a million documents is not a search, it is a million forward passes.

So the two-stage design is not a preference. The cheap stage exists to make the candidate set small enough that the expensive stage is affordable, and the expensive stage exists because the cheap stage cannot rank. Reranking is that second stage, and the candidate list it receives is a hard ceiling on what it can ever return.

The shapes a reranker comes in

Three families appear in practice, and they differ in what they can see.

Cross-encoders read one pair at a time. This is the standard reranker: the query and a document in one input, one relevance score out, and the candidates scored independently. PrithivirajDamodaran/FlashRank is a Python library built around this shape, and our analysis of it describes it as adding cross-encoder and LLM reranking to an existing search or RAG pipeline without requiring Torch or Transformers, small enough to run on CPU, with the caveat that its default model is a trade-off rather than a free win. huggingface/sentence-transformers exposes the same family through a CrossEncoder class, alongside embedding, sparse and multi-vector model classes, and our analysis calls it the default entry point for Python teams that need semantic search without training a model first.

Late-interaction models sit between the two. The document is encoded into a vector per token rather than one vector, and relevance is computed by comparing every query token against every document token and taking the best match per query token. It is more accurate than a bi-encoder and still indexable, which is why NovaSearch-Team/RAG-Retrieval bundles fine-tuning code for embedding, late-interaction and reranker models together, along with a small pip-installable library for calling rerankers behind one interface.

Listwise rerankers score a whole list of candidates in one pass, so a document's score depends on which documents are beside it. FlashRank's description states that it supports listwise and pairwise reranking based on LLMs and cross-encoders. This is the most expensive and the most sensitive to the surrounding window, which is a pitfall rather than a feature.

The same families appear outside Python. Anush008/fastembed-rs is a Rust library that our analysis describes as wrapping ONNX Runtime and Hugging Face tokenizers into a synchronous API for text, sparse, image and reranker models, and it notes the library fits Rust services wanting local inference without a Python sidecar while fitting poorly when you need GPU batching or a model outside its fixed enum.

When it helps, and when the money is wasted

Reranking pays when retrieval found the answer and ranked it badly. It is wasted when retrieval did not find it at all, because a better order of the wrong list is still the wrong list.

It helps most with long, natural-language questions. A keyword filter is already exact, and a bi-encoder handles short topical queries well; the gap opens when the question is a paragraph and the document has to satisfy all of it, which is exactly the case a cross-encoder's joint attention was built for.

It helps when the corpus is heterogeneous. When documents differ in format, length and register, a single vector per document averages away the distinctions, and the reranker sees the actual text.

It helps when the first stage was tuned for recall. A first stage configured to return a large candidate set, with the reranker responsible for the ordering, is the intended division of labour.

It does not help when the candidate set is too small. If the first stage returns five documents and they are already in the right order, the reranker is latency for nothing.

It does not help when the query is a filter rather than a question, when the reranker's input is being truncated so badly that the deciding passage never reaches it, or when the latency budget has no room for a serial second pass. A cross-encoder over a few hundred candidates is a few hundred forward passes, and on a busy system that is a queue behind your first stage rather than a rounding error.

The failure modes

Five of them explain most disappointing reranking results.

The ceiling. Whatever the first stage did not put in the list is gone. Teams often respond by increasing the reranker quality, when the answer is to increase the candidate count or fix the retriever, and the two changes look the same from a results page.

Truncation. A cross-encoder has a context limit, and chunks are frequently longer than it. The part that decides relevance is often not the part that fits, and the reranker scores a summary rather than the document.

Neighbour dependence. A listwise reranker scores a document in the company it keeps, so the same document gets a different score in a different list. That makes results dependent on candidate-set composition, which is awkward to cache and harder to explain.

Training distribution. Rerankers are often trained on synthetic queries generated from documents, which teaches a model to recognise paraphrases of documents it has seen and to score documents that answer questions rather than documents that would be shown to a person. The gap shows up as good numbers on benchmarks and disappointing behaviour in an application.

Measurement. An improvement measured against a first stage that was tuned with the reranker in mind, or against a candidate set the reranker helped choose, is not an improvement. DataScienceUIBK/Rankify is the tool for avoiding this: our analysis describes it as bundling forty pre-retrieved benchmark datasets, several sparse and dense retrievers, more than twenty reranking models and a generator layer behind a single call, and it is aimed at researchers comparing stacks on the same data rather than teams shipping a search box.

How the projects divide

Five roles, and picking the wrong one is how teams end up maintaining their own reranker.

The library you probably want. huggingface/sentence-transformers is the default entry point, and a CrossEncoder object is usually all a Python team needs.

The service. michaelfeil/infinity is an MIT-licensed Python serving engine that our analysis describes as exposing text embeddings, rerankers, CLIP, CLAP and ColPALI models behind a FastAPI REST API. It is aimed at teams that want OpenAI-shaped endpoints without sending documents to a hosted provider, and our analysis names the trade-off directly: you own the GPU, the model download and the capacity planning.

The dependency-free option. FlashRank runs on CPU without Torch or Transformers, which makes it the pragmatic choice when adding a deep learning stack to a small service is the bigger problem than reranking accuracy.

Training and research. NovaSearch-Team/RAG-Retrieval provides the fine-tuning code for embedding, late-interaction and reranker models behind one interface, and our analysis notes it is MIT licensed with a last push on 2026-08-28. Rankify provides the comparison harness.

Applications that embed a reranker in a pipeline. CortexReach/memory-lancedb-pro is an OpenClaw plugin that our analysis describes as storing preferences and decisions in LanceDB and recalling them in later sessions, using hybrid vector plus BM25 retrieval with cross-encoder reranking and a management CLI, plus one real hardware constraint to check before installing. CodeBendKit/codeseek indexes a repository with Tree-sitter, LanceDB, Tantivy and a cross-encoder reranker and exposes the result as MCP tools, which our analysis describes as a small MIT-licensed project aimed at agent workflows whose first screen is a setup wizard. weiwill88/Local_Pdf_Chat_RAG is the readable version of the whole thing: an MIT-licensed Python reference that runs document parsing, chunking, embeddings, FAISS, BM25, reranking and generation as separate modules behind a Gradio UI and a FastAPI service, built for inspection rather than for a production knowledge base.

In practice

Reranking is a precision stage whose input is a candidate list, and almost every disappointing result is explained by the list rather than by the reranker. Measure recall at the depth your reranker receives before tuning the reranker, keep chunks inside the reranker's context so the deciding passage is not truncated, and remember that a listwise model's output depends on its neighbours. For implementation, huggingface/sentence-transformers is the shortest path, michaelfeil/infinity is the service path, and FlashRank is the path that avoids a deep learning dependency; for evaluation, Rankify is the tool that keeps the comparison honest, and for training your own reranker, NovaSearch-Team/RAG-Retrieval is where the code is.