PyTerrier makes IR experiments a pipeline you can compare

A pipeline is an expression

PyTerrier is a framework for performing information retrieval experiments, built on Terrier, and its overview is a single sentence: build sparse, learned sparse or dense indexing and retrieval pipelines for search and RAG use-cases, and conduct experiments on standard datasets.

The four verbs in that line are the four verbs of the framework: retrieve, rerank, answer, experiment, which is how the README header puts it.

The core idea is that a retrieval pipeline is an expression that you build up and can then evaluate. The example given is a re-ranking pipeline combining a Terrier BM25 retriever and the MonoT5 neural reranker, both of which are PyTerrier Transformer classes:

import pyterrier as pt
import pyterrier_t5
bm25 = pt.terrier.TerrierIndex.from_hf("pyterrier/vaswani.terrier").bm25() % 100
monot5 = bm25 >> pt.get.get_text(pt.get_dataset('irds:vaswani')) >> pyterrier_t5.MonoT5ReRanker()
monot5.search("What are chemical reactions?")

Four lines that carry a lot. The first retrieves a BM25 index from a hosted copy of the Terrier index for the Vaswani collection and takes the top 100 documents. The second retrieves the text of those documents. The third reranks them with MonoT5. And the fourth runs a search.

The operator that does the composing is the double angle bracket, and it is the piece of design that makes the framework what it is. A pipeline is built by applying one component to another, and the result is itself a component, so pipelines nest. The reranker is composed of a retriever, a document text fetcher and a reranker, and you cannot tell from that final expression whether any of it is a neural model.

That composability is what makes the experiment function possible, because a pipeline is a single value regardless of how many stages it has.

In notebook environments, PyTerrier transformers and pipelines can be visualised, and there is a schematic in the documentation showing the BM25 and MonoT5 pipeline. Being able to see the stages of a pipeline rather than infer them from the code is a real convenience when you are debugging one.

The project is Python with a primary language of Jupyter Notebook, which says something about how it is meant to be used. It is MPL-2.0 licensed, with the default branch master and the last push on 2026-10-20. The releases are 1.1.2 from 2026-07-19, 1.1.1 from 2026-07-18 and 1.1.0 from 2026-06-25, and the README header says v1.1.

The experiment call

Once you have pipelines, the framework's job is to compare them, and that is one function.

from pyterrier.measures import *
pt.Experiment(
  [bm25, monot5]
  pt.get_dataset('vaswani').get_topics(),
  pt.get_dataset('vaswani').get_qrels(),
  [nDCG@10, AP@100]
)

The Experiment function takes a list of systems to compare, the topics to run them on, the relevance judgements, and the measures to compute. What comes back is a table.

The structure of that call is worth reading as a specification of what an IR experiment needs. You need the systems, which are pipelines. You need the topics, which come from the dataset. You need the qrels, which say which documents are relevant to which topic. And you need the measures.

The measures here are nDCG@10 and AP@100, and the choice of them is the interesting part. nDCG at ten asks how good the top ten results are, weighted by relevance and normalised so runs are comparable. Average precision at a hundred asks about the whole ranking, and it is much more sensitive to where the relevant documents sit.

The metrics module is imported with a wildcard, which is the convention in the examples: from pyterrier.measures import * brings the measure names into scope so they can be written as plain identifiers.

The datasets come from a separate package. You can easily perform retrieval experiments using many standard datasets, including all from the ir_datasets package, and the example given is pt.datasets.get_dataset for the irds:medline/2004/trec-genomics-2004 identifier, which gives the TREC Genomics 2004 dataset. A full catalogue of ir_datasets is available on the ir-datasets site.

So the dataset layer is not PyTerrier's own corpus management. PyTerrier plugs into ir_datasets, which already indexes hundreds of standard collections with their topics and judgements, and that is why a dataset identifier looks like a path plus a name.

That division is worth noting for anyone extending the framework. Adding a new dataset means contributing to ir_datasets rather than to PyTerrier, and the two projects are maintained by overlapping but separate groups.

Installation and the Java dependency

The installation section starts with a suggestion that is unusual in a research framework and worth taking. The easiest way to get started with PyTerrier is to use one of the Colab notebooks, and the README points at the Open In Colab badges.

That is a deliberate choice. The framework's examples involve neural rerankers and hosted indexes, and a Colab notebook gives you a working environment with the right dependencies without you installing anything. For a framework whose main use is exploratory experimentation, the path of least resistance to the first result is a notebook you can open in a browser.

For a local installation, on Linux, Google Colab, Windows or macOS, there are two steps. You run pip install with the all extra, and you may need to set the JAVA_HOME environment variable if Pyjnius cannot find your Java installation.

The Java dependency is the significant one, and it comes from the architecture. PyTerrier is a Python framework built on Terrier, and Terrier is a Java information retrieval engine. The bridge is Pyjnius, which lets Python call into the JVM directly rather than going through a subprocess or a service boundary.

The comment in the package metadata says it exactly: pyjnius, used by pt.java to execute Java code, such as Terrier. So calling the Java engine from Python is not an optional extra, it is how the sparse retrieval side works, and the python package requirements list numpy, pandas, more_itertools and tqdm as the core, with requests, ir_datasets, scipy, ir_measures, pytrec_eval_terrier, jinja2, statsmodels, dill, lz4 and beautifulsoup4 after that.

Looking at that list tells you what the framework does beyond the plumbing. ir_measures and pytrec_eval_terrier are two independent implementations of the evaluation measures, which is a deliberate redundancy so that a bug in one does not silently corrupt your results. scipy is for significance testing and statsmodels is for multiple testing correction, so the framework does not just report numbers but tells you whether the difference between two systems is likely to be real. jinja2 is for getting colour into the Pandas table that Experiment produces, and beautifulsoup4 is for parsing SGML-formatted TREC topics, which are not all well-formed XML.

The Java installation note is the one practical gotcha. JAVA_HOME unset is a common cause of a PyTerrier install that imports but fails when it first touches the engine.

The extensions are separate repositories

PyTerrier has additional plugins for everything from dense retrieval to RAG, and the list is worth reading because it shows how the framework has grown by keeping each capability in its own package.

Pyterrier_DR is single-representation dense retrieval. Pyterrier_RAG is retrieval augmented generation and LLM access. PyTerrier_ColBERT2 is multiple-representation dense retrieval and or neural reranking. PyTerrier_PISA is fast in-memory indexing and retrieval using the PISA engine. PyTerrier_T5 is neural reranking with monoT5 and duoT5. PyTerrier_GenRank is generative listwise reranking with RankVicuna and RankZephyr. PyTerrier_doc2query is neural augmented indexing. And PyTerrier_SPLADE is neural augmented indexing as well.

The taxonomy here is the standard one in neural retrieval, and the naming gives it away. Single-representation versus multiple-representation is the distinction between a model that produces one vector per document and one that produces many. The latter is what ColBERT does, and the difference in retrieval quality versus the cost of storage and search is the trade those two packages exist to let you make.

Learned sparse sits in its own category too, with doc2query and SPLADE. A learned sparse retriever produces a sparse vector whose weights are learned rather than counted, which keeps the speed of a sparse index with a better ranking. The two packages cover the two main ways of doing it.

And the rerankers are split by cost. A monoT5 reranker processes one query-document pair at a time. A duoT5 processes a passage and a document together, which is a different formulation rather than merely a bigger model. A generative listwise reranker, which is what GenRank provides, does something else entirely: it shows the model a list and asks it to reorder it, which is closer to how a person would judge a set of results.

Every one of these is a separate repository, and the examples of how to use them are published in the Search Solutions 2022 tutorial, with notebooks that run on Google Colab.

The PyTerrier_T5 package is the one used in the example at the top of the README, and it is worth knowing that it is a separate install rather than part of the base package.

The entry points, and what they say about the design

The package metadata contains a set of entry points that reveal more about the architecture than the prose does.

There are four entry points under pyterrier.java.init. pyterrier.java maps to a CoreJavaInit, pyterrier.java.24 maps to a Java24Init, pyterrier.java.colab maps to a ColabJavaInit, and pyterrier.terrier.java maps to a TerrierJavaInit.

That is a registry of ways to initialise the Java side, and the variants tell you what the framework has to cope with. There is a generic initialiser, a specific one for Java 24, a special one for Colab, and one for the Terrier engine specifically.

The Colab variant is the interesting one. Colab environments differ from a local machine in ways that break the usual ways of locating a JVM, and having a dedicated initialiser means the Colab path does not need the local path to be made flexible. The Java 24 variant is the same idea in time rather than in environment: the JVM interface has moved over the years and the framework pins a specific one.

Then there are the artifact entry points. pyterrier.artifact maps sparse_index.terrier to a TerrierIndex, which is how the Java engine registers the things it provides so that Python code can find them by name.

The same registry pattern appears for the metadata adapter, which is how the index knows how to store the document metadata.

It is a plugin architecture implemented with packaging metadata rather than with a hand-written registry, which means the pieces are discoverable by the standard tooling and that a third-party package can register itself with the framework by declaring the right entry point.

The package metadata is otherwise conventional: setuptools with a dynamic version read from the pyterrier module's own version attribute, the dependencies read from requirements.txt, the tests directory excluded from the package find, and the README and licence shipped as package data.

Licensing, citation and the people behind it

PyTerrier is subject to the terms detailed in the Mozilla Public License Version 2.0, and the licence file is at the root of the repository.

But there is a second condition, and it is unusual enough to be worth reading in full. The source and binary forms are subject to a citation licence. By downloading and using PyTerrier you agree to cite the paper describing PyTerrier in any kind of work you produce where PyTerrier was used to conduct search or experimentation, whether it be a research paper, dissertation, article, poster, presentation, or documentation.

The list of output types is broad. A dissertation and a poster count, and so does documentation. In practice this is a request that is easy to honour and easy to forget, since the natural moment to cite is when you write the paper, and a poster or a set of documentation slides is exactly the kind of thing that gets forgotten.

The paper is Declarative Experimentation in Information Retrieval using PyTerrier, by Craig Macdonald and Nicola Tonellotto, in the Proceedings of ICTIR 2020, and the README gives the citation in BibTeX form:

@inproceedings{pyterrier2020ictir,
    author = {Craig Macdonald and Nicola Tonellotto},
    title = {Declarative Experimentation in Information Retrieval using PyTerrier},
    booktitle = {Proceedings of ICTIR 2020},
    year = {2020}
}

And the credits list is worth reading because it shows the shape of the project. Craig Macdonald, Sean MacAvaney, Alex Tsolov, Irene Anu, Chentao Xu and Sarawoot Kongyoung at the University of Glasgow, Nicola Tonellotto at the University of Pisa, Arthur Câmara and Marcus Schutte at TU Delft, Alberto Ueda at the Federal University of Minas Gerais, and Zhan Su at Copenhagen University.

That is a collaboration spanning five countries and the authors are also the maintainers of the underlying Terrier engine, which is the pattern that makes PyTerrier's design coherent: the Python framework and the Java engine come from the same group, so the boundary between them was designed rather than inherited.

The repository layout supports the research use case. There is a pyterrier/ package, a terrier-python-helper/ directory for the code that talks to the JVM, an examples/ directory with a notebooks.md index, an experiments/ directory, a docs/ directory, an extras/ directory, a tests/ directory and separate requirements files for the runtime and the test environment.

Editorial conclusion

Adopt PyTerrier if you are running information retrieval research and want pipelines you can assemble in a notebook and evaluate reproducibly, since the experiment harness handles the dataset loading, the measures and the statistics for you. Skip it if you need a production search service, since this is a research framework, and if you cannot install a Java runtime, since the Terrier engine runs on the JVM. Verify first which extension packages you need, since dense retrieval, neural reranking and RAG each live in a separate repository, and that you are willing to cite the paper, because the licence makes citation a condition of use.

Frequently asked questions

Official sources

1. License: MPL-2.0
2. Project website
3. README
4. Releases
5. terrier-org/pyterrier on GitHub