PISA is an in-memory inverted index built for indexing experiments

A search engine framed as a laboratory

PISA is described as a text search engine able to run on large-scale collections of documents, and the stated purpose is to allow researchers to experiment with state-of-the-art techniques in an ideal environment for rapid development. The name spells out the priority, Performant Indexes and Search for Academia. The features are written as a list of what the project contains rather than what a user can do: it is written in C++ for performance, it has parsing, indexing and sharding capabilities, many index compression methods implemented, many query processing algorithms implemented, and implementation of document reordering. The intended audience is stated separately. The primary use case is to conduct experiments to further the understanding of the field of information retrieval, and the project is focused mostly on the scalability and efficiency side of that field rather than on result quality. The documentation does add that PISA is also perfectly suitable as a simple general purpose indexing and search system, for people who just want to play with one.

The inverted index lives in main memory

The data structure is an inverted index, which the documentation explains plainly as storing, for each unique term, a list of the documents that contain it. The lists can be traversed rapidly to find documents matching the query terms, and those documents can then be scored, ranked and returned. At query time PISA stores its index in main memory for rapid retrieval, and that single sentence defines both the strength and the limit of the system. The documentation is explicit that PISA is an in-memory system aiming to serve queries based on an index residing entirely in main memory, and that as such the possible scale will be limited by main memory. Everything else about the architecture follows from that choice: the compression codecs exist to make the index fit, the sharding exists to break large corpora into subsets, and the query algorithms are designed around data that is already resident.

Boolean matching, then ranking

The query surface starts at the most basic level with Boolean AND and OR queries, and the documentation walks through the example of finding Wikipedia documents matching oolong tea, either as a conjunction of oolong and tea or as a disjunction. Beyond that, documents are ranked, and the ranking logic is the classic one. The rankers assume that the more rare a term is, the more important the word is, and that the more often a word appears in a document, the more likely the document is about that word, and longer documents contain more words and so tend to score higher, which is why normalization is conducted to ensure all documents are treated equally. The documentation points readers at the TF/IDF article for the method. The project is also more closely related to Lucene than to grep, though it does not currently support as many query types as Lucene does.

The scale figures come from the ClueWeb corpora

The scale discussion uses concrete corpora rather than vague claims. PISA can easily index the ClueWeb09B or ClueWeb12B corpora, which each contain over 50 million web documents and close to 500 GiB of compressed textual data, resulting in indexes in the range of 10 to 40 GiB depending on the compression codec used, and these indexes can be built from scratch in around 10 to 20 hours. Larger collections are handled by index sharding, which breaks large corpora into smaller subsets. The performance figures are attributed rather than asserted: a study on index compression and DAAT query processing methods by Mallia, Siedlaczek and Suel at ECIR 2019 showed PISA returning the top 10 and top 1000 documents with an average latency in the range of 10 to 40 and 20 to 50 milliseconds respectively on a 50 million document collection. PISA also appears in Tantivy's search engine benchmark game alongside Tantivy, Lucene and Rucene.

The lineage runs through ds2i and partitioned Elias-Fano

The project history is short and specific. PISA is a fork of the ds2i project started by Giuseppe Ottaviano, and the ds2i project contained the source code for a number of important efficiency innovations in information retrieval, including the Partitioned Elias-Fano compression method. That inheritance explains the shape of the codebase, since partitioned Elias-Fano is a technique for compressing the sorted integer sequences that underlie the posting lists of an inverted index, and it is the kind of component you implement yourself rather than take from a library. The repository also carries a `bump.json`, a `CHANGELOG.md` and a `docs/` directory whose published form is the book at the project homepage, so the documentation is a manual rather than a set of wiki pages.

The build is CMake with a Docker path

The repository ships a Dockerfile that shows the toolchain and the flags. The builder stage is based on a gcc 12 image with cmake pinned to 3.18 and libtool to 2.4, the source is copied in, and the build is configured with a Release build type, tools enabled and testing disabled:

RUN cmake -DCMAKE_BUILD_TYPE=Release -DPISA_BUILD_TOOLS=ON -DPISA_ENABLE_TESTING=OFF ..
RUN cmake --build . --config Release -- -j 4

The final stage copies only the built binaries into a debian bookworm image, so the runtime container carries no compiler. The default command runs the test suite through ctest with verbose output and four jobs. The top level of the repository has a `CMakeLists.txt` and a `clang.cmake` alongside `.clang-format` and `.clang-tidy`, so the project supports both GCC and Clang builds with a shared style. There are separate directories for `benchmarks/`, `microbench/`, `test/`, `tools/`, `script/`, `include/`, `src/` and an `external/` directory with git submodules, and the `pisactl/` directory is presumably the command line entry point.

Editorial conclusion

Adopt PISA if you are doing information retrieval research and need to compare index compression or query processing algorithms on real corpora, since the whole design is a laboratory rather than a product. Do not adopt it as a production search service, because it is explicitly an in-memory system whose scale is bounded by main memory and whose release cadence has slowed. Verify first which compression codec you need and how much memory the index will take, that the corpus you have matches the sharding and parsing assumptions, and that the tools build in your environment before you plan a multi-day indexing run.

Frequently asked questions

Official sources

1. License: Apache-2.0
2. pisa-engine/pisa on GitHub
3. Project website
4. README
5. Releases