PECOS: a framework for problems with a hundred million candidate outputs

The problem is the size of the output space

PECOS is a versatile and modular machine learning framework for fast learning and inference on problems with large output spaces, such as extreme multi-label ranking and large-scale retrieval. Its design is intentionally agnostic to the specific nature of the inputs and outputs, and it is envisioned to be a general-purpose framework for multiple distinct applications.

The concrete shape of the task is given in one sentence. Given an input, PECOS identifies a small set of 10 to 100 relevant outputs from amongst an extremely large candidate set of roughly 100 million, and ranks these outputs in terms of relevance.

Those numbers are the whole justification. A hundred million labels is the regime where you cannot scan, cannot use a plain linear model over the full label space, and cannot afford to run a transformer over every candidate.

The library is on PyPI as libpecos, Apache-2.0 licensed, and the homepage is libpecos.org. The default branch is mainline rather than main, which is a small detail that matters if you script against the repository. The most recent release is v1.2.8 from 2025-01-31, preceded by v1.2.7 in October 2024 and v1.2.5 in June 2024, and the last push was on 2026-09-01.

Three model families behind one import path

The features section lists three components, and the way they are presented tells you the design is a shared skeleton with swappable models.

X-Linear lives at pecos.xmc.xlinear and is a set of recursive linear models that learn to traverse an input from the root of a hierarchical label tree to a few leaf node clusters, then return the top-k relevant labels within those clusters as predictions. The listed strengths are fast real-time inference in C++ and the ability to handle a 100 million output space. The technical details are in the PECOS paper.

XR-Transformer lives at pecos.xmc.xtransformer and is a transformer based XMC framework that fine-tunes pre-trained transformers recursively on multi-resolution objectives. It can be used to generate top-k relevant labels for a given instance, or simply as a fine-tuning engine for task aware embeddings. Its listed strengths are easy extension with the many pre-trained transformer models from Hugging Face transformers, and establishing the state of the art on public XMC benchmarks.

ANN Search with HNSW lives at pecos.ann.hnsw and implements the Hierarchical Navigable Small World Graphs algorithm. Its listed strengths are support for both sparse and dense input features, SIMD optimisation for both dense and sparse distance computation, thread-safe graph construction in parallel on multi-core shared memory machines, and thread-safe searchers that do inference in parallel to reduce inference overhead.

The HNSW module is the odd one out, and that is the point. It is not a multi-label model at all, it is the retrieval half of the framework, which means you can use PECOS for nearest neighbour search without touching the classification code.

The toy example is the whole API in five lines

The quick tour defines the XMR problem as two matrices. The instance-to-feature matrix X, of shape N by D, and the instance-to-label matrix Y, of shape N by L. Both are in SciPy CSR format, which is the sparse representation you would expect for a matrix with a hundred million columns and a handful of non-zeros per row. Some toy data matrices are available in the tst-data folder in the repository.

Training is three calls. PECOS constructs a hierarchical label tree and learns linear models recursively, and the example is XR-Linear:

>>> from pecos.xmc.xlinear.model import XLinearModel
>>> from pecos.xmc import Indexer, LabelEmbeddingFactory

# Build hierarchical label tree and train a XR-Linear model
>>> label_feat = LabelEmbeddingFactory.create(Y, X)
>>> cluster_chain = Indexer.gen(label_feat)
>>> model = XLinearModel.train(X, Y, C=cluster_chain)
>>> model.save("./save-models")

The names make the structure legible. LabelEmbeddingFactory.create turns the label matrix into label features, Indexer.gen builds the cluster chain that is the hierarchical label tree, and XLinearModel.train fits the recursive linear models with that chain as the C argument.

Prediction and evaluation follow the same shape, using the metrics helper from pecos.utils.smat_util:

>>> from pecos.utils import smat_util
>>> Yt_pred = model.predict(Xt)
# print precision and recall at k=10
>>> print(smat_util.Metrics.generate(Yt, Yt_pred))

And for inference in production there is the optimized C++ implementation, loading the saved model in predict-only mode:

>>> model = XLinearModel.load("./save-models", is_predict_only=True)
>>> for i in range(X_tst.shape[0]):
>>>   y_tst_pred = model.predict(X_tst[i], threads=1)

The save and load pair is what makes the library usable in a service, since the expensive part is index construction and training, not prediction.

Installation is a wheel, and the supported platforms are narrow

The requirements are Python 3.9, 3.10, 3.11 or 3.12, and pip 19.3 or newer. The README points to setup.py for the other dependencies, and recommends installing PECOS in a virtual environment, with a link to the Python documentation for anyone unfamiliar with them.

The supported platforms are listed explicitly: Ubuntu 20.04 and 22.04, and Amazon Linux 2. That is a short list, and it is the first thing to check if you are on a different distribution or on a container base image that is not one of those.

From a wheel, it is one command:

python3 -m pip install libpecos

From source, you need the builder tools first, and the two package lists differ by distribution. On Ubuntu 20.04 and 22.04:

``` bash sudo apt-get update && sudo apt-get install -y build-essential git python3 python3-distutils python3-venv

On Amazon Linux 2:

bash sudo yum -y install python3 python3-devel python3-distutils python3-venv && sudo yum -y groupinstall 'Development Tools'

Then the checkout and an editable install:

bash git clone https://github.com/amzn/pecos cd pecos python3 -m pip install --editable ./ ```

The editable install is the developer path, and it matters because the C++ pieces are compiled from source rather than shipped prebuilt in the wheel, so a source checkout exercises the build that a wheel install skips.

The C++ core and the strict style pipeline

The repository layout shows a Python package directory alongside the usual packaging files, with aws_infra/ for infrastructure, examples/, tutorials/, test/, a bibtex/ directory for the citations, and a THIRD-PARTY-LICENSES.txt alongside the Apache LICENSE and NOTICE files.

The Makefile is where the C++ side becomes visible. The top of it is an Amazon copyright header, and then two build knobs are documented in comments. The first is a verbose flag, with the usage line for it. The second is a C/C++ compiler warning as error flag, for code cleaning purposes, used by the libpecos, test and other targets, and it sets a PECOS_MANUAL_COMPILE_ARGS value of -Werror.

That flag is a small sign of how the project treats its native code. Turning warnings into errors is a practice that costs a developer something every day and pays off when you are maintaining a library that other people compile, which is exactly the situation of a library with a C++ inference core.

The style and type checks are three separate targets, and the pattern is worth noting because the same three tools are applied twice, once to the source and once to the tests. flake8 checks the source files with a configuration under .github/style_type_check_cfg, black checks formatting with its own configuration there, and mypy checks types with a configuration file in the same directory. Each is run over ./pecos and then over ./test.

The setup.py is where the version comes from. A VersionHelper class figures the current package version from the git tag, running git describe with the tags option and asserting the result matches a version pattern, and it writes the value into a generated pecos/_version.py file. When the source is not in a git repository, or the tag information is missing, it falls back to a dummy version of 0.0.0, and the generated file is excluded from the distribution.

The examples directory is where the framework gets applied

The examples directory is the part of the repository that tells you what people actually use PECOS for, and it is organised by paper rather than by task.

There is an example for each of the major results the project cites. pecos-xrlinear-jmlr22 and xrlinear-mscm-www22 cover the X-Linear work, xr-transformer-neurips21 covers the transformer model, and giant-xrt is the large-scale XRT variant. There is an ann-hnsw-pq4bits example for the HNSW module with product quantisation at 4 bits, and examples for the FM for XMC, PINA which uses side information via predicted instance neighbourhood aggregation, and the qp2q and spmm approaches.

Several examples are named after the datasets or the setting rather than the method. msmarco-rankllama and overlap-xmc handles overlapping labels, pefa-wsdm24 is from a 2024 workshop, and MACLR is another classification setup. but the pattern is the same throughout: each example is a self-contained reproduction of a specific result.

The tutorials/ directory sits alongside, and the citation section in the README lists the papers behind each of them, from the PECOS paper in JMLR 2022 through work on uncertainty quantification in extreme classification, graph based approximate nearest neighbour search, end-to-end learning to index and search in large output space, and the relevance under the iceberg paper on reasonable prediction for extreme multi-label classification.

That is the honest shape of this project. It is a research library with a paper, and the examples are the evidence for the claims the README makes about state of the art results.

Editorial conclusion

Adopt PECOS if you face a label or candidate space in the hundreds of millions and need top-k retrieval with a model rather than a brute force scan, and you are working on a platform the wheel supports, which the README lists as Ubuntu 20.04 and 22.04 and Amazon Linux 2. Do not adopt it for a small label set, because the hierarchical machinery and the index construction cost more than a plain linear model. Before you commit, read the paper and check the release cadence, since the most recent tag is v1.2.8 from 2025-01-31 and the last push was on 2026-09-01 on the mainline branch.

Frequently asked questions

Official sources

1. amzn/pecos on GitHub
2. License: Apache-2.0
3. Project website
4. README
5. Releases