MAPIE for sklearn Prediction Intervals and Risk Control

Why MAPIE Exists in a scikit-learn Workflow

MAPIE addresses a specific gap between ordinary model prediction and uncertainty-aware prediction. A fitted model may return a class label, regression value, or segmentation result without telling the user how much confidence to place in that output. MAPIE adds a separate conformalization step that uses a calibration dataset to estimate prediction intervals or prediction sets. The project is aimed at Python users working with predictive models, including teams using scikit-learn, TensorFlow, or PyTorch through compatible wrappers.

The scope is broader than interval estimation. The README identifies regression, classification, and time series tasks for prediction intervals and prediction sets. It also lists risk control for multi-label classification and semantic segmentation, with probabilistic guarantees associated with metrics such as recall and precision. Recent project documentation additionally names LLM-as-Judge and image segmentation as risk-control use cases. These are distinct operational questions. An interval asks how uncertain a numerical prediction may be, while a prediction set asks which labels or outcomes should be retained. Risk control addresses whether a chosen metric remains within a specified level.

MAPIE is therefore most relevant when the model itself is already selected and the remaining question concerns uncertainty or controlled error. It is not a replacement for model training, feature engineering, or data validation. The conformalization dataset is a separate input to the process, so a project that has only training data and no suitable calibration data cannot use the documented workflow as-is.

Conformalization Data and Model Wrappers

Based on the repository documentation, MAPIE estimates uncertainty on a conformalization dataset. This separates the data used to fit the predictive model from data used to estimate prediction uncertainty. The project describes its methods as belonging to conformal prediction and distribution-free inference, with theoretical guarantees under minimal assumptions on the data and model. The README also states that peer-reviewed algorithms are implemented.

MAPIE is model agnostic in the practical sense that it can work with scikit-learn, TensorFlow, and PyTorch models. For models that do not already satisfy the expected interface, the documentation points to a scikit-learn-compatible wrapper. That design is useful because it avoids requiring a prediction model to come from one machine-learning framework. It does not mean that every model can be passed to every MAPIE method without adaptation. The estimator must have compatible prediction behavior, and the task type must match the method being selected.

The conformalization dataset is also where the assumptions become visible. A model can produce a point prediction, but MAPIE still needs data from which to estimate the uncertainty or risk quantity. The README mentions exchangeability tests as a newer feature intended to help users verify when MAPIE can legitimately be applied. That feature is relevant because conformal guarantees depend on the relationship between calibration and future observations. The repository documentation should be consulted for the exact test, estimator, and data-splitting procedure before relying on a particular guarantee.

MAPIE API Surface Across Prediction Tasks

The repository layout shows examples grouped under classification, regression, calibration, conditional conformal prediction, exchangeability testing, data, and risk control. This indicates that MAPIE is organized around task-specific workflows rather than a single universal prediction wrapper. The top-level package directory is `mapie/`, while the public examples are under `examples/`. The README links to separate documentation pages for uncertainty quantification in regression and classification.

For regression, the documented output is a prediction interval, so the underlying estimator must produce the numerical predictions needed by the selected conformal method. For classification, the documented output is a prediction set rather than necessarily a single label. For multi-label classification and semantic segmentation, the project describes risk control for metrics such as recall and precision. The available examples also include conditional conformal prediction, which is a distinct approach from ordinary global calibration and should not be treated as an interchangeable default.

The project has undergone an API transition. The README says that MAPIE v1 is live and points readers to release notes for major API changes. This matters for adoption because examples written for an earlier MAPIE release may not map directly to the current interface. The repository contains dedicated calibration and exchangeability-testing examples, but the README excerpt does not provide a complete class name, estimator signature, or end-to-end code sample for each task. Those details should be taken from the linked documentation rather than inferred from the package name.

Installing MAPIE and Starting With the Documentation

The README lists three installation methods:

$ pip install mapie
$ conda install -c conda-forge mapie
$ pip install git+https://github.com/scikit-learn-contrib/MAPIE

The first command installs the package through `pip`. The second uses the `conda-forge` channel, and the third installs directly from the GitHub repository. After installation, the README directs users to the documentation rather than presenting a self-contained quickstart in the excerpt. The first documented use is uncertainty quantification for a regression task, with a separate page for classification.

There is a version constraint discrepancy that should be resolved before installation. The README requirements section states Python `>=3.9`, NumPy `>=1.23`, and scikit-learn `>=1.4`. The `pyproject.toml` instead specifies `requires-python = ">=3.10"`, `numpy>=1.24.1`, and `scikit-learn>=1.4`. The project metadata also lists Python 3.10 through 3.13 as supported classifiers. For reproducible setup, follow `pyproject.toml`, because it is the package build configuration, then consult the selected release documentation if an older environment must be supported.

A first real workflow therefore begins by installing the package, opening the regression quickstart, and selecting the conformal estimator documented for the intended task. The README does not include enough API detail to write a verified prediction-interval example here, so adding constructor names or method names would risk producing code that does not belong to the current release.

Guarantees, Calibration, and Exchangeability Constraints

MAPIE's value depends on the conditions attached to conformal prediction and risk control. The project describes probabilistic guarantees for supported methods, but the README does not claim that these guarantees apply automatically to every model or dataset. The conformalization dataset must be handled according to the selected method's assumptions, and newer exchangeability tests are specifically intended to help determine whether MAPIE can be applied legitimately.

This creates an important practical constraint. The model may be model agnostic, but the statistical procedure is not free of assumptions about the relationship between calibration data and future data. A user who cannot reserve or otherwise supply suitable conformalization data may receive an implementation that runs without producing the intended guarantee. MAPIE should not be treated as a generic confidence score that can be appended to any production prediction endpoint.

Risk control is similarly conditional on the selected task and metric. The README names recall and precision for multi-label classification and semantic segmentation, but it does not provide a universal metric configuration in the repository excerpt. A team adopting MAPIE for a safety threshold should identify the exact risk-control method, calibration procedure, metric definition, and acceptable error level before connecting its output to an operational decision.

Maintenance, Licensing, and Upgrade Work

The repository is not archived. The last recorded push was on 2026-10-16, and recent releases include v1.5.0 dated 2026-08-05, v1.4.1 dated 2026-06-08, and v1.4.0 dated 2026-04-30. These dates indicate recent repository activity, but they do not by themselves guarantee compatibility with a particular application. The v1 API change mentioned in the README is the more concrete upgrade concern because it can require changes to estimator imports, method calls, and surrounding pipeline code.

The package is distributed under the BSD-3-Clause license. `pyproject.toml` also identifies `LICENSE` and `THIRD_PARTY_NOTICES.md` as license-related files. This is useful information for dependency review, but it is not legal advice. Users should inspect those files and account for their own distribution, third-party, and compliance requirements.

The project classifies its intended audience as Science/Research and Developers, with support declarations for Windows, POSIX, Unix, and MacOS. Development checks are visible in the `Makefile`: `ruff` handles linting and formatting checks, `mypy` handles type checking, and `pytest` runs the test suite. The repository also requires coverage of 100 percent in its `coverage` target. These files describe the project's own quality gates, but they do not replace application-specific validation of calibration behavior.

When MAPIE Is the Wrong Choice

MAPIE is the wrong choice when the application needs a point estimate but has no uncertainty or risk-control requirement, or when the available data cannot support a conformalization step. It is also a poor fit for a workflow that treats generated intervals as ordinary confidence intervals without checking coverage assumptions. The project's methods are designed around calibration data, selected estimators, and task-specific metrics.

A model-agnostic interface does not remove modeling requirements. Regression needs appropriate numerical predictions, classification needs compatible score or label outputs, and segmentation or multi-label risk control needs a precisely defined metric. If the application cannot state which prediction interval, prediction set, or risk constraint it needs, selecting a MAPIE method would be premature.

For teams that need a simpler uncertainty estimate but do not need conformal guarantees, a conventional statistical or model-specific approach may be more appropriate. If the requirement is to train a new model, MAPIE is not the primary tool. Its role begins after a predictive model exists and a separate uncertainty or risk-control procedure is needed.

Editorial conclusion

MAPIE fits teams already using Python predictive models that need calibrated prediction intervals, prediction sets, or explicit risk-control procedures. It is less suitable when exchangeable calibration data, an appropriate estimator, or a clear metric requirement is unavailable. Verify the installed version, Python and NumPy compatibility, estimator choice, calibration split, and target risk before adopting it for a production decision.

Frequently asked questions

Official sources

1. License: BSD-3-Clause
2. Project website
3. README
4. Releases
5. scikit-learn-contrib/MAPIE on GitHub