crepes: conformal prediction wrappers for scikit-learn classifiers and regressors

The gap crepes fills between a point prediction and a decision

A random forest returns a label. It does not tell you how often that label is right for this particular input, and it gives you no way to say "these two classes are both plausible" in a way that carries a stated error rate. crepes addresses that specific gap. The README describes the package as implementing conformal classifiers, regressors and predictive systems on top of any standard classifier and regressor, turning original predictions into well-calibrated p-values and cumulative distribution functions, or prediction sets and intervals with coverage guarantees. The audience is therefore narrow and specific: engineers and researchers who already have a working scikit-learn model and now need uncertainty quantification around it, rather than a better model. The topics list confirms the intended scope, naming prediction intervals, prediction sets, quantile regression and conformal test martingales. If you are looking for a library that improves accuracy, this is not it. crepes does not change what your model predicts; it changes what you can claim about those predictions.

Wrapping, calibrating, then predicting: the actual data flow

The mechanism is a wrapper plus a second fit on held-out data. The quickstart fetches the qsar-biodeg dataset from OpenML, splits it into train and test with train_test_split, then splits the training half again so that 25 percent becomes a calibration set. A RandomForestClassifier is passed into WrapClassifier, fitted on the proper training set, and then calibrated on the calibration set with rf.calibrate(X_cal, y_cal). From that point the wrapper exposes predict_p, which returns one p-value column per class, and predict_set, which returns prediction sets at a requested confidence level. The README shows predict_set returning label arrays such as ['1', '2'] and, with labels=False, binary indicator rows. The important structural point is the three-way split. The model never sees the calibration data during fitting, and the calibration data is what supplies the empirical distribution of non-conformity scores. crepes also exposes evaluate, which the README runs with all metrics as the default and which returns a dictionary containing error, avg_c, one_c, empty, ks_test and two timing fields. That dictionary is the honest part of the design: avg_c and one_c tell you how large and how often singleton your prediction sets are, which is where the practical cost of a coverage guarantee shows up.

Mondrian categories, class-conditional calibration and online updates

Marginal coverage is not always what you want. crepes supports Mondrian conformal classifiers and regressors, formed by passing a function or a MondrianCategorizer from crepes.extras as the mc argument to calibrate. The README demonstrates this by using the model's own predicted labels as categories, so that coverage is controlled separately within each predicted class. A special case is the class-conditional classifier, produced by setting class_cond=True in the calibrate call, where categories come from the true labels. The README's example output for the class-conditional variant shows a lower error and a higher avg_c than the standard classifier on the same test set, which is the expected trade-off: tighter control over per-group error costs you larger prediction sets. The package also addresses a subtler problem. The README states plainly that for an inductive conformal predictor the predicted p-values and the resulting errors on a test set are not independent, and that semi-online conformal predictors can make them independent by updating the calibration set immediately after each prediction, assuming the true label is then available. This is enabled by setting online=True on the prediction methods while supplying the true labels. That is a real constraint, not a feature toggle: the online mode only makes sense in a setting where ground truth arrives promptly after each prediction.

Getting it installed and the smallest working example

Installation is a single command, either pip install crepes from PyPI or conda install conda-forge::crepes from conda-forge. The README pins the current version at 0.9.1 in both badges. The minimal working sequence, taken directly from the quickstart, is to import WrapClassifier and RandomForestClassifier, construct the wrapper around the estimator, call fit on the proper training split, call calibrate on the calibration split, then call predict_p or predict_set. Two keyword arguments carry most of the configuration surface: confidence, which the README uses at 0.99 in its examples, and labels, which switches predict_set between label arrays and binary indicator arrays. For Mondrian behaviour, mc takes either a callable or a MondrianCategorizer, and class_cond=True is the shortcut for conditioning on true labels. For regressors the README's description points to standard, normalized and Mondrian conformal regressors and predictive systems, with crepes.extras supplying standard options for difficulty estimates, non-conformity scores and Mondrian categories. If you need custom scores, the README states you can supply your own functions rather than using the extras module. That is the extension point to plan around, because the choice of non-conformity score drives interval width more than any other decision.

Exchangeability is the assumption, and martingales are how you check it

Every coverage guarantee here rests on exchangeability between the calibration set and the test points. The README is explicit that the crepes.martingales module exists for testing that underlying assumption, offering conformal test martingales. This is the limitation worth taking seriously. If your data drift, if your test points are temporally ordered, or if calibration and deployment populations differ, the stated confidence level is a number you printed rather than a property you hold. The package gives you the diagnostic tool but does not remove the assumption. A second, quieter cost is the calibration split itself. Carving out 25 percent of the training data, as the quickstart does, means the underlying model is fitted on less data than it could be, and the wrapper will not recover that. On small datasets that trade can hurt more than the calibration helps. Third, the online mode requires true labels to be available immediately after prediction, which excludes most batch scoring pipelines. None of these are defects in the implementation; they are properties of inductive conformal prediction, and crepes is honest about them in the places where it matters.

Where crepes sits relative to MAPIE and the Venn-Abers family

The closest comparison in the Python ecosystem is MAPIE, which also targets conformal prediction on top of scikit-learn estimators. The difference in approach is visible in the API shape. crepes uses an explicit wrapper object that owns the calibration state: you construct WrapClassifier or WrapRegressor around your estimator, and the calibration set is passed once to calibrate, after which the wrapper exposes predict_p, predict_set and evaluate. MAPIE instead tends to be used through wrapper estimators that follow the scikit-learn fit and predict convention more closely, which makes it easier to drop into an existing pipeline or cross-validation loop, at the cost of a less direct route to raw p-values per class. The other family worth naming is Venn-Abers, which produces calibrated probability estimates rather than prediction sets. If what you need is a well-calibrated probability per class, Venn-Abers is the more direct answer; if you need sets or intervals with a stated error rate, crepes is aimed at that. The distinguishing element in crepes is the breadth of the conformal surface it exposes: Mondrian categorizers, normalized regressors, predictive systems producing full cumulative distribution functions, and the martingale module for testing exchangeability, all in one package rather than spread across separate tools.

Version cadence, licence and what maintenance actually involves

The release history shows 0.8.0 in March 2025, 0.9.0 in October 2025 and 0.9.1 in June 2026, with the last push to the repository in July 2026. That is a steady minor-version cadence rather than a rapid one, and the pre-1.0 numbering means the API is not declared stable. The practical consequence for an adopter is that pinning a version and reading the release notes before upgrading is the sensible default, since minor releases at this stage can carry interface changes. The licence is BSD-3-Clause, stated in the README badge and in the repository licence file. That is a permissive licence, which generally means you can use, modify and redistribute the code including in commercial settings, provided the copyright notice and licence text are retained and you do not use the author's name to endorse derived work. This is a description of what BSD-3-Clause typically requires, not legal advice; if the licence terms matter to your organisation, have counsel read the LICENSE file rather than relying on a summary. Maintenance cost on your side is low but not zero: the calibration split has to be rebuilt whenever the underlying model is retrained, and any custom non-conformity function or Mondrian categorizer you write becomes code you own.

Editorial conclusion

Adopt crepes if you already have a fitted scikit-learn estimator and need calibrated prediction sets or intervals, and you can spare a held-out calibration split. Do not adopt it if your observations are strongly dependent over time or if you need a library that trains the underlying model for you, since crepes wraps rather than builds. Before committing, verify three things on your own data: that a proper training and calibration split is feasible, that your chosen non-conformity score and any Mondrian categories are available at prediction time, and that the exchangeability assumption holds well enough that the martingale classes in crepes.martingales do not flag a violation.

Official sources

1. henrikbostrom/crepes on GitHub
2. Issues
3. License: BSD-3-Clause
4. README
5. Releases