pytorch-ood: a detector library built around one scoring convention

The problem pytorch-ood addresses

Out-of-distribution detection papers usually ship as a single script attached to one model and one dataset. Reproducing a comparison between two methods means reimplementing both, matching preprocessing, and hoping the score directions agree. pytorch-ood packages the methods as importable detectors so that the scoring function is separate from the network that produces logits or features. The README lists what the library provides: detection methods, loss functions, datasets, network architectures with pre-trained weights, augmentations and utilities. It also states that the library covers neighbouring fields, naming Open-Set Recognition, Novelty Detection, Confidence Estimation and Anomaly Detection. The intended user is someone training with PyTorch who wants to measure how well a model separates inliers from outliers without building the measurement apparatus. The README notes compatibility with pytorch-lightning and pytorch-segmentation-models, so the library assumes an existing training stack rather than replacing one. It is not a serving component and it is not a labelling tool. Its job is scoring and evaluation.

One convention that governs every detector

The README states an important convention adopted in pytorch-ood: OOD detectors predict outlier scores that should be larger for outliers than for inliers. It then warns that if a detector's scores do not match the formulas in the corresponding publication, the score calculation may have been adjusted to comply. That single sentence is the most consequential thing in the repository for anyone porting code. A method published with an inlier score, or with a sign flipped for numerical convenience, will appear inverted here. The warning is honest, but it also means you cannot copy a formula from a paper, paste it into your own evaluation loop, and expect the numbers to line up with what pytorch-ood reports. The convention is a good one for evaluation, since thresholding and ranking become uniform across methods, and OODMetrics can consume any detector's output without knowing which method produced it. The cost is a translation step every time you cross between the library and the literature. If you are comparing your own implementation against pytorch-ood's, check the direction first.

How a detector is wired to a model

The quick start example shows the data flow. You load a network with load_model, for instance the identifier "wrn-40-2/cifar10/energy/s1", and load the matching preprocessing with load_transform using the same identifier. The model is moved to CUDA. The detector is then constructed around the model, in the example EnergyBased(model). During evaluation you iterate your own data loader, apply the transform, call the detector on the batch, and pass the resulting scores plus labels into OODMetrics. The README states that OOD data must be marked with labels below zero. That is the entire contract between your data pipeline and the library: a label sign, not a separate OOD dataset class. Metrics are accumulated with update and read out with compute. The design keeps the loop visible in your code, which is useful when you need to log intermediate values, and it means the library never takes ownership of your training loop. The flip side is that batching, device placement and transform application are your responsibility in every script. The quick start is roughly a dozen lines, and most of them are the loop you would have written anyway.

Pre-trained weights and where they come from

load_model takes a string identifier rather than a checkpoint path. The example "wrn-40-2/cifar10/energy/s1" encodes architecture, dataset, training loss and seed. The README describes this particular model as a WideResNet-40 used in major publications, pre-trained on CIFAR-10 with Energy-Bounded Learning Loss, with weights taken from the original paper. That is a meaningful convenience: a WideResNet-40 trained on CIFAR-10 with a specific loss is not something you want to retrain just to try a detector. The identifier format also implies a registry of models behind the string, and the README does not enumerate it. If you need a model that is not in the registry, the material does not describe an obvious path to register your own, so you would fall back to constructing the detector around your own nn.Module. That is supported by the detector API, but it means you supply the logits and the transform yourself. The pre-trained weights are a shortcut for benchmarking, not a model zoo for production. Nothing in the README suggests the weights are intended for deployment.

The benchmark harness and its cache keys

The README labels the benchmark feature as beta. The example uses CIFAR10_OpenOOD with a root directory and a transform, then calls benchmark.evaluate for each detector with loader_kwargs, a device, and caching options. Two parameters stand out. cache=True with cache_dir and cache_key means logits and pooled features are written to disk and reused across calls, which the README says is for evaluating several detectors on the same benchmark. The cache_key in the example is a human-readable string, "cifar10-openood-wrn-cifar10-pt", so correctness depends on you choosing a key that reflects the model and transform you are caching. Change the transform and keep the key, and you will silently evaluate against stale features. That is a real failure mode, and the API does not appear to derive the key from the model or transform automatically. The example also sets num_workers to 12 and batch_size to 128, which are ordinary data loader settings. Results come back as rows that the example merges into a pandas DataFrame with a Detector column. pandas is listed as an optional dependency for the examples, so the benchmark path assumes it is installed.

Installation and the dependency split

Installation is a single command: pip install pytorch-ood. The README lists torch, torchvision, scipy and torchmetrics as dependencies, and scikit-learn, gdown, pandas and segmentation-models-pytorch as optional. The split matters because two detectors depend on it. The README states that scikit-learn is needed for ViM and k-NN. If you install the base package and reach for ViM, you will hit a missing import rather than a graceful degradation. gdown is listed for downloading some datasets and model weights, so a restricted network environment can break load_model or benchmark setup in ways that look like code errors. pandas is only needed for the example output formatting, so you can skip it if you consume the result rows directly. The package requires Python 3.8 or newer according to the README badge. The project is Apache-2.0 licensed, which permits commercial use and modification, but the README also points to model weights taken from original papers, and those weights carry their own provenance. Check the terms attached to any checkpoint you redistribute, since the library licence does not automatically cover third-party weights.

Where pytorch-ood is the wrong choice

The library is tied to PyTorch at the tensor level. The detectors wrap a model and call it, so a TensorFlow or JAX training stack cannot use them without a port. The benchmark harness is marked beta, which is the maintainers' own signal that the API may move. The scoring convention means published numbers will not match unless you account for the adjustment the README warns about, so anyone expecting drop-in reproduction of a paper's table should expect to reconcile directions. There is no described mechanism for serving detectors behind an API or for monitoring a live stream; the loop in the quick start is an offline evaluation loop. The cache_key behaviour is a correctness hazard rather than a bug, but it is the kind of hazard that produces plausible-looking wrong numbers. Finally, the README does not describe calibration guarantees. A detector score is a ranking signal, and the library will not tell you what threshold to deploy. If your problem is deciding when to abstain in production, you still own that decision.

A concrete alternative and the difference in approach

PyTorch Lightning's own ecosystem is the nearest thing to compare against, but the more instructive contrast is scikit-learn's outlier detection estimators, such as IsolationForest or LocalOutlierFactor. Those operate on feature matrices and know nothing about your network. pytorch-ood operates on the network itself: EnergyBased needs the logits, ViM needs pooled features, and the README's dependency note shows the library reaching into scikit-learn for the latter. The difference is where the representation comes from. With scikit-learn you extract features yourself and the detector is a standalone object you can pickle and call from any process. With pytorch-ood the detector holds a reference to a torch model and runs a forward pass per batch, which is why the quick start moves the model to CUDA and applies a transform. Choose scikit-learn when the representation is already fixed and you want a model-agnostic estimator. Choose pytorch-ood when the question is how a specific trained network behaves on inputs outside its training distribution, and you want the method implementations that were designed for logits and penultimate features rather than for tabular rows.

Maintenance cost and what to verify first

The repository is active, not archived, with releases at v0.3.1, v0.3.2 and v0.3.3 across 2026 and a default branch named dev, so upgrade churn is real but the cadence looks measured rather than constant. The benchmark feature carries a beta label, which is the part most likely to change under you. The Apache-2.0 licence is permissive; the pre-trained weights are a separate question and the README does not spell out their terms. If you adopt the library, pin the version, because a detector's score calculation can be adjusted to comply with the outlier-score convention and that adjustment changes your numbers. Before writing evaluation code, confirm that your loader marks OOD samples with labels below zero, since that is the only signal the library uses to separate the two groups. Then run one detector on a small subset and print the raw scores for inliers and outliers side by side, because that is the fastest way to catch a direction mismatch introduced by the convention. If you use the benchmark cache, derive the cache_key from the model identifier and transform rather than typing a free-form string, so a changed transform cannot silently reuse old features.

Editorial conclusion

Adopt pytorch-ood if you already train in PyTorch and want MSP, EnergyBased or ViM compared on the same data loader without writing the scoring code yourself; skip it if you need a maintained scikit-learn style estimator or a model-agnostic service. Before committing, verify three things: that OOD samples in your loader carry labels below zero, that the detector you picked returns larger scores for outliers under the library's own convention, and that the extra dependencies your method needs (scikit-learn for ViM and k-NN, gdown for some downloads) are acceptable in your environment.

Official sources

1. kkirchheim/pytorch-ood on GitHub
2. License: Apache-2.0
3. Project website
4. README
5. Releases