Quantus: Scoring XAI Explanations Instead of Eyeballing Them

The evaluation gap Quantus was built to close

A side-by-side plot of Saliency, Integrated Gradients, GradientShap and FusionGrad on the same input is not evidence that one of them is better. The README makes this point directly, noting that visual comparison is often insufficient, yet remains common practice when ground truth for the explanation itself does not exist. That is the gap. You have four heatmaps, no labels, and a reviewer asking why you chose the one you chose. Quantus answers with numbers instead of pictures. The stated goal is to collect the evaluation metrics that already exist in the XAI literature and automate what the authors call XAI quantification. The audience is therefore narrow and identifiable: researchers who publish explanation comparisons, and engineers who have to defend an explanation choice to someone outside the team. It is not a tool for generating explanations. It scores them. If you have not yet produced an explanation, Quantus has nothing to work on.

Six metric categories and what each one actually measures

The README groups the metrics into six categories: faithfulness, robustness, localisation, complexity, randomisation (also called sensitivity), and axiomatic metrics. Faithfulness is described as quantifying the extent to which explanations follow the predictive behaviour of the model, on the premise that more important features should play a larger role in the outcome. That definition matters because it is the one category where the metric is checking the explanation against the model's own behaviour rather than against a human expectation. The other five categories test different properties, and the README documents each one in a collapsible section rather than summarising them in a sentence. Two named metrics, EfficientMPRT and SmoothMPRT, sit in the randomisation category and are attributed to Hedström et al. (2023). The repository also states that it offers more than 35 metrics in six categories, and elsewhere says 30+. Those two figures appear in the same README, so treat the exact count as approximate and check the metric list for your pinned version. The paper, published in JMLR volume 24, is the authoritative source for definitions.

How the evaluation pipeline is wired

Quantus does not define its own explanation methods. The README states extended built-in support for Captum, tf-explain and zennit, which means the normal flow is: produce attributions with one of those libraries, hand the model, the inputs and the attributions to a Quantus metric, and read back a score. The library supports image, time-series and tabular data, with NLP listed as next up, and supports PyTorch and TensorFlow models. That breadth is the design centre. A metric object takes the model and the explanation as inputs and returns a scalar, and because the metrics are grouped by category you can run several against the same explanation to get a profile rather than a single number. The README describes a sensitivity analysis use case where a single parameter, for example the pixel replacement strategy inside a faithfulness test, changes the ranking of the XAI methods. That is the most useful thing the toolkit does, and it is also the most uncomfortable: if a ranking flips when you change one setting, the ranking was never a fact about the methods. Quantus is valuable precisely because it makes that instability visible instead of hiding it behind one number.

Installation, the tutorial path, and the version warning

The README provides a pip install command as the installation route, and the badge line lists Python 3.8 through 3.11. The homepage is quantus.readthedocs.io, which is where the API-level detail lives. Two runnable entry points are advertised: a Google Colab notebook titled Tutorial_ImageNet_Example_All_Metrics.ipynb, and a Binder link that opens the tutorials directory. The Colab notebook is the faster path because it runs the full metric set against ImageNet without a local setup. The README also carries a warning in italics that Quantus is under active development and that you should note the release version carefully to ensure reproducibility. That is not boilerplate. The gap between v0.5.3 in December 2023 and v0.6.0 in July 2025 is roughly nineteen months, and the batch implementation of twelve faithfulness metrics arrived as a pull request rather than in a versioned release note. Pin the version in your environment file and record it alongside any number you publish.

Where Quantus is the wrong tool

The first limitation is definitional. Quantus inherits its metrics from the XAI literature, and the README explicitly asks that you cite the original authors of each metric you apply. That means the toolkit cannot give you a metric for a property nobody has published a definition for, and it cannot resolve the disagreements between papers about what faithfulness should mean. You are choosing a school of thought when you choose a metric. The second limitation is scope. Training data attribution is not covered; the README points to a separate project, quanda, for that. NLP is listed as next up, so text models are out of scope today. The third is the one that bites in practice: a Quantus score is a number produced by a procedure with parameters, and the README's own sensitivity-analysis example shows that changing one parameter can reorder the methods. A single faithfulness score reported without the replacement strategy, the number of steps and the dataset slice is not reproducible. The toolkit gives you the machinery to vary those settings. It does not stop you from reporting one configuration and calling it the answer.

Quantus against Captum: scoring versus producing

Captum is the closest reference point because Quantus builds on it. The two libraries do different jobs and the README treats Captum as an explanation source, not a rival. Captum implements attribution algorithms: you give it a model and an input and it returns attributions. Quantus takes attributions that already exist and returns scores about them. If you need to know what a model attended to, Captum is the answer and Quantus is irrelevant. If you have four sets of attributions and need to justify a choice, Captum has nothing to say and Quantus is the answer. The practical consequence is that adopting Quantus usually means adopting Captum alongside it, since the built-in integration is the path of least resistance. The same relationship holds for tf-explain on the TensorFlow side and zennit for layer-wise relevance propagation. Quantus sits downstream of all three, and its value depends entirely on the quality of what those libraries produce upstream.

Maintenance, licence, and what the repository does not tell you

Development is active: the last push recorded is September 2025, and v0.6.0 landed in July 2025. The contribution channel named in the README is a Discord server, and the repository invites contributions and extensions. The licence field reports NOASSERTION, which means the repository metadata does not resolve to a standard SPDX identifier. The README does not state licence terms in the material available here, so if you intend to use Quantus inside a commercial product, read the LICENSE file in the repository and get your own legal review rather than inferring anything from the badge list. On upgrade cost, the practical burden is not the library itself but the citation and version discipline it demands: the README asks you to cite the JMLR paper plus the original author of every metric you use, and to note the release version for reproducibility. That is a documentation task that lands on whoever writes the methods section. Budget for it, because a Quantus number without its version and its metric citations is not defensible.

Editorial conclusion

Adopt Quantus if you already produce explanations with Captum, tf-explain or zennit and need a defensible number for a paper, an audit or an internal comparison, and if you can pin the release version in your environment. Do not adopt it if you need a metric that has no published definition you can cite, or if you expect the library to tell you which explanation is correct rather than merely how it scores on a chosen axis. Before you trust any result, install a pinned version, run the tutorial notebook that ships with the repository, and confirm the metric you intend to report is still present under the same name in that release.

Official sources

1. Issues
2. Project website
3. README
4. Releases
5. understandable-machine-intelligence-lab/Quantus on GitHub