# Inseq: post-hoc attribution for sequence generation models

> Inseq is a PyTorch toolkit that wraps Hugging Face style generation models and runs feature attribution over the generated output. It is the right shape of tool for researchers who need token-level explanations of what a model attended to, and the wrong shape for anyone who wants a production monitoring hook.

**inseq-team/inseq** — Interpretability for sequence generation models 🐛 🔍

- Repository: https://github.com/inseq-team/inseq
- Website: https://inseq.org
- Stars: 476 · Forks: 42
- Language: Python
- License: Apache-2.0
- Published: 2026-09-10 · Updated: 2026-09-10 · Language: en
- Canonical page: https://hysenlabs.com/projects/inseq-team-inseq

## The gap Inseq fills between generation and explanation

A generation model produces a string. That string is the whole observable output. If you want to know which input tokens or which earlier output tokens pushed the model toward a particular continuation, you have to instrument the forward pass yourself, and for encoder-decoder architectures you have to do it twice, once over the source and once over the target prefix. Inseq exists to make that instrumentation reusable rather than bespoke. The project describes itself as a hackable toolkit to democratize access to common post-hoc interpretability analyses of sequence generation models, and the emphasis on post-hoc matters: it explains generations you already have, rather than changing how they are produced. The intended audience is interpretability researchers and engineers who work with Hugging Face style models and want attribution results they can compare across methods without rewriting the plumbing each time. The topic list on the repository points at the same audience: attribution-methods, captum, explainable-ai, language-generation, transformers.

## How attribution flows through the toolkit

The pipeline has three visible stages. First, a model and tokenizer are wrapped so Inseq can hook the forward pass and capture the activations that attribution methods need. Second, an attribution method is applied to a specific generated sequence, producing per-step scores that link output tokens to input tokens or to earlier generated tokens. Third, those raw scores are passed through aggregators, which is the part v0.7.0 expanded. Aggregators matter because raw attribution matrices are unwieldy: a single generated sequence over a long prompt yields a score per output step per input position, and aggregators collapse that into something you can rank or plot. The dependency on Captum is the clearest signal of the design: Inseq is not reimplementing gradient-based attribution from scratch, it is adapting an existing attribution library to the sequence generation setting, where the target of the explanation is a generated token rather than a class logit. The v0.6.0 release notes mention a context attribution CLI, which tells you the project has been pushing attribution from a notebook-only workflow toward a command-line entry point.

## Installation and the extras that decide your workflow

The install is a single pip command: pip install inseq. The README states support for Python 3.10 through 3.13. If you want visualizations inside Jupyter or attribution over Hugging Face datasets, the documented form is pip install inseq[notebook,datasets]. For a development install the repository uses uv rather than plain pip: make uv-download fetches the uv package manager, make install syncs the package and dependencies, and make install-dev pulls in quality, docs and extras. The README also states that after installation make fast-test and make lint should run without errors. One installation caveat is called out explicitly in the FAQ: the tokenizers package needs a Rust compiler, installed from rustup.rs, with $HOME/.cargo/env added to your environment. That is a real friction point on locked-down CI images, and it comes from a transitive dependency rather than from Inseq itself.

## Where Inseq stops being the right tool

Attribution is expensive. Every explanation requires additional forward and backward passes over the model, so the cost scales with the number of generated sequences you want to explain, not with the number of requests you serve. Inseq is therefore a poor fit for inline serving instrumentation, where you would be paying attribution cost on every user request. It is also a poor fit when your model is not a sequence generation model in the sense the toolkit assumes: the whole abstraction is built around generating tokens step by step, and a classifier or a retrieval model has no generated sequence to attribute over. A third boundary is architectural coverage. The README does not enumerate supported model families, so before adopting you need to confirm your specific architecture is handled, particularly if you are working with a custom decoding loop or a non-standard attention layout. Finally, attribution scores are not causal claims. A high score on an input token tells you the method assigned it weight under its own assumptions, and different methods can disagree on the same generation. Inseq gives you the machinery to run those comparisons; it does not adjudicate between them.

## How Inseq differs from Captum and from attention-weight inspection

Captum is the closest reference point and Inseq depends on it rather than competing with it. The difference is the unit of explanation. Captum explains a model output given an input tensor, which works naturally for classification, where the output is a fixed set of logits. Sequence generation breaks that assumption because the output is a variable-length sequence and each generated token depends on the tokens generated before it. Inseq's contribution is the adapter layer that makes Captum-style methods meaningful in that setting, plus the aggregators that turn per-step scores into something readable. The other common alternative is reading attention weights directly. That requires no extra library, but attention weights are a single, model-internal signal, whereas Inseq exposes a family of attribution methods over the same generation so you can check whether two methods agree. If you only ever want attention heatmaps, Inseq is more machinery than you need.

## Release cadence and the cost of staying current

The version history shows an uneven rhythm. v0.5.0 landed in December 2023, v0.6.0 in April 2024, and v0.7.0 in February 2026, roughly a two-year gap before the most recent release. That is not abandonment, since the repository is not archived and the last push is recent, but it does mean the project moves in bursts rather than continuously. The practical consequence is that API surface can shift between minor versions: v0.6.0 added attribution methods and performance work, v0.7.0 added interactive visualizations and new aggregators. If you pin Inseq in a research pipeline, pin the version too, and read the release notes before upgrading, because aggregator behaviour changing underneath you will silently change your plots. On licensing, Inseq is Apache-2.0, which permits commercial use and modification provided you preserve the licence and notices and state significant changes. That is a permissive arrangement, but the usual caveat applies: if you redistribute a modified Inseq inside a product, the notice obligations travel with it. This is a description of the licence terms, not legal advice, and your counsel should review any redistribution plan.

## Who should install it and what to check first

Install Inseq if you are doing interpretability work on generation models and you want a library that already handles the awkward parts: hooking the forward pass, attributing over generated steps rather than logits, and aggregating the result. The notebook and datasets extras exist precisely because the intended workflow is exploratory. Skip it if you need per-request explanations in a serving path, if your model is not a generation model, or if attention-weight inspection already answers your question. The verification step before you commit is concrete: run pip install inseq on your target Python version, confirm the tokenizers Rust build succeeds in your environment, then run one attribution on a small model of the same architecture family you plan to use and check that the aggregator output matches the shape you expect. The v0.7.0 focus on interactive visualizations suggests the project's centre of gravity is the notebook, and that is where you should evaluate it.

## Conclusion

Adopt Inseq if you are a researcher or an interpretability engineer working with encoder-decoder or decoder-only generation models and you need token-level attribution you can plot and aggregate, not a dashboard. Do not adopt it if you need a low-overhead runtime hook inside a serving path, since attribution is a post-hoc pass over generations rather than an inline one. Before committing, verify three things: that your model family is covered by the supported architectures, that the specific attribution method you want is exposed in the installed version, and that the notebook or datasets extras you need install cleanly on your Python version, which the README pins to 3.10 through 3.13.

## FAQ

### What is inseq and what does it do?

It is a PyTorch-based toolkit for post-hoc feature attribution on sequence generation models, covering encoder-decoder and decoder-only Transformers with gradient-based, internals-based and perturbation-based methods, plus aggregator classes, visualizations in notebooks, a browser and a CLI, and a CLI path for attributing whole datasets.

### Which attribution methods does inseq support?

Thirteen methods in three families. Seven gradient-based including integrated gradients, DeepLIFT, gradient SHAP and discretized and sequential variants, one internals-based method that reads attention weights, and four perturbation-based including occlusion, LIME, value zeroing and a recent model-agnostic method. All method identifiers can be listed at runtime.

### What does it take to install inseq?

pip install inseq on Python 3.10 through 3.13, or install from the repository. The tokenizer backends arrive as hard dependencies of the Transformers extra, so a Rust compiler is needed for the fast tokenizer and a native toolchain including cmake and a configuration utility is needed for sentencepiece.

### Does inseq need a GPU?

GPU support is available and NVIDIA CUDA is listed among the environment classifiers. The dependency list also includes CUDA 11 runtime libraries for Linux as direct dependencies, so a Linux installation pulls them regardless of whether the machine has an NVIDIA GPU.

### What are step functions in inseq and why do they matter?

Step functions extract a score from the model at each step of the attribution process, and the supported ones are logits, probability with an option for log-probability, and entropy. They can also be passed as the target for methods that work from model outputs, which is what lets different method families be pointed at the same quantity and compared.

### How do I develop against inseq locally?

Clone the repository and run the make target that downloads and installs the uv package manager, then the install target which syncs dependencies. A developer variant installs quality, documentation and extra dependencies and also installs and autoupdates the pre-commit hooks. The quick test and lint targets should then run without errors.

## Sources

- [inseq-team/inseq on GitHub](https://github.com/inseq-team/inseq)
- [License: Apache-2.0](https://github.com/inseq-team/inseq/blob/main/LICENSE)
- [Project website](https://inseq.org)
- [README](https://github.com/inseq-team/inseq/blob/main/README.md)
- [Releases](https://github.com/inseq-team/inseq/releases)

---

Hysen Labs editorial analysis, written from the project's own repository and release notes. Cite the canonical page: https://hysenlabs.com/projects/inseq-team-inseq