Inseq: post-hoc attribution for sequence generation models

The gap Inseq fills between generation and explanation

A generation model produces a string. That string is the whole observable output. If you want to know which input tokens or which earlier output tokens pushed the model toward a particular continuation, you have to instrument the forward pass yourself, and for encoder-decoder architectures you have to do it twice, once over the source and once over the target prefix. Inseq exists to make that instrumentation reusable rather than bespoke. The project describes itself as a hackable toolkit to democratize access to common post-hoc interpretability analyses of sequence generation models, and the emphasis on post-hoc matters: it explains generations you already have, rather than changing how they are produced. The intended audience is interpretability researchers and engineers who work with Hugging Face style models and want attribution results they can compare across methods without rewriting the plumbing each time. The topic list on the repository points at the same audience: attribution-methods, captum, explainable-ai, language-generation, transformers.

How attribution flows through the toolkit

The pipeline has three visible stages. First, a model and tokenizer are wrapped so Inseq can hook the forward pass and capture the activations that attribution methods need. Second, an attribution method is applied to a specific generated sequence, producing per-step scores that link output tokens to input tokens or to earlier generated tokens. Third, those raw scores are passed through aggregators, which is the part v0.7.0 expanded. Aggregators matter because raw attribution matrices are unwieldy: a single generated sequence over a long prompt yields a score per output step per input position, and aggregators collapse that into something you can rank or plot. The dependency on Captum is the clearest signal of the design: Inseq is not reimplementing gradient-based attribution from scratch, it is adapting an existing attribution library to the sequence generation setting, where the target of the explanation is a generated token rather than a class logit. The v0.6.0 release notes mention a context attribution CLI, which tells you the project has been pushing attribution from a notebook-only workflow toward a command-line entry point.

Installation and the extras that decide your workflow

The install is a single pip command: pip install inseq. The README states support for Python 3.10 through 3.13. If you want visualizations inside Jupyter or attribution over Hugging Face datasets, the documented form is pip install inseq[notebook,datasets]. For a development install the repository uses uv rather than plain pip: make uv-download fetches the uv package manager, make install syncs the package and dependencies, and make install-dev pulls in quality, docs and extras. The README also states that after installation make fast-test and make lint should run without errors. One installation caveat is called out explicitly in the FAQ: the tokenizers package needs a Rust compiler, installed from rustup.rs, with $HOME/.cargo/env added to your environment. That is a real friction point on locked-down CI images, and it comes from a transitive dependency rather than from Inseq itself.

Where Inseq stops being the right tool

Attribution is expensive. Every explanation requires additional forward and backward passes over the model, so the cost scales with the number of generated sequences you want to explain, not with the number of requests you serve. Inseq is therefore a poor fit for inline serving instrumentation, where you would be paying attribution cost on every user request. It is also a poor fit when your model is not a sequence generation model in the sense the toolkit assumes: the whole abstraction is built around generating tokens step by step, and a classifier or a retrieval model has no generated sequence to attribute over. A third boundary is architectural coverage. The README does not enumerate supported model families, so before adopting you need to confirm your specific architecture is handled, particularly if you are working with a custom decoding loop or a non-standard attention layout. Finally, attribution scores are not causal claims. A high score on an input token tells you the method assigned it weight under its own assumptions, and different methods can disagree on the same generation. Inseq gives you the machinery to run those comparisons; it does not adjudicate between them.

How Inseq differs from Captum and from attention-weight inspection

Captum is the closest reference point and Inseq depends on it rather than competing with it. The difference is the unit of explanation. Captum explains a model output given an input tensor, which works naturally for classification, where the output is a fixed set of logits. Sequence generation breaks that assumption because the output is a variable-length sequence and each generated token depends on the tokens generated before it. Inseq's contribution is the adapter layer that makes Captum-style methods meaningful in that setting, plus the aggregators that turn per-step scores into something readable. The other common alternative is reading attention weights directly. That requires no extra library, but attention weights are a single, model-internal signal, whereas Inseq exposes a family of attribution methods over the same generation so you can check whether two methods agree. If you only ever want attention heatmaps, Inseq is more machinery than you need.

Release cadence and the cost of staying current

The version history shows an uneven rhythm. v0.5.0 landed in December 2023, v0.6.0 in April 2024, and v0.7.0 in February 2026, roughly a two-year gap before the most recent release. That is not abandonment, since the repository is not archived and the last push is recent, but it does mean the project moves in bursts rather than continuously. The practical consequence is that API surface can shift between minor versions: v0.6.0 added attribution methods and performance work, v0.7.0 added interactive visualizations and new aggregators. If you pin Inseq in a research pipeline, pin the version too, and read the release notes before upgrading, because aggregator behaviour changing underneath you will silently change your plots. On licensing, Inseq is Apache-2.0, which permits commercial use and modification provided you preserve the licence and notices and state significant changes. That is a permissive arrangement, but the usual caveat applies: if you redistribute a modified Inseq inside a product, the notice obligations travel with it. This is a description of the licence terms, not legal advice, and your counsel should review any redistribution plan.

Who should install it and what to check first

Install Inseq if you are doing interpretability work on generation models and you want a library that already handles the awkward parts: hooking the forward pass, attributing over generated steps rather than logits, and aggregating the result. The notebook and datasets extras exist precisely because the intended workflow is exploratory. Skip it if you need per-request explanations in a serving path, if your model is not a generation model, or if attention-weight inspection already answers your question. The verification step before you commit is concrete: run pip install inseq on your target Python version, confirm the tokenizers Rust build succeeds in your environment, then run one attribution on a small model of the same architecture family you plan to use and check that the aggregator output matches the shape you expect. The v0.7.0 focus on interactive visualizations suggests the project's centre of gravity is the notebook, and that is where you should evaluate it.

Editorial conclusion

Adopt Inseq if you are a researcher or an interpretability engineer working with encoder-decoder or decoder-only generation models and you need token-level attribution you can plot and aggregate, not a dashboard. Do not adopt it if you need a low-overhead runtime hook inside a serving path, since attribution is a post-hoc pass over generations rather than an inline one. Before committing, verify three things: that your model family is covered by the supported architectures, that the specific attribution method you want is exposed in the installed version, and that the notebook or datasets extras you need install cleanly on your Python version, which the README pins to 3.10 through 3.13.

Official sources

1. inseq-team/inseq on GitHub
2. License: Apache-2.0
3. Project website
4. README
5. Releases