perceiver-pytorch: cross-attention to a fixed latent set, with the PerceiverIO and PerceiverLM wrappers

The problem: attention cost that scales with input length

Standard transformer self-attention compares every input element with every other input element. For a 224 by 224 image flattened into tokens, or a long audio clip, that quadratic term is what limits how much input you can feed in. The Perceiver papers address this by never attending over the raw input directly. Instead the model keeps a fixed set of latents and repeatedly pulls information from the input into that set. The README describes num_latents as the number of latents, or induced set points, or centroids, noting that different papers give it different names.

The audience for this repository is narrow and specific. It is for researchers and engineers who have read the Perceiver or Perceiver IO papers and want a working PyTorch module to drop into an existing training script. It is not a framework. There is no trainer, no dataset loader, no checkpoint. The README shows model construction and a forward pass, and stops there.

How the latent bottleneck works in this implementation

The Perceiver class takes input_channels, input_axis, num_freq_bands and max_freq. With fourier_encode_data set to True, which the README says is the default, the model Fourier-encodes the positional information itself using the axis count you supply: 2 for images, 3 for video. If you are already encoding positions yourself, you turn that off.

The depth argument controls how many times the cross-attention plus self-attention cycle repeats. The README spells out the resulting shape explicitly: depth times (cross attention, then self_per_cross_attn self attention blocks). So depth is not a plain layer count, and self_per_cross_attn is a multiplier inside each cycle. The README also notes that weight_tie_layers is optional and is indicated in the diagram, which means the same weights can be reused across cycles instead of allocating a fresh set.

Head configuration is split in two. cross_heads and cross_dim_head govern the attention from latents to input, and the README notes the paper said 1 head for this. latent_heads and latent_dim_head govern self-attention among the latents, with 8 in the example. That asymmetry is the architecture, not a tuning detail: the expensive path is the one over the input, so it is kept narrow.

PerceiverIO and the output-side decoupling

The original Perceiver produces a fixed classification output. PerceiverIO is the follow-up that the README describes as allowing a flexible number of output sequence length, and it is imported as a separate class from the same package. Its constructor takes dim, queries_dim and logits_dim alongside the latent settings.

The mechanism is visible in the example. You pass a sequence of shape (1, 512, 32) and a separate queries tensor of shape (128, 32). The returned logits have shape (1, 128, 100), which the README annotates as batch, decoder seq, logits dim. The number of queries you supply determines the output length, and it is independent of the 512 input positions. That is the whole point of the IO variant, and this library exposes it as a plain forward argument rather than a separate configuration mode.

PerceiverIO also exposes seq_dropout_prob, described as structured dropout that drops a fraction of input sequence tokens. The README frames this as saving compute and having a regularizing effect. Dropping tokens before cross-attention is a direct way to cut the cost of the input-side path, and it is one of the few knobs here that trades accuracy for throughput in an obvious way.

Getting it running, and the arguments you must set

Installation is a single command, pip install perceiver-pytorch, per the README. Then you import Perceiver from perceiver_pytorch and construct it with the keyword arguments shown. The README's image example uses input_channels=3, input_axis=2, num_freq_bands=6, max_freq=10., depth=6, num_latents=256, latent_dim=512, cross_heads=1, latent_heads=8, cross_dim_head=64, latent_dim_head=64, num_classes=1000, attn_dropout=0., ff_dropout=0., weight_tie_layers=False, fourier_encode_data=True and self_per_cross_attn=2.

Feeding a tensor of shape (1, 224, 224, 3) returns (1, 1000). Note the layout: channels last, and the batch dimension is present even for a single image.

The language modelling wrapper is PerceiverLM, imported from the same package. It takes num_tokens, dim, depth, max_seq_len and the same latent arguments, and its forward call accepts a mask. The README example builds it with max_seq_len=2048, then passes a sequence of shape (1, 512) and a boolean mask of the same shape, receiving logits of shape (1, 512, 20000). The mask is a required-looking argument in that example, not an optional extra.

There is one import change for the experimental variant. Instead of the top-level package you use from perceiver_pytorch.experimental import Perceiver. The README states this variant adds bottom-up attention alongside the top-down path, following the Induced Set Attention Block scheme from the Set Transformers paper. Everything else about the call signature is unchanged, which makes it cheap to try, but the README labels it experimental and gives no accuracy comparison.

Where this library stops

The repository is a model definition, and the README treats it as one. There is no training script, no data pipeline, no evaluation harness, and no pretrained weights mentioned anywhere in the supplied material. If you need a model you can fine-tune on day one, this is not that.

The release history is the second constraint. The most recent release listed is 0.8.8 from August 2023, following 0.8.7 in January 2023 and 0.8.6 in December 2022. The repository itself shows a push in June 2026, so the code has moved since the last tagged release. That gap matters in practice: pip install perceiver-pytorch gives you whatever version is published, which may not match the current main branch. If you need a fix that landed after 0.8.8, you are installing from the repository rather than from PyPI, and you should check that before writing your training code around it.

There is also a scaling caveat that the README does not discuss. The latent array is fixed at construction time via num_latents, so the model cannot grow its capacity with input size. That is the design, and it is what keeps cost flat, but it means the architecture is a poor fit when the task genuinely requires fine-grained output at every input position.

How this differs from a standard transformer implementation

The natural comparison is a plain transformer encoder, or a library like Hugging Face transformers that supplies one alongside tokenizers, trainers and a model hub. The difference in approach is structural, not cosmetic. A transformer encoder attends over the full input sequence, so memory grows with the square of sequence length and every layer sees the same token set. Here, cross-attention reads the input into a small latent set once per depth cycle, and all the subsequent self-attention happens among num_latents vectors, which in the README example is 256.

That has a practical consequence. With a transformer you change max_seq_len and the cost curve changes with it. With PerceiverIO you can hold the encoder side roughly fixed and vary the number of decoder queries to change output length, which is what the (128, 32) queries example demonstrates. The trade is that you give up the per-token representation a transformer hands you for free. If your downstream task needs an embedding for every input position at full resolution, the latent bottleneck is working against you, and a transformer encoder or a convolutional backbone is the simpler answer.

The experimental Perceiver import sits between the two. It adds bottom-up attention in the Induced Set Attention Block style, which is closer to how Set Transformers and slot-attention models build representations. The README points to that lineage through its citation of the inverted-attention work. It is worth reading the paper before assuming it is a strict improvement.

Licence, maintenance and what to verify before adopting

The repository is MIT licensed. That is permissive: it allows commercial use, modification and redistribution provided the copyright notice and licence text are retained. This is a description of the licence identifier, not legal advice, and if you are shipping a product you should have someone check how MIT interacts with your own dependency and notice obligations.

Maintenance cost is low in the sense that there is nothing to operate. You install a package and import a class. The cost that does exist is version drift. With the last tagged release in August 2023 and repository activity later than that, you should pin the version you install and record which commit or release it corresponds to, because the published package and the current source are not the same thing.

Three things are worth checking before you build on this. First, confirm whether the PyPI version you get matches the constructor signatures in the README, since the README reflects the repository rather than necessarily the last release. Second, decide which class you need, because Perceiver, PerceiverIO and PerceiverLM have different arguments and only PerceiverIO takes a queries tensor. Third, if you intend to use the experimental import, note that the README offers no comparison against the standard one, so you will be measuring that yourself.

Editorial conclusion

Adopt it if you want the Perceiver architecture in PyTorch without reimplementing cross-attention against a latent array, and you are comfortable that the last release on PyPI is 0.8.8 from August 2023 while the repository has been pushed to more recently. Do not adopt it if you need a maintained training loop, a checkpoint zoo, or a supported alternative to a full vision or language stack. Before committing, verify the installed version against the repository state, and confirm which of Perceiver, PerceiverIO, PerceiverLM or perceiver_pytorch.experimental.Perceiver you actually need, because they are separate classes with separate constructor signatures.

Official sources

1. Issues
2. License: MIT
3. lucidrains/perceiver-pytorch on GitHub
4. README
5. Releases