What is Transformer?

How a transformer block computes

The architecture is small and the same everywhere. A sequence of tokens is turned into vectors, given position information, and passed through a stack of identical blocks. Each block does two things: a self-attention sublayer, where every position gathers information from every other position, and a feed-forward sublayer applied to each position independently. Both are wrapped in a residual connection and a normalisation step, and the attention sublayer usually comes first.

The attention sublayer is a weighted average with learned weights, as described for any attention mechanism: queries, keys and values are projected out of the representation, the scores are scaled, normalised into a distribution over positions, and used to mix the value vectors. The feed-forward sublayer is two linear layers with an activation between them, applied the same way at every position, which is where most of the parameters live.

Everything else follows from that. Because there is no recurrence, the whole sequence can be processed in one parallel pass during training, which is the property that made large models practical. Because attention connects any two positions in one step, the path length between the first and last token does not grow with the sequence, which is what lets a model relate distant things without a stack of intermediate steps.

The three families differ in which sublayers each block has. Encoder-only blocks attend bidirectionally and are used to produce representations. Decoder-only blocks use causal masking so a position sees only the past, and they are the shape behind most language models. Encoder-decoder models keep a full encoder and a decoder whose blocks also cross-attend to the encoder output, which is the original sequence-to-sequence design.

What the block buys and what it charges

Two properties explain why the architecture displaced recurrent networks for sequence work, and both of them have a price.

Parallel training. A recurrent layer processes position t using the output of position t minus one, so training is sequential in the batch dimension. A transformer has no such dependency between positions, so a training step processes the entire sequence at once. On modern accelerators that is the difference between a practical and an impossible training run.

Constant path length. In a recurrent network, information from position 1 to position n travels through n steps, and gradients have n steps to survive. In a transformer, one attention operation connects them directly. This is why a stack of a few dozen blocks can model dependencies that a recurrent model of the same depth cannot.

The charge is quadratic. Every position is compared with every other, so both the work and the memory of the attention computation grow with the square of the sequence length. On short sequences this is invisible. On long ones it is the constraint that shapes the field.

The second charge appears only at inference. During generation the model has to keep the keys and values it has already computed so it does not recompute them, and that cache grows linearly with the context and has to be read on every step. A transformer that trains beautifully can be awkward to serve, and the gap between the two is where most inference engineering lives.

The third charge is appetite. Parameter counts in these models are large enough that training and serving them are infrastructure problems, and the reason distillation, quantization, mixture-of-experts routing and speculative decoding all exist as separate bodies of work.

Where the architecture strains

Four situations are where a transformer is the wrong shape, and the alternatives are informative.

Very long sequences. The quadratic term means doubling the context roughly quadruples the attention work, and the cache at inference time is the practical wall. Architectures that approximate or replace attention exist precisely for this case.

Small data and small compute. A transformer has a large parameter count and no useful behaviour before training. On a small dataset a smaller model, or a model built around the structure of your problem, will beat it, and this is a well-known regime rather than a scandal.

Devices that cannot host the cache. A phone or a browser cannot hold the key-value cache of a large model, which is why the ecosystem contains quantized and distilled variants, and why client-side runtimes ship different, smaller models rather than the same ones as the server.

Architectures that exist to avoid attention. BlinkDL/RWKV-LM is the clearest case. Our analysis of RWKV-LM describes it as the reference training repository for an RNN architecture that runs in linear time with constant memory and trains with full parallelism, and its own description says it can be trained directly like a GPT transformer while combining what it sees as the best of the recurrent and transformer designs. It is not a transformer, and that is the point: it exists to make the attention cost go away, at the price of giving up the exact parallel training of a real transformer at the sequence level.

Where the structure of your data is fixed, none of this applies. A table, a key lookup and a form do not need a model that can relate arbitrary positions.

Failure modes and the habits that follow from them

Five things go wrong often enough to be worth naming.

The quadratic term surprises people. A context window that fits in training will not fit at serving time, because the cache is a different object from the activations, and the failure is an out-of-memory error rather than a slow one.

Positional information is assumed, not automatic. The attention operation is permutation-equivariant, so a model with no positional signal knows which tokens it has and not what order they were in. Every implementation adds this somewhere, and it is worth knowing where.

Masks are hand-written. Causal masking is a line of indexing code in most implementations, and getting it subtly wrong produces a model that trains and then leaks.

Attention weights are read as explanations. Class activation methods such as those in jacobgil/pytorch-grad-cam are a deliberate way of asking what drove a prediction, and our analysis of pytorch-grad-cam describes it as implementing more than eighteen such methods for computer vision, working with CNNs and Vision Transformers and with tasks including object detection and semantic segmentation. The point of that library is that the answer to a question about attention is produced by a method you can choose and check, rather than read off the attention weights.

One name, many architectures. Transformer is a family, not a model. Two checkpoints described as transformer models can share nothing but the block, so a question about a transformer model is usually a question about a specific checkpoint, its context handling and its licences.

How the ecosystem is arranged

The projects here sit at different layers of the stack, and picking the wrong layer is the usual mistake.

The definition layer. huggingface/transformers is described as the model-definition framework for state-of-the-art models in text, vision, audio and multimodal work, for inference and training. Our analysis puts it precisely: it defines pretrained models once so training frameworks and inference engines can agree on them, and it also says the definition-first design gets in the way sometimes. If two tools disagree about a model, this is where the disagreement is settled.

Post-training. huggingface/trl wraps the Transformers trainer with SFT, DPO, GRPO and KTO, and adds a CLI so the recipes can be run without writing code. Our analysis of TRL says it fits teams whose data is already in the Hub format and who want the standard recipes, and is the wrong tool when you need a custom training loop or a model that is not a Transformers model.

Embeddings and retrieval. huggingface/sentence-transformers wraps embedding, cross-encoder, sparse and multi-vector models behind four classes, and our analysis calls it the default entry point for Python teams that need semantic search without training a model first. The distinction between a bi-encoder and a cross-encoder matters more than any of the four names.

Vision. huggingface/pytorch-image-models provides pretrained image encoders and backbones for PyTorch, with training, validation and inference scripts, covering ResNet, EfficientNet, Vision Transformer, Swin, ConvNeXt and others. lucidrains/vit-pytorch is the readable counterpart: our analysis says it collects dozens of Vision Transformer variants behind one import and ships no pretrained weights, which is the trade you make for a readable implementation.

Execution elsewhere. huggingface/transformers.js ports the pipeline API to JavaScript and runs ONNX models locally through WASM or WebGPU. Our analysis says it fits client-side inference and static sites, and is not a server-side training or batch inference stack.

Transformers as infrastructure. xai-org/x-algorithm is the open-sourced Rust codebase behind the ranking model for the X For You feed, and its description notes the transformer implementation is ported from the Grok-1 open source release, adapted for recommendation use. Our analysis says it exposes the ranking model, scoring weights, the visibility filter pipeline and a transparency tool, which is a rare thing to find: a deployed transformer with its scoring path visible. facebookresearch/vggt is the research end of the same idea, a model from Meta AI and Oxford's Visual Geometry Group that regresses cameras, depth, point maps and tracks in a single forward pass, described as a research release rather than a pipeline.

In practice

A Transformer is a stack of identical attention-plus-feed-forward blocks, and its two defining properties, parallel training and constant path length between positions, are also the source of its two real costs, the quadratic attention term and the inference cache. Use huggingface/transformers as the definition of record so every tool agrees on the model, add huggingface/trl for the standard post-training recipes, and pick a runtime to match where the model will run. If the context is long enough that the cache is the wall, look at what the ecosystem does about it before assuming the architecture must stay, and keep the caveat from BlinkDL/RWKV-LM in mind as a design that removes the problem instead of mitigating it.