Implementing a Transformer from scratch
Write a tokenizer and attention by hand, build a complete small GPT, train it on over thirty thousand Tang poems until it can write poetry, then cover generation, the KV cache, and what's still missing between a small GPT and a real LLM.
Lessons
- 01Writing a BPE tokenizer by hand
Module 01 said what the model sees is tokens. This lesson writes a byte-level BPE tokenizer by hand and trains it on Tang poetry, to watch it piece together Chinese characters and common words from bytes step by step, and to see clearly where it falls short on small data.
45 minutes · Intermediate - 02The attention mechanism
To understand its own meaning, a character needs to look at the characters before it. Start from the simplest idea, "average all the characters before", then add queries, keys and values, the causal mask and scaling step by step to write a complete attention head, and check it against PyTorch's implementation.
50 minutes · Advanced - 03Multi-head attention and positional encoding
One attention head can look at earlier characters by only one standard; open several heads and it can look at several things at once. Attention also can't tell what order characters are in, so the position has to be supplied. This lesson makes both clear with experiments, and turns up one unexpected result.
40 minutes · Advanced - 04Building a complete GPT
Assemble attention, a feed-forward network, residual connections and LayerNorm into a Transformer block, stack a few, add embeddings and an output layer, and you have a complete GPT. Read gpt.py section by section and account for where all 1.6 million parameters are.
50 minutes · Advanced - 05Training it: teaching the model to write Tang poetry
Train last lesson's GPT on thirty-four thousand Tang poems for seven minutes on a laptop CPU, and watch it progress step by step from gibberish to five- and seven-character lines. Then give it only three hundred poems, and watch it memorise poems instead of learning to write them.
50 minutes · Advanced - 06Generating text and the KV cache
A trained model only scores the next character, so how do scores become a poem? This lesson uses our own GPT to try temperature, top-k, continuation and acrostic poems, then implements the KV cache and measures how much repeated computation it saves.
40 minutes · Advanced - 07From a small GPT to an LLM
Our small GPT writes poems but can't answer questions. Between it and an LLM you can chat with lie scale, pre-training, instruction tuning and preference alignment. This lesson uses real numbers from papers to explain each step.
30 minutes · Intermediate
Module 01 said an LLM does only one thing: predict the next token. In this module you build a model that does exactly that.
We'll write a GPT from scratch: tokenizer, attention, multiple heads, positional encoding, Transformer blocks, all by hand, with no ready-made model library. Then we train it on thirty-five thousand regulated-verse poems selected from the Complete Tang Poems. It has only 1.61 million parameters, trains in about 7 minutes on a laptop CPU, and can then write passable five- and seven-character-line poems.
The differences between it and ChatGPT or DeepSeek are mainly in scale and training method; the principles are the same. Once you've finished this module, you'll be able to follow the main structure of real LLM code.
Why this order
Tokenization first, because the model's input is tokens. Then attention, the core of the Transformer; Lesson 3 adds multiple heads and position information, and Lesson 4 assembles all the parts into a complete model. Lesson 5 trains it, and Lesson 6 uses it to generate text and explains the most important inference optimisation, the KV cache. The last lesson has no code; it covers the steps still between our small GPT and a real LLM you can chat with.
This module's code is all in code/09-transformer/, with gpt.py being the model shared by the later lessons. You need to have finished Module 08 first, especially backpropagation, PyTorch and cross-entropy.
You're done when
- You can write a byte-level BPE tokenizer by hand, and explain why it splits text very finely when data is scarce.
- You can write attention with a causal mask and scaling, and say what q, k and v each do.
- You can explain why multi-head attention and positional encoding are needed, and that the causal mask itself leaks order information.
- You can read
gpt.pysection by section and work out how many parameters each part has. - You can train a small poetry-writing GPT on your own computer, and judge from the loss curve and generated output whether it's learning or reciting.
- You can explain temperature, top-k and the KV cache, and measure the speed-up the KV cache gives.
- You can say what pre-training, instruction tuning and preference alignment each do.
Code for this module
Code and program output are shown exactly as they ran, so comments and printed output are in Chinese.