Neural network basics
Start from a straight line, write gradient descent and backpropagation by hand, train a first small network, then switch to PyTorch to recognise handwritten digits, and see overfitting and the ways to fight it with your own eyes.
Lessons
- 01Linear regression with numpy
Start from the simplest problem there is, predicting house prices from floor area. Use a straight line as the model and mean squared error to measure it, then try ninety thousand lines one by one, the dumbest way possible, to see clearly what "training" is actually looking for.
35 minutes · Beginner - 02Gradient descent
Stop trying blindly and instead compute, at every step, which way to change the parameters and by how much. Understand derivatives starting from slope, compute gradients by hand, write the gradient descent loop, and see for yourself what happens when the learning rate is too small, too large, or the data isn't standardised.
45 minutes · Beginner - 03Backpropagation by hand
Write a class of a few dozen lines in which every number remembers how it was computed, and you can start from the loss and pass gradients all the way back to every parameter with the chain rule. Check it against numerical differentiation, then use it to train a small network to learn XOR.
60 minutes · Intermediate - 04Getting started with PyTorch
Redo what the first three lessons wrote by hand, this time with PyTorch: tensors, automatic differentiation, nn.Module, optimisers. Every step is compared with the hand-written version, and the gradients and trained parameters come out exactly the same.
45 minutes · Beginner - 05Recognising handwritten digits
The first real classification task: have a network recognise 8×8-pixel handwritten digits. The difference between classification and regression, what softmax and cross-entropy do, why training uses mini-batches, and finally a confusion matrix to see where it goes wrong.
45 minutes · Intermediate - 06Overfitting and how to fight it
A network that gets the training set entirely right but makes no progress on data it hasn't seen is overfitting. We cause overfitting on purpose, then try weight decay, dropout, early stopping and more data, and see from real results which helps most.
40 minutes · Intermediate
Part 2, "Understanding LLMs from scratch", starts here.
In Part 1, an LLM was an interface to you: send text, get text back. This part opens it up to understand how it's trained. This module lays the foundation: however large the model, the core steps of training it are the same: compute the loss, compute the gradients, update the parameters. This module walks through those steps by hand on the smallest possible examples.
No GPU is needed; all the code runs in seconds on the CPU of an ordinary laptop. The maths goes no further than high-school derivatives, and every formula is checked with code and numbers.
Why this order
numpy by hand first, then PyTorch. Lessons 1 to 3 use no deep learning framework: a straight line to understand "what training is looking for", gradient descent to understand "how it looks", and backpropagation to understand "where the gradients come from". With those in hand, PyTorch in Lesson 4 isn't a black box; you know what every function is doing underneath.
The last two lessons deal with real data. Lesson 5 is a classification task that introduces softmax and cross-entropy, which the next module uses unchanged in GPT. Lesson 6 covers overfitting, a problem you'll meet training any model.
You're done when
- You can say what "training" does: find a set of parameters that makes the loss as small as possible.
- You can write a gradient descent loop by hand, say what happens when the learning rate is too small or too large, and why inputs should be standardised.
- You can explain backpropagation with the chain rule, and check a gradient computation you wrote against numerical differentiation.
- You can write a complete training loop in PyTorch: model, loss function, optimiser, zeroing gradients, backpropagation, updating parameters.
- You can say why classification tasks use softmax and cross-entropy, and roughly what the loss is when a 10-class network starts training.
- You can recognise overfitting in a training log, and say what each of several countermeasures does and where it falls short.
Code for this module
Code and program output are shown exactly as they ran, so comments and printed output are in Chinese.