Backpropagation by hand

Last lesson's gradients were derived by hand from one specific formula. But a neural network is a complex function of layer upon layer, with hundreds or thousands of parameters, and deriving each one's derivative by hand is impossible.

This lesson writes a small tool that has the computer calculate every parameter's gradient automatically. It's called backpropagation. It's PyTorch's most central feature, and this lesson writes a version of just a few dozen lines in pure Python; once it's written, you'll know what's happening behind the single line loss.backward().

The chain rule

Start with a simple example. Suppose y = 3x and z = y². If x changes a tiny bit, how much does z change?

Think of it in two steps: when x changes a tiny bit, y changes 3 times as much (the derivative of y with respect to x is 3); when y changes a tiny bit, z changes 2y times as much (the derivative of z with respect to y is 2y). So when x changes a tiny bit, z changes 3 × 2y times as much.

z 对 x 的导数 = (z 对 y 的导数) × (y 对 x 的导数)

This is the chain rule: the derivative of a chain of operations equals the product of each step's derivative. Each step's derivative depends only on that step itself, and is called the local derivative.

A neural network is a long chain of operations: multiply, add, pass through a nonlinear function, layer after layer, until the loss is computed. As long as we know each step's local derivative, we can use the chain rule to multiply our way back step by step from the loss, getting the derivative of the loss with respect to every parameter.

Make every number keep accounts

The approach: every time an operation is performed, record "the local derivative of the result with respect to each input". Write a class Num that, besides its value, stores a list: [(input number, local derivative), ...].

class Num:
    """一个会记账的数：记下自己的值、梯度，以及"我对每个上游数的局部导数"。"""

    def __init__(self, value, parents=()):
        self.value = value
        self.grad = 0.0
        self.parents = parents  # [(上游的 Num, 局部导数), ...]

    def __add__(self, other):
        other = other if isinstance(other, Num) else Num(other)
        # a + b 对 a 的导数是 1，对 b 的导数也是 1
        return Num(self.value + other.value, [(self, 1.0), (other, 1.0)])

    def __mul__(self, other):
        other = other if isinstance(other, Num) else Num(other)
        # a × b 对 a 的导数是 b，对 b 的导数是 a
        return Num(self.value * other.value, [(self, other.value), (other, self.value)])

    def __pow__(self, n):
        # x 的 n 次方，导数是 n × x 的 (n-1) 次方
        return Num(self.value ** n, [(self, n * self.value ** (n - 1))])

    def tanh(self):
        t = math.tanh(self.value)
        # tanh 的导数是 1 - tanh²
        return Num(t, [(self, 1 - t * t)])

Each operation only needs to know its own local derivatives:

• Addition a + b: when a changes a little, the result changes by the same amount, so the local derivatives with respect to a and to b are both 1.
• Multiplication a × b: when a changes a little, the result changes b times as much, so the local derivative with respect to a is b, and with respect to b is a.
• Power xⁿ: the local derivative is n × xⁿ⁻¹, from high-school maths.
• tanh: a function that squashes any number into the range -1 to 1, common in neural networks; its derivative is 1 - tanh².

With Python's operator overloading (__add__, __mul__ and so on), ordinary expressions like a * b + c automatically produce Num objects and quietly record the whole computation.

(The full code also has subtraction and a few lines letting plain numbers take part in operations; see code/08-neural-nets/backprop.py.)

Backpropagation

Once the computation is done, every Num knows which numbers it was computed from, and the whole computation forms a graph. Backpropagation starts from the final result (the loss) and walks back along this graph:

    def backward(self):
        """从这个数（通常是损失）出发，把梯度传给所有上游的数。"""
        order, seen = [], set()

        def visit(node):  # 先访问完所有上游，再把自己放进列表：得到一个"从上游到下游"的顺序
            if id(node) not in seen:
                seen.add(id(node))
                for parent, _ in node.parents:
                    visit(parent)
                order.append(node)

        visit(self)
        self.grad = 1.0  # 损失对自己的导数是 1
        for node in reversed(order):  # 从下游往上游，链式法则：上游梯度 += 下游梯度 × 局部导数
            for parent, local in node.parents:
                parent.grad += node.grad * local

Two steps:

1. Put things in order. visit ensures a number always comes after all the numbers it was computed from. Traversing in reverse walks back from the loss, and by the time it reaches a given number, everything that depends on it has already been processed, so its gradient is fully accumulated.
2. Pass gradients back. The derivative of the loss with respect to itself is 1. At each step backwards, apply the chain rule: the upstream gradient gets "the downstream gradient × the local derivative" added to it.

Why "add" (+=) instead of assigning directly? Because a number may be used more than once. In y = x * x, for example, x is both inputs of the multiplication, and the gradients coming back along the two paths have to be added together.

Checking: compare with numerical differentiation

Test it once written. Use a small expression f = (a × b + c)², with a=2, b=-3, c=10:

a, b, c = Num(2.0), Num(-3.0), Num(10.0)
f = (a * b + c) ** 2
f.backward()
print(f"  f = {f.value}")
print(f"  自动算出的梯度：df/da={a.grad}, df/db={b.grad}, df/dc={c.grad}")

Then check it with last lesson's numerical method (this time the more accurate "nudge a little on each side"):

def numeric(fn, x, h=1e-6):
    return (fn(x + h) - fn(x - h)) / (2 * h)

== 1. f = (a × b + c)²，a=2, b=-3, c=10
  f = 16.0
  自动算出的梯度：df/da=-24.0, df/db=16.0, df/dc=8.0
  数值求导核对： df/da≈-24.0000, df/db≈16.0000, df/dc≈8.0000

A perfect match. You can verify by hand too: a × b + c = 4, f = 4² = 16; the derivative of f with respect to (a×b+c) is 2 × 4 = 8, so df/dc = 8, df/da = 8 × b = -24, df/db = 8 × a = 16.

Building a neural network with it

With automatic differentiation, we can build neural networks.

What a neuron does is simple: multiply each input by a weight, add them up, add a bias, and finally pass the result through tanh:

class Neuron:
    def __init__(self, n_inputs):
        self.w = [Num(random.uniform(-1, 1)) for _ in range(n_inputs)]
        self.b = Num(0.0)

    def __call__(self, xs):
        total = self.b
        for w, x in zip(self.w, xs):
            total = total + w * x
        return total.tanh()

Remove the final tanh and it's the straight line from Lesson 1, only with several inputs instead of one. A nonlinear function like tanh (also called an activation function) is essential: without it, many layers of straight lines stacked together still make a straight line, unable to learn any complex pattern.

A row of neurons forms a layer, and two layers stacked make a small network: 2 inputs → 4 hidden neurons → 1 output. Count the parameters: the hidden layer has 4 neurons, each with 2 weights plus 1 bias, 12 in all; the output layer has 1 neuron with 4 weights plus 1 bias, 5 in all. 17 parameters altogether.

Learning XOR

XOR: output -1 when the two inputs are the same, 1 when they differ (since tanh outputs between -1 and 1, we use -1 and 1 in place of the usual 0 and 1).

It's a classic example because a straight line can't do it: plot the four points on a plane and you can't separate (0,1), (1,0) from (0,0), (1,1) with one straight line. It needs a hidden layer and a nonlinear function.

The training loop is exactly like last lesson's gradient descent, except the gradient step becomes loss.backward():

data = [([0, 0], -1), ([0, 1], 1), ([1, 0], 1), ([1, 1], -1)]
lr = 0.1
for epoch in range(1, 301):
    loss = Num(0.0)
    for xs, y in data:
        pred = output(hidden(xs))[0]
        loss = loss + (pred - y) ** 2
    for p in params:
        p.grad = 0.0  # 每一轮都要清零，否则梯度会一直累加
    loss.backward()
    for p in params:
        p.value -= lr * p.grad

Note that the gradients must be reset to zero at the start of every round. Because backward uses +=, without resetting, this round's gradients get added on top of last round's. This is a very classic mistake, and the same applies in PyTorch (next lesson's optimizer.zero_grad()).

== 2. 网络：2 个输入 → 4 个隐藏神经元 → 1 个输出，共 17 个参数
  第   1 轮  损失 4.1022
  第  10 轮  损失 3.7902
  第  50 轮  损失 0.1093
  第 100 轮  损失 0.0346
  第 200 轮  损失 0.0134
  第 300 轮  损失 0.0081
  训练后的预测：
    输入 [0, 0] → -0.965（目标 -1）
    输入 [0, 1] → +0.952（目标 +1）
    输入 [1, 0] → +0.953（目标 +1）
    输入 [1, 1] → -0.951（目标 -1）

At first the loss is 4.1 and almost all four predictions are wrong. Progress is slow for the first 10 rounds, then the loss suddenly starts dropping, reaching 0.1 by round 50. After 300 rounds, all four predictions are very close to their targets.

A pattern a straight line can't learn, a small network with 17 parameters has learned. At no point did anyone tell it "what XOR is"; it just computed the loss, computed the gradients and adjusted the parameters against the gradient, over and over.

What we wrote

Looking back, this lesson's few dozen lines already contain the most central parts of a deep learning framework:

• Automatic differentiation: every operation records local derivatives, and backpropagation passes gradients back with the chain rule.
• Neurons and layers: weighted sum, plus bias, through an activation function.
• The training loop: compute the loss forward, zero the gradients, backpropagate, update the parameters.

It is, of course, extremely slow: every number is a Python object, and a slightly bigger network would have millions of them. Next lesson switches to PyTorch, which does exactly the same thing, but processes a whole batch of numbers (a tensor) at once and computes with optimised low-level code, thousands of times faster.

Exercises

1. Add a relu method to Num: output the input unchanged when it's greater than 0, otherwise output 0. What is its local derivative? Once written, check it against numerical differentiation.
2. Delete the two lines in the training loop that zero the gradients, rerun, and see what happens.
3. Change the number of hidden neurons from 4 to 1 or 2. Can it still learn XOR? Try a few different random seeds too.

Self-check

1. What does the chain rule say, and how does it relate to backpropagation?

The chain rule: the derivative of a chain of operations equals the product of each step's local derivative. Backpropagation applies the chain rule systematically: starting from the loss, it walks back along the computation, multiplying by each step's local derivative as it passes, and ends up with the derivative of the loss with respect to every parameter.

2. In backpropagation, why accumulate gradients with += instead of assigning directly?

A number may be used several times in the computation; in y = x × x, for example, x appears twice. Each use brings a gradient back along its own path, and these must be added together to get the full derivative.

3. Why can't a straight line learn XOR, while a network with a hidden layer and tanh can?

XOR's four points can't be separated by a single straight line. Stacking several layers of straight lines still gives a straight line, so the key is the nonlinear activation function tanh: it lets the network combine curved boundaries that separate the four points.