Gradient descent

Last lesson found the best line with grid search, and also saw its dead end: with more parameters, the number of combinations to try becomes astronomical.

Imagine standing blindfolded on a hillside and needing to get to the lowest point of the valley. You can't see the whole mountain, but you can feel with your feet which way the ground slopes. So take a small step downhill, feel again, take another step. Keep going and you end up at the bottom of the valley.

This is gradient descent. The "mountain" is the loss, where you stand is the current parameters, and "which way it slopes" is the gradient.

The derivative: it's just the slope

First look at just one parameter, w, holding b = 0. If we nudge w a tiny bit to the right, how much does the loss change? The change divided by the distance moved is the slope of the loss at that point. Let's measure it numerically:

def mse(w, b, x=area, y=price):
    return np.mean((w * x + b - y) ** 2)


w, b = 1.0, 0.0
for h in [0.1, 0.01, 0.001]:
    slope = (mse(w + h, b) - mse(w, b)) / h
    print(f"  w 往右挪 {h:<6}，损失变化 / 挪动距离 = {slope:10.2f}")

== 1. 在 w=1.0, b=0 这一点，损失对 w 的斜率
  w 往右挪 0.1   ，损失变化 / 挪动距离 =   -7300.21
  w 往右挪 0.01  ，损失变化 / 挪动距离 =   -8256.23
  w 往右挪 0.001 ，损失变化 / 挪动距离 =   -8351.83

The smaller the nudge, the more stable the measured slope, gradually approaching a fixed value. That "slope as the nudge becomes infinitely small" is the derivative. It tells us two things:

• The sign: here it's negative, meaning that moving w right (making it bigger) makes the loss smaller. So w should be increased.
• The size: a large absolute value means the slope is steep here and the loss is very sensitive to w.

The gradient: one derivative per parameter

We have two parameters, w and b. Take the derivative with respect to each separately (treating b as a constant when differentiating with respect to w, and vice versa), and the two numbers together are called the gradient.

Measuring numerically is too slow and imprecise. Fortunately, the derivative of mean squared error can be written as a formula. The loss is mean((w×x + b - y)²), and by the rules of differentiation:

损失对 w 的导数 = 平均( 2 × (w×x + b - y) × x )
损失对 b 的导数 = 平均( 2 × (w×x + b - y) )

You don't need to memorise this derivation; just know that it's "the slope-measuring process above, computed exactly in one go with algebra". In code:

def gradients(w, b, x=area, y=price):
    error = w * x + b - y
    dw = np.mean(2 * error * x)  # 损失对 w 的导数
    db = np.mean(2 * error)  # 损失对 b 的导数
    return dw, db

== 2. 公式算出的梯度：dw=-8362.45，db=-79.98

dw = -8362.45, very close to the numerical measurement above (-8351.83 with a nudge of 0.001). This is a good habit: check a gradient formula you derived yourself against the numerical method, and you'll catch many mistakes. We'll do it again next lesson when writing backpropagation.

The gradient descent loop

The gradient points in the direction where the loss increases fastest. We want the loss to decrease, so we take a small step in the opposite direction:

w = w - 学习率 × dw
b = b - 学习率 × db

The learning rate controls how big each step is. Starting from w = 0, b = 0, repeat that step:

def descend(lr, steps, x=area, y=price, report=()):
    """从 w=0, b=0 出发走 steps 步。发散时返回 None。"""
    w, b = 0.0, 0.0
    for step in range(1, steps + 1):
        dw, db = gradients(w, b, x, y)
        w -= lr * dw
        b -= lr * db
        if abs(w) > 1e6:  # 越走越远，已经发散了
            print(f"  第 {step} 步：w 已经变成 {w:.3g}，发散了")
            return None
        if step in report:
            print(f"  第 {step:5d} 步：w={w:7.4f} b={b:7.3f} 损失 {mse(w, b, x, y):10.2f}")
    return w, b

That's the entire core logic of training. When Module 09 trains GPT, the loop has exactly the same structure; there are just more parameters, and the gradients are computed with next lesson's backpropagation instead.

Taking a walk: not so smooth

With a learning rate of 0.00005, take 20,000 steps:

== 3. 学习率 0.00005，走 20000 步
  第     1 步：w= 1.4804 b=  0.014 损失     244.38
  第    10 步：w= 1.3935 b=  0.014 损失     164.22
  第   100 步：w= 1.3934 b=  0.027 损失     164.18
  第  1000 步：w= 1.3922 b=  0.155 损失     163.81
  第  5000 步：w= 1.3871 b=  0.711 损失     162.27
  第 20000 步：w= 1.3695 b=  2.615 损失     157.43

In the first step, the loss drops from several thousand to 244; after 10 steps, to 164. Then it barely moves: after 20,000 steps the loss has only fallen to 157, while last lesson's grid search found a lowest loss of 143. b is surprisingly slow: in 20,000 steps it has moved only from 0 to 2.6, still far from the best value (about 14.5).

Why? The next two experiments reveal the reason.

Learning rate: too small is slow, too large explodes

Try a few learning rates, 1,000 steps each:

== 4. 不同的学习率（都走 1000 步）
  学习率 1e-05  ：w=1.3932 b=0.041 损失 164.14
  学习率 5e-05  ：w=1.3922 b=0.155 损失 163.81
  学习率 9e-05  ：w=1.3912 b=0.268 损失 163.50
  第 115 步：w 已经变成 1.03e+06，发散了
  学习率 0.0001 ：发散

The larger the learning rate, the faster it goes and the further b moves. But raising it from 0.00009 to 0.0001, only a tiny bit more, makes it blow up completely: at step 115, w is over a million.

Divergence happens like this: the slope in the w direction is too steep (we measured -8362 above), so a slightly larger step jumps right over the bottom of the valley and lands higher on the other side; the slope there is even steeper, so the next step jumps further... Each step lands further from the bottom than the last, and the values grow and grow until they overflow.

The root of the problem: the slope is too steep in the w direction and too gentle in the b direction. dw is -8362, db only -80, a factor of a hundred. The learning rate has to be small enough to keep w from diverging, so b can only move extremely slowly. One learning rate can't serve both directions.

Standardisation: make the slope about equally steep in every direction

Why is the slope so steep in the w direction? Because the area values are large, from tens to over a hundred square metres. In the gradient formula dw = mean(2 × error × x), the larger x is, the larger dw is.

The fix is to convert the area into numbers with "mean 0 and standard deviation 1", which is called standardisation:

mean, std = area.mean(), area.std()
area_n = (area - mean) / std

After standardisation, the area values are all roughly between -2 and 2, and the slopes in the w and b directions are about equally steep. Now we can safely use a learning rate of 0.1 and take only 100 steps:

== 5. 面积标准化之后（均值 98.0，标准差 32.1），学习率 0.1 只走 100 步
  第     1 步：w= 8.0605 b= 27.588 损失   13360.27
  第    10 步：w=35.9749 b=123.129 损失     381.15
  第    50 步：w=40.3018 b=137.939 损失     143.05
  第   100 步：w=40.3023 b=137.941 损失     143.05
  换算回原单位：w=1.2571 b=14.806

It reaches the lowest point in 50 steps, with a loss of 143.05, slightly lower even than last lesson's grid search result of 143.1 (grid search only tried every 0.01, so the best point might fall between two grid points). Converted back to the original units, w = 1.2571, b = 14.806.

What 20,000 steps couldn't achieve before, 50 steps achieve after standardisation. This is a very practical lesson for training neural networks: input values should be on similar scales. The various kinds of "normalisation" you'll see later (LayerNorm, BatchNorm) all start from the same idea.

Gradient descent in summary

• The derivatives of the loss with respect to each parameter together form the gradient, which points where the loss increases fastest.
• Each step moves a little in the opposite direction of the gradient, with the step size set by the learning rate.
• Too small a learning rate is slow; too large diverges. It's the number that most needs tuning in training.
• When inputs are on very different scales, training is hard to tune; standardise first.

Our straight line has only two parameters, so the gradient formula can be derived by hand. But a neural network has hundreds or thousands of parameters, layer upon layer, and deriving each parameter's derivative by hand is impossible. Next lesson covers how to have the computer calculate all the gradients automatically: backpropagation.

Exercises

1. On the standardised data, change the learning rate to 0.5, 1.0 and 1.1, take 100 steps each, and look at the results. Roughly what is the critical learning rate?
2. Without standardisation, using only a learning rate of 0.00009, how many steps does it take for the loss to fall below 144? Estimate, then run it to check.
3. In numerical differentiation, is a smaller h always better? Change h to 1e-10 and 1e-15 and look at the measured slopes. Did the result actually get worse? Think about why (hint: computers store decimals with limited precision).

Self-check

1. Why does gradient descent move in the "opposite direction" of the gradient?

The gradient points in the direction where the loss increases fastest. Our goal is to decrease the loss, so we move the opposite way.

2. Raising the learning rate only from 0.00009 to 0.0001 made training diverge. What happened?

In the w direction the loss slopes very steeply. With a slightly larger learning rate, one step jumps over the lowest point and lands higher on the other side, where the slope is steeper still, so the next step jumps further. Each step ends up further from the lowest point than the last, and the values keep growing until they overflow.

3. Why did standardising the area make training hundreds of times faster?

Without standardisation, the area values are large, so the gradient in the w direction is a hundred times that in the b direction. The learning rate has to accommodate the steep w direction and be set very small, so b moves extremely slowly. After standardisation, both directions slope about equally, a much larger learning rate can be used, and both parameters reach the lowest point quickly.