Linear regression with numpy

In Part 1 we were always using LLMs that had already been trained. Part 2 answers a question: how are models trained?

We start with the simplest example. Not a neural network, not an LLM, just a straight line. Don't underestimate it: every concept in the next few lessons, loss, gradient, learning rate, backpropagation, can be seen clearly on this line first, then carried over unchanged to neural networks, and finally to GPT.

This lesson needs only numpy:

uv add numpy

The problem: predicting price from floor area

Suppose you've collected the floor area and sale price of 50 homes, and want to find a pattern so that when you see a home's area in future, you can estimate its price.

We make up the data ourselves, so we know what the "right answer" is and can check against it:

import numpy as np

rng = np.random.default_rng(0)

# 造一组数据：房价（万元）= 1.2 × 面积（平方米）+ 20，再加上一些随机的波动
area = rng.uniform(40, 150, size=50)
price = 1.2 * area + 20 + rng.normal(0, 12, size=50)

default_rng(0) fixes the random seed, so the data you get is exactly the same as mine. The true pattern is "12,000 yuan per square metre, plus 200,000", but every home has some random deviation (standard deviation 120,000), just as in the real world orientation, floor and decoration push prices up or down. (Prices in the code are in units of 10,000 yuan.)

The first 5 homes look like this:

前 5 套房子：
   110.1 平方米   156.4 万元
    69.7 平方米    89.1 万元
    44.5 平方米    73.4 万元
    41.8 平方米    78.1 万元
   129.5 平方米   159.9 万元

The model: a straight line

The simplest guess is that price and area are related by a straight line.

预测价格 = w × 面积 + b

w is the slope: how much the price rises for each extra square metre. b is the intercept: the price when the area is 0 (think of it as a fixed base price). These two numbers are the model's parameters.

def predict(w, b, x):
    """模型：一条直线。w 是斜率（每平方米多少万），b 是截距。"""
    return w * x + b

"Training a model" comes down to this: finding the best set of parameters. For this model, that means finding the best w and b.

LLMs like GPT are the same, except their parameters number not 2 but billions or hundreds of billions, and the model isn't a straight line but an extremely complex function. In Module 09 you'll train a small GPT with over a million parameters yourself, and at heart it's still doing this.

Loss: how to measure "good"

What does "the best straight line" mean? We need a concrete standard that produces a number.

For every home, the model gives a predicted price, which differs from the real price by some amount. Square all the differences and take the average; that number is called the mean squared error (MSE):

均方误差 = 平均值( (预测价格 - 真实价格)² )

def mse(w, b):
    """损失：预测值和真实值之差的平方，取平均。"""
    errors = predict(w, b, area) - price
    return np.mean(errors ** 2)

Why square them? Two reasons. First, differences can be positive or negative, and averaging them directly would let them cancel out; squared, they're all positive. Second, squaring makes big errors more "visible": a difference of 10 squares to 100, a difference of 30 squares to 900. The model is forced to attend first to the points it gets badly wrong.

This number measuring "how bad the model is" is generally called the loss. The smaller the loss, the better the model. The goal of training can be put more precisely: find the parameters that minimise the loss.

A few guesses

First guess a few pairs of w and b by intuition and see what the loss is:

for w, b in [(1.0, 0.0), (1.0, 30.0), (1.5, 0.0), (1.2, 20.0)]:
    print(f"  w={w:<4} b={b:<5} 均方误差 {mse(w, b):9.1f}")

随手猜几条直线，看看误差：
  w=1.0  b=0.0   均方误差    1810.1
  w=1.0  b=30.0  均方误差     310.8
  w=1.5  b=0.0   均方误差     284.5
  w=1.2  b=20.0  均方误差     146.6

The line w=1.0, b=0 is far from the data, with a loss of 1810. Add an intercept of 30 and the loss drops to 311. w=1.2, b=20 is exactly the true pattern we used to make the data, with a loss of only 146.6.

Notice that even the true pattern doesn't have a loss of 0. Because the data contains random fluctuation, no straight line can pass through every point. The loss can never reach 0, which is normal in real problems.

The dumbest way: try everything

Since we can compute the loss, let's try every possible w and b and pick the pair with the smallest loss. w from 0 to 3 in steps of 0.01; b from -50 to 100 in steps of 0.5:

best = (None, None, float("inf"))
tries = 0
for w in np.arange(0.0, 3.0, 0.01):
    for b in np.arange(-50.0, 100.0, 0.5):
        tries += 1
        loss = mse(w, b)
        if loss < best[2]:
            best = (w, b, loss)

网格搜索：试了 90000 条直线，用了 0.2 秒
  最好的一条：w=1.26 b=14.5，均方误差 143.1

Out of 90,000 lines tried, the best is w=1.26, b=14.5, with a loss of 143.1, slightly lower even than the true pattern's (w=1.2, b=20) 146.6. That's not surprising: we have only 50 data points, with random fluctuation, and the line that best fits those 50 points isn't necessarily the one that generated them.

The w and b found differ from the true values, and this will stay with us from now on: a model learns the parameters that "best explain the data in front of it"; if the data is skewed, the learned parameters are skewed.

Grid search has another, easily missed problem: it can only find points on the grid. With w tried every 0.01, if the truly best w is 1.2571, the best it can give is 1.26. Make the grid finer, and the number of tries multiplies. Next lesson's gradient descent has no such limit; it can keep closing in on the best value.

Why this approach doesn't work

Two parameters, a few hundred values each, is 90,000 combinations, taking 0.2 seconds.

With 3 parameters it would be tens of millions of combinations; with 10 parameters the number is astronomical, and the lifetime of the universe wouldn't be enough to try them all. Even the smallest neural network has dozens or hundreds of parameters, and Module 09's small GPT has over a million.

We need a far smarter method: not trying blindly, but knowing at every step which way to change the parameters and by how much. That's next lesson's gradient descent.

Exercises

1. Change the random fluctuation in the data from 12 to 0 (rng.normal(0, 0, size=50)) and rerun. Is the best line exactly w=1.2, b=20? What's the loss?
2. Replace the loss function with "mean absolute error", np.mean(np.abs(errors)), and run the grid search again. Is the line it finds the same as the one mean squared error found?
3. Add an extreme point to the data: a 60-square-metre home that sold for 5 million (area = np.append(area, 60); price = np.append(price, 500)). Run the grid search with mean squared error and with mean absolute error. Which is more affected by this point, and why?

Self-check

1. What does "training a model" mean?

Finding the best set of parameters, the ones that make the model's loss on the available data as small as possible. For a straight line, that's finding the best slope w and intercept b. LLMs are the same, only with far more parameters and a far more complex form.

2. Why does mean squared error square the errors instead of averaging them directly?

Errors can be positive or negative, and averaging them directly lets them cancel out, so even a very bad model could get an average error near 0. Squared, they're all positive, and big errors are magnified more, so the model prioritises reducing the predictions it gets badly wrong.

3. Grid search took only 0.2 seconds in this example. Why can't it be used to train neural networks?

The number of combinations to try grows exponentially with the number of parameters. 2 parameters is 90,000 combinations, 3 is tens of millions, and neural networks have hundreds, thousands or even billions of parameters, giving an astronomical number of combinations that could never all be tried.