Overfitting and how to fight it

Last lesson's network scored 100% on the training set and 97.8% on the test set. That 2.2-point gap is the part the network "memorised" rather than truly learned.

When the gap is small it doesn't matter. But with little training data and a big model, the network memorises the entire training set: perfect on the training set, exposed as soon as it meets new data. This is called overfitting. This lesson causes overfitting on purpose, then tries several common ways of fighting it.

python overfitting.py

Three sets of data

This lesson splits the data three ways:

X_pool, X_rest, y_pool, y_rest = train_test_split(X, y, test_size=0.5, random_state=0)
X_val, X_test, y_val, y_test = train_test_split(X_rest, y_rest, test_size=0.5, random_state=0)
X_small, y_small = X_pool[:100], y_pool[:100]  # 故意只用 100 张

可用于训练的图 898 张（先只用其中 100 张），验证集 449 张，测试集 450 张
模型共 301066 个参数

• Training set: used to update the parameters.
• Validation set: used during training to observe and adjust settings (how large the learning rate should be, when to stop), but not to update parameters.
• Test set: looked at once, after all the settings are fixed, as the final score.

Why the extra validation set? Because every time you look at the test score and adjust settings accordingly, a little of the test set's information leaks into your decisions. After dozens of adjustments, the settings you pick may simply happen to suit this particular test set. So tune against the validation set and keep the test set for the end.

The model is deliberately large: two hidden layers of 512 neurons, 300,000 parameters, given only 100 images. That's three thousand parameters per image, so memorising them is trivial.

What overfitting looks like

== 什么都不加
  第   1 轮  训练损失 2.2147 准确率 26.0%  |  验证损失 2.2507 准确率 16.5%
  第  10 轮  训练损失 1.3077 准确率 79.0%  |  验证损失 1.5603 准确率 62.6%
  第  25 轮  训练损失 0.1552 准确率 98.0%  |  验证损失 0.4420 准确率 87.3%
  第  50 轮  训练损失 0.0058 准确率 100.0%  |  验证损失 0.3772 准确率 88.4%
  第 100 轮  训练损失 0.0013 准确率 100.0%  |  验证损失 0.4038 准确率 89.8%
  第 200 轮  训练损失 0.0004 准确率 100.0%  |  验证损失 0.4371 准确率 90.0%
  第 300 轮  训练损失 0.0002 准确率 100.0%  |  验证损失 0.4667 准确率 89.8%
  最终：测试集准确率 88.7%，测试损失 0.5089

(The first epoch's loss of 2.21 is close to the 2.30 from last lesson: at the start the network knows nothing.)

By epoch 50 the training set is 100% correct with a training loss of 0.006, which keeps falling all the way to 0.0002. But validation accuracy hovers around 90%, and validation loss starts rising from 0.377 at epoch 50 to 0.467 at epoch 300.

The two curves have parted: training loss still falling, validation loss rising. That's the sign of overfitting. Validation loss rises because the network grows ever more "confident" about the training set, so when it gets an unseen image wrong it is confidently wrong, and cross-entropy punishes that kind of error heavily.

As a chart, it looks roughly like this:

损失
 │ ＼
 │  ＼  验证损失
 │   ＼＿＿＿＿／￣￣￣￣
 │    ＼
 │     ＼＿  训练损失
 │        ￣￣＿＿＿＿＿＿
 └──────────────────────── 轮
          ↑
    验证损失最低的地方

Trying some remedies

The following are all common remedies for overfitting. Each changes one thing only; everything else is identical (same random seed, same 100 images).

Weight decay: at every update, pull all the weights a little towards 0. The idea is that a model with small parameter values is "simpler" and less likely to overreact to individual samples. The AdamW optimiser's weight_decay parameter controls how hard it pulls:

optimizer = torch.optim.AdamW(model.parameters(), lr=0.001, weight_decay=weight_decay)

Dropout: during training, at every step, randomly let some neurons "rest" (set their output to 0). Here it's set to 0.5, so at every step half the hidden neurons are idle. The network can't rely on particular neurons to memorise particular images, and has to learn more robust patterns. When evaluating, turn dropout off with model.eval(), and back on for training with model.train():

nn.Linear(64, 512), nn.ReLU(), nn.Dropout(dropout),

Early stopping: check the validation loss every epoch and record the parameters at its lowest point. If validation loss hasn't hit a new low for 20 epochs in a row, stop, and go back to that lowest point:

if val_loss < best_val:
    best_val, best_epoch, patience = val_loss, epoch, 0
    best_state = {k: v.clone() for k, v in model.state_dict().items()}
else:
    patience += 1
if early_stop and patience >= 20:  # 验证损失连续 20 轮没有创新低，就停下来
    break

And there's the plainest remedy of all: more data. Use all 898 images available for training.

Results: a somewhat different conclusion

== 权重衰减 weight_decay=0.1
  第  50 轮  训练损失 0.0059 准确率 100.0%  |  验证损失 0.3762 准确率 88.4%
  第 300 轮  训练损失 0.0002 准确率 100.0%  |  验证损失 0.4543 准确率 89.8%
  最终：测试集准确率 88.0%，测试损失 0.5008

== 权重衰减 weight_decay=1.0
  第  50 轮  训练损失 0.0079 准确率 100.0%  |  验证损失 0.3674 准确率 89.1%
  第 300 轮  训练损失 0.0012 准确率 100.0%  |  验证损失 0.4243 准确率 89.3%
  最终：测试集准确率 88.0%，测试损失 0.4574

== dropout 0.5
  第  50 轮  训练损失 0.0561 准确率 100.0%  |  验证损失 0.3784 准确率 88.4%
  第 300 轮  训练损失 0.0003 准确率 100.0%  |  验证损失 0.3879 准确率 90.2%
  最终：测试集准确率 88.2%，测试损失 0.4321

== 早停
  第 55 轮停止：验证损失从第 35 轮之后就没再下降
  最终：测试集准确率 87.3%，测试损失 0.4053

== 什么都不加，但用 898 张图训练
  第   1 轮  训练损失 2.2261 准确率 34.6%  |  验证损失 2.2390 准确率 27.4%
  第  10 轮  训练损失 1.3170 准确率 88.1%  |  验证损失 1.3714 准确率 84.4%
  第  25 轮  训练损失 0.2208 准确率 94.4%  |  验证损失 0.2675 准确率 91.1%
  第  50 轮  训练损失 0.0437 准确率 99.3%  |  验证损失 0.1072 准确率 96.9%
  第 100 轮  训练损失 0.0074 准确率 100.0%  |  验证损失 0.0854 准确率 97.1%
  第 200 轮  训练损失 0.0016 准确率 100.0%  |  验证损失 0.0850 准确率 97.1%
  第 300 轮  训练损失 0.0007 准确率 100.0%  |  验证损失 0.0875 准确率 97.3%
  最终：测试集准确率 96.2%，测试损失 0.1682

测试集汇总：
  什么都不加            准确率 88.7%  损失 0.5089
  权重衰减 0.1         准确率 88.0%  损失 0.5008
  权重衰减 1.0         准确率 88.0%  损失 0.4574
  dropout 0.5      准确率 88.2%  损失 0.4321
  早停               准确率 87.3%  损失 0.4053
  898 张训练数据        准确率 96.2%  损失 0.1682

These results differ somewhat from what many tutorials say, so they deserve a careful look.

Weight decay, dropout and early stopping didn't raise accuracy. Test accuracy for all of them is between 87% and 89%, about the same as the 88.7% with nothing added, and early stopping is even a little lower. With 450 test images, 1 percentage point is only four or five images, so differences this small say nothing about which is better.

They did lower the test loss, from 0.51 to between 0.40 and 0.50, most noticeably with early stopping. In other words, the network gets the same questions right, but it's no longer so "confidently wrong", and its probabilities are more trustworthy. In some situations that matters, for example when you decide from the probability whether to send a result for human review.

More data far outperformed every other remedy. The same model with nothing added, trained on 898 images instead of 100, jumped from 88.7% to 96.2% test accuracy, with test loss falling from 0.51 to 0.17. Validation loss no longer rose noticeably either, settling around 0.085.

How to make sense of it

These regularisation methods (weight decay, dropout and early stopping are collectively called regularisation) aren't useless, but they have their limits. They can ease overfitting, but they can't conjure up information that isn't in the data. With 100 images, an average of only 10 per digit, the many ways each digit can be written simply haven't all been seen. However the model is constrained, it can't recognise a way of writing it has never seen.

When training models in practice, the rough order for fighting overfitting is:

1. First, find a way to get more data. More, and more varied, data is almost always the most effective.
2. Always have a validation set, and watch it. However nicely the training loss falls, if the validation metrics don't improve, it means nothing. Early stopping costs almost nothing; leave it on by default.
3. Then fine-tune with regularisation. The strength of weight decay and dropout has to be found by trying on the validation set; there are no universally right values.
4. The model needn't be huge from the start. With little data, a smaller model is often enough.

LLM training works the same way. Their regularisation isn't complicated; the real key is massive, varied, clean training data. When you train a small GPT in Module 09, you'll see for yourself that with too little data it memorises poems word for word instead of learning to write poetry.

Exercises

1. Set the training data to 200 and then 400 images, and make a table of "training data size → test accuracy". Does accuracy rise evenly with the amount of data?
2. Change the model's hidden layers from 512 to 32 and train on just 100 images. Is test accuracy higher or lower than the 300,000-parameter model's?
3. Set dropout to 0.2 and 0.8 and weight decay to 0.01 and 5.0, find the best combination on the validation set, and finally look at the test set once.

Self-check

1. How can you tell from the training process that overfitting has set in?

Training loss keeps falling while validation loss stops falling or even starts rising, and the two curves part. Training accuracy far above validation accuracy is another sign.

2. If you already have a test set, why do you need a validation set?

Tuning means repeatedly looking at scores and changing settings. If you look at the test set, after many rounds the settings you choose may just happen to suit that test set, and its score no longer represents performance on new data. So tune against the validation set and look at the test set only once, at the end.

3. In this lesson's experiment, weight decay, dropout and early stopping didn't raise accuracy. Why did more data raise it so much?

Regularisation can only constrain the model from relying too much on details of the training set; it can't supply new information. With about 10 images per digit in 100, many ways of writing weren't seen, and no constraint lets the model recognise them. More data lets the model see more ways of writing directly, so it works far better than the other remedies. In this experiment, those methods mainly lowered the test loss, so the model stopped being confidently wrong.