From a small GPT to an LLM

Our small GPT can write lines like "江山古馆响幽幽，山鸟无人见白头". But ask it "what's the line after 床前明月光?", and it will just carry on from those characters and write a poem, because it does only one thing: carry on writing.

DeepSeek and ChatGPT are built on the same structure, yet they answer questions, write code and revise articles as asked. What's missing in between? This lesson has no code; it breaks that journey into steps and explains each one.

Step 1: scale

First compare the numbers (the LLM numbers are quoted from paper abstracts):

                     参数            训练数据
我们的小 GPT         161 万          155 万个字符，训练了约 8 遍
GPT-3（2020）        1750 亿         （论文正文给出了数据量，这里不展开）
DeepSeek-V3（2024）  6710 亿，       14.8 万亿个词元
                     每个词元用 370 亿

DeepSeek-V3 has over four hundred thousand times as many parameters as ours, and about ten million times as much training data. Its full training took 2.788 million H800 GPU hours. Our model trained for 7 minutes on one CPU.

Scale brings more than "writing better". The GPT-3 paper ("Language Models are Few-Shot Learners") found that once a model is large enough, without dedicated training, it can do tasks like translation, question answering and arithmetic just from a few examples in the prompt. That's the few-shot prompting of Module 02, Lesson 2; it works because of scale.

How large is large enough? DeepMind's 2022 Chinchilla paper studied this question. The conclusion: for a fixed amount of compute, a model's parameters and training data should grow in proportion. They trained a 70-billion-parameter model on 4 times the data, and it outperformed the 280-billion-parameter Gopher and the 175-billion-parameter GPT-3 across the board. Many earlier LLMs were "too big a model, too little data".

We saw something similar in Module 08, Lesson 6: going from 100 images to 898 improved results far more than any other technique.

Step 2: pre-training

What we did on Tang poetry is called pre-training: take a large amount of text and have the model predict it token by token. Real pre-training uses web pages, books, code, papers and all sorts of other text, heavily cleaned and deduplicated.

What pre-training produces is called a base model. From massive amounts of text it has learned language, knowledge and some reasoning ability, but its behaviour is still just "carry on writing". Give it a question and it may carry on with more questions, because on web pages a question is often followed by another question.

That's where our small GPT is now, except that all it has read is Tang poetry.

Step 3: instruction tuning

To make a model "answer" instead of "continue", you have to teach it the format of a conversation. The method is direct: prepare lots of "question → good answer" examples and keep training.

<用户>床前明月光的下一句是什么？
<助手>疑是地上霜。出自李白的《静夜思》。

Training works exactly as in pre-training, still predicting the next token, except the data is now conversations like this, and the loss is usually computed only on the "assistant" part. This step is called instruction tuning, also known as supervised fine-tuning (SFT).

Instruction tuning uses far less data than pre-training, but the quality bar is very high. The model's knowledge comes mainly from pre-training; instruction tuning teaches it "how to use that knowledge to respond to people".

In Module 10 you'll do a fine-tune yourself.

Step 4: preference alignment

After instruction tuning, the model answers questions, but the quality of its answers is uneven: sometimes rambling, sometimes invented, sometimes harmful.

"Good answers" are hard to write out as examples in full, but it's easy for people to compare two answers and say which is better. Hence the fourth step: have the model generate several answers to the same question, have people rank them, then use those rankings to train the model to lean towards answers people prefer. This step is called alignment, and the first widely known method was reinforcement learning from human feedback (RLHF).

How much difference does it make? OpenAI's 2022 paper ("Training language models to follow instructions with human feedback") reported that the 1.3-billion-parameter InstructGPT, trained with instruction tuning and human feedback, was preferred in human evaluations over the 175-billion-parameter GPT-3, with 100 times fewer parameters. ChatGPT was built along this path.

LLMs "can chat" mainly thanks to these last two steps; they "know a lot" mainly thanks to the first.

Step 5: training reasoning with reinforcement learning

The 2025 DeepSeek-R1 paper ("DeepSeek-R1: Incentivizing Reasoning Capability in LLMs via Reinforcement Learning") went a step further. Its approach: give the model problems whose answers can be checked automatically, such as maths and programming problems, reward it for right answers and not for wrong ones, and train it repeatedly with reinforcement learning.

The paper reports that with this reward alone, without any human-written worked solutions, the model developed reasoning behaviours of its own such as reflection, verification and trying another approach. The "thinking mode" mentioned in Module 01, where the model outputs some reasoning before giving its answer, is a product of this kind of training.

What our small GPT still lacks

Looking back at our GPT, it has completed step 2, just at a much smaller scale and with data that's only Tang poetry. Following this lesson's steps, to make it more like an "assistant" it would still need:

• More, and more varied, data: having read only Tang poetry, all it can write is Tang poetry.
• A bigger model: 1.61 million parameters can't hold much knowledge.
• Instruction tuning and alignment: so it learns to respond to requests rather than just carrying on writing.

There are also many engineering problems: how to train on thousands of GPUs at once, how to keep training running for months without failure, how to clean data of more than ten trillion tokens. These are beyond the scope of this course. But the model itself you've already built with your own hands.

Exercises

1. Find an open-source model's model card (such as a Qwen or DeepSeek model on Hugging Face), find its parameter count and amount of training data, and see which versions it distinguishes (base, instruct, reasoning and so on).
2. If you have access to a base model (many open-source models release a base version too), send it and the corresponding instruct version the same question and compare their answers.
3. By the Chinchilla paper's conclusion, when a model's parameters double its training data should double too. If our small GPT had 10 times the parameters, do you think these 35,000 poems would still be enough? Try training it and look at the validation loss.

Self-check

1. How does a base model differ from an instruct model?

A base model has only been pre-trained, and its behaviour is to carry on writing from what came before; given a question, it won't necessarily answer. An instruct model is a base model further fine-tuned on lots of "question → answer" conversation data, and has learned to respond to users in conversation format.

2. How does the training method of instruction tuning differ from pre-training?

The method is basically the same: predict the next token, with cross-entropy as the loss. The difference is the data: pre-training uses massive amounts of ordinary text, while instruction tuning uses far less, but very high-quality, conversation data, usually computing the loss only on the answer part.

3. What does the InstructGPT experiment show?

The 1.3-billion-parameter InstructGPT, after instruction tuning and human feedback training, was preferred in human evaluations over the 175-billion-parameter GPT-3. It shows that for how useful a model is, the training method and data sometimes matter more than its size.