The model does one thing: predict the next token

You've probably heard it said that "a large language model is really just predicting the next word". That's true, but on its own it's hard to picture how a program that "predicts the next word" can write code, solve math problems and answer questions. More importantly, the idea explains a lot of what you'll run into later: why the model confidently makes things up, why the same question is sometimes right and sometimes wrong, and why the way you word a prompt matters so much.

In this lesson we make the model show its "thinking" at every step.

Watching it choose step by step

The last lesson mentioned that the API has a logprobs parameter. Turn it on and the model tells you the probability of every token it outputs, and can also list the most likely candidates at each position. DeepSeek supports this parameter in non-thinking mode:

import math
import os

from openai import OpenAI

client = OpenAI(
    api_key=os.environ["LLM_API_KEY"],
    base_url=os.environ.get("LLM_BASE_URL", "https://api.deepseek.com"),
)
MODEL = os.environ.get("LLM_MODEL", "deepseek-flash")


def show_candidates(prompt, max_tokens=6):
    response = client.chat.completions.create(
        model=MODEL,
        messages=[{"role": "user", "content": prompt}],
        max_tokens=max_tokens,
        logprobs=True,  # 返回每个输出词元的概率
        top_logprobs=5,  # 同时返回每一步概率最高的 5 个候选
        extra_body={"thinking": {"type": "disabled"}},
    )
    print(f"问：{prompt}")
    print(f"答：{response.choices[0].message.content}")
    for step, item in enumerate(response.choices[0].logprobs.content, 1):
        # 接口返回的是概率的自然对数，用 exp 变回 0～1 之间的概率
        options = "  ".join(f"{c.token}({math.exp(c.logprob):.1%})" for c in item.top_logprobs)
        print(f"  第 {step} 步选了 {item.token!r:8} 候选：{options}")
    print()


show_candidates("床前明月光，下一句是什么？只回答下一句。")
show_candidates("请续写这句话，只写接下来的几个字：周末我打算去")

What I got (a few of the candidates were newlines, which break lines when printed, so I removed them):

问：床前明月光，下一句是什么？只回答下一句。
答：疑是地上霜。
  第 1 步选了 '疑'      候选：疑(100.0%)  疑似(0.0%)  下一(0.0%)  低头(0.0%)  举(0.0%)
  第 2 步选了 '是'      候选：是(100.0%)  是中(0.0%)  是全(0.0%)  <｜｜end▁of▁sentence｜｜>(0.0%)  是高(0.0%)
  第 3 步选了 '地上'     候选：地上(100.0%)  地上的(0.0%)  明月(0.0%)  银河(0.0%)  地下(0.0%)
  第 4 步选了 '霜'      候选：霜(100.0%)  <｜｜end▁of▁sentence｜｜>(0.0%)  妆(0.0%)  箱(0.0%)  光(0.0%)
  第 5 步选了 '。'      候选：<｜｜end▁of▁sentence｜｜>(96.4%)  。(3.6%)

问：请续写这句话，只写接下来的几个字：周末我打算去
答：爬山，顺便看看日出。
  第 1 步选了 '爬山'     候选：爬山(74.0%)  图书馆(12.7%)  公园(5.9%)  山里(3.6%)  超市(1.1%)
  第 2 步选了 '，'      候选：，(67.0%)  。(31.9%)  <｜｜end▁of▁sentence｜｜>(0.7%)  放松(0.3%)  /(0.1%)
  第 3 步选了 '顺便'     候选：顺便(48.8%)  呼吸(44.9%)  放松(3.3%)  亲近(1.1%)  或者(0.8%)
  第 4 步选了 '看看'     候选：看看(51.6%)  看(20.3%)  拍(19.0%)  透(3.9%)  呼吸(3.4%)
  第 5 步选了 '日出'     候选：日出(86.1%)  山(4.7%)  日落(4.7%)  秋天的(2.7%)  春天的(0.3%)
  第 6 步选了 '。'      候选：。(100.0%)  <｜｜end▁of▁sentence｜｜>(0.0%)

Every step of "疑是地上霜" (the famous next line of Li Bai's poem) is 100% after rounding. The model has seen this line countless times in its training data, and there's no suspense at all.

Continuing "周末我打算去" (this weekend I'm planning to go…) is different. At step 3, "顺便" (on the way, also) and "呼吸" (breathe) are almost tied, 48.8% against 44.9%. This time it drew "顺便", which led to "顺便看看日出" (and catch the sunrise while I'm at it); if it had drawn "呼吸", what followed might have been "呼吸新鲜空气" (breathe some fresh air). An answer is full of forks like this, and which way each one goes decides where the whole answer ends up. That's why the same question gets a different answer each time; lesson 3 goes into it.

One more detail: at step 5 of the first question, the most likely candidate is <｜｜end▁of▁sentence｜｜> (96.4%), with only a 3.6% chance of a full stop, and the full stop is what got drawn. end▁of▁sentence is a special token meaning "I'm done". The model doesn't know when to stop; it simply predicts this "end" token at some step, and the program stops generating when it sees it. That's where the last lesson's finish_reason: stop comes from.

How an answer is generated

Put what we just saw together, and generating an answer is a loop:

输入："床前明月光，下一句是什么？"
  → 模型算出下一个词元的概率分布 → 抽中"疑" → 把"疑"接到输入后面
输入："床前明月光，下一句是什么？疑"
  → 模型算出下一个词元的概率分布 → 抽中"是" → 接到后面
输入："床前明月光，下一句是什么？疑是"
  → ……
一直重复，直到抽中"结束"词元，或者达到 max_tokens 的上限

At every step, the model takes everything so far (your question plus what it has already written) as input and predicts just one token. Every character it writes becomes input for the next step.

This mechanism has some direct consequences:

• Longer output is slower and more expensive. Generating 100 tokens takes 100 steps. That's one reason output costs more than input.
• It can't go back and edit. If it writes a wrong word early on, it can only carry on from that mistake, or talk its way back round later. Part of why thinking mode helps is that it gives the model a place to draft: the draft can contain mistakes that get corrected, and the formal answer contains only the final conclusion.
• What comes first shapes what comes after. Ask the model to "give the conclusion first, then the reasons" and it writes the conclusion, then finds reasons for it; ask it to "analyze first, then conclude" and the conclusion is written on top of the analysis. The two can differ a lot in accuracy; lesson 3 of module 02 runs the experiment.

From continuation machine to assistant

A model that only "predicts the next word" really did start out able only to continue text. Give it "床前明月光" (the first line of the poem) and it carries on with the poem; give it "怎么学 Python？" (how do I learn Python?) and it might carry on with ten more similar questions, because lists of questions online often look like that.

It became an assistant that answers questions in roughly three stages:

1. Pre-training: practising "predict the next token" on an enormous amount of text (web pages, books, code). To predict well, the model has to pick up grammar, facts and patterns of reasoning. This step decides what the model "knows", and it takes the vast majority of the compute.
2. Instruction tuning: further training on a large number of "question → good answer" conversations, so it learns to "answer a question when it sees one".
3. Preference alignment: people or other models score its answers, and it is trained to lean towards answers that are helpful, honest and safe.

Lesson 7 of module 09 covers these three steps in detail. For now it's enough to know that the assistant's behaviour is trained, and that the underlying mechanism never changed from start to finish: it is still predicting the next token.

So how do the system, user and assistant parts of a conversation become "a piece of text" for the model? DeepSeek's published tokenizer configuration contains a "chat template" that specifies how messages are joined. Stripped of details, a simple conversation gets assembled like this:

<｜begin▁of▁sentence｜>你是一个说话简短的助手。<｜User｜>Python 里的列表和元组有什么区别？<｜Assistant｜>

<｜User｜> and <｜Assistant｜> are special tokens in the vocabulary. Once it's assembled, the model's job is still continuation: predicting onwards after <｜Assistant｜>. The last lesson noted that "there are more input tokens than characters in the messages"; the extras are these formatting markers. (This is the template shipped in the tokenizer archive. The templates for thinking mode and tool calling are more complicated, but the principle is the same.)

Where hallucination comes from

Once you understand "predict the next token", you can understand why models make things up, which is what's usually called hallucination.

At each step the model picks "what looks most like what comes next", not "what is true". Most of the time the two agree, because in the training data the true content is the most common. But when the model knows little about something, or the question itself implies a premise that doesn't exist, "what looks most likely" and "what is true" come apart. The model has no "I don't know" switch, unless training has taught it to say "I don't know" in such cases.

I asked deepseek-flash a specific question about httpx (thinking off):

问：httpx 0.27.2 这个版本的更新日志里具体修复了哪些 bug？逐条列出。

答：httpx 0.27.2 的更新日志中，没有列出任何 bug 修复。

该版本（发布于 2024 年 9 月 25 日）是一个纯安全修复版本，只包含一条内容：

- 修复了一个安全漏洞：在特定条件下，之前已重定向的请求在后续重定向中会错误地保留
  Authorization 头，可能导致凭证泄露到非同源主机。该问题通过 CVE-2024-46319
  相关补丁修复，行为改为在跨源重定向时剥离 Authorization 头。

(The original answer had another paragraph after this, which I've left out.)

The tone is sure of itself, with a date, a CVE number and technical details; it looks very credible. I went through the CHANGELOG.md in the httpx repository, and the entry for 0.27.2 actually says:

## 0.27.2 (27th August, 2024)

### Fixed

* Reintroduced supposedly-private `URLTypes` shortcut. (#2673)

The date is wrong, the content is wrong, and the CVE number and the "security-only release" claim are all invented. The model has seen plenty of changelogs along the lines of "version so-and-so of library such-and-such fixes the Authorization header leaking on redirect", so it generated a piece of text that "looks like an httpx changelog".

To be fair, the model does well in front of obvious traps. When I asked "httpx's Client has a retry_on_status parameter; how do I use it?" (no such parameter exists), it said straight away that "there is no such parameter" and gave a correct alternative (I checked it against the httpx docs and the source of the third-party library it recommended, and both matched). When I asked it to recommend three academic papers studying httpx's connection pool, it said it hadn't found any such papers and warned me not to ask it to invent them. The trouble is in questions that look perfectly normal: a specific version number, a specific piece of history. The more specific and obscure, the easier it is to invent.

So remember:

• Fluent and confident doesn't mean correct. Invented content reads just as smoothly as true content.
• The more specific, the more it needs checking. Version numbers, dates, figures, names, API parameters and citations are where it goes wrong most.
• The most effective fix is to give the model something to go on. Put the real material into the input and have it answer from that material, which is module 04's RAG; or let it look things up itself, which is module 05's agents. The course's running project, RepoBot, deliberately keeps this problem in v1 so you can watch it answer wrongly, then fixes it in v2.

Exercises

1. Try show_candidates with some different openings: "我最喜欢的编程语言是" (my favourite programming language is), "1+1=", "从前有座山" (once upon a time there was a mountain). Which has the most concentrated candidates, and which the most spread out?
2. Change top_logprobs to 20 and see whether the candidates for "周末我打算去" include any you didn't expect.
3. Ask the model a specific question you know very well but that's fairly obscure (a detail of your company's product, a small street in your hometown, a particular version of an open-source library you know), with thinking off, and see whether it makes things up. Ask again with thinking on; is there a difference? Write down what you find.

Self-check

1. How does the model know when an answer should end?

It doesn't "know". The vocabulary contains a special "end" token; at some step the model predicts that token with high probability and draws it, and the program stops generating. The other case is hitting the max_tokens limit and being cut off.

2. Why is it hard for a model to fix a mistake it notices halfway through writing?

At every step the model takes everything it has written so far as input and predicts only the next token; there's no mechanism for going back to edit. An earlier mistake becomes input for later steps and affects what's predicted next. Thinking mode eases this somewhat: the model can draft and correct itself while thinking, and write only the conclusion into the formal answer.

3. Why would a model invent a CVE number that doesn't exist?

At each step the model picks "what looks most like what comes next", not "what is true". It has seen a great deal of text like "version X fixes security issue Y, CVE-xxxx", and when it doesn't know the specific facts, it follows that pattern and generates something that looks plausible. The more specific and obscure a fact, the more likely it is to be invented, so this kind of information must always be checked.