Multi-turn chat: the model remembers nothing

Anyone who has used a chatbot has the same intuition: it remembers what I said earlier. Tell it "my name is Xiao Wang" and a few messages later ask "what's my name?", and it answers.

That intuition is wrong, at least as far as the API goes. The model itself remembers nothing; every call is brand new. A chatbot "remembers" because the program sends the entire conversation so far back to the model every time. In this lesson you write a chat program yourself and see this clearly.

First, see that the model doesn't remember

code/03-llm-apps/conversation.py is a command-line chat program. With the --stateless flag, it sends only the current message each time, with no history. I pipe two messages into it:

printf '%s\n' "我叫小王，最近在用 httpx 写一个爬虫。" "我叫什么名字？" | python conversation.py --stateless

你：我叫小王，最近在用 httpx 写一个爬虫。
助手：好的，httpx 很适合爬虫。有什么具体问题需要帮忙吗？  （本轮输入 32 词元）
你：我叫什么名字？
助手：你没有告诉我你的名字。  （本轮输入 23 词元）

From the model's point of view, the second call contains only "我叫什么名字？" (what's my name?), so of course it doesn't know.

Memory is the message list

To make it "remember", all you have to do is put the earlier conversation into messages. The heart of the program is just these lines:

SYSTEM = {"role": "system", "content": "你是一个说话简短的编程助手，每次回答不超过三句话。"}
history = []  # 只放 user 和 assistant 消息，system 每次单独加在最前面

while True:
    user_text = input("你：").strip()
    if not user_text:
        break
    messages = [SYSTEM] + history + [{"role": "user", "content": user_text}]
    response = client.chat.completions.create(model=MODEL, messages=messages, extra_body=NO_THINKING)
    answer = response.choices[0].message.content
    print(f"助手：{answer}  （本轮输入 {response.usage.prompt_tokens} 词元）")
    history += [{"role": "user", "content": user_text}, {"role": "assistant", "content": answer}]

Each turn, what the user said is appended to history as a user message, and the model's answer as an assistant message. On the next call, the whole history is sent along with the new question.

I feed it six messages:

你：我叫小王，最近在用 httpx 写一个爬虫。
助手：好的，httpx 支持同步和异步请求，写爬虫时注意设置 timeout 和重试。有什么具体问题吗？  （本轮输入 32 词元）
你：我叫什么名字？
助手：你叫小王。  （本轮输入 68 词元）
你：我在用什么库做什么？
助手：你在用 httpx 写爬虫。  （本轮输入 81 词元）
你：推荐一个解析 HTML 的库。
助手：推荐 parsel 或 BeautifulSoup，异步场景可选 selectolax。  （本轮输入 102 词元）
你：它和 httpx 怎么配合用？一句话。
助手：把 httpx 返回的 `response.text` 传给解析库（如 `BeautifulSoup(response.text, "lxml")`）即可。  （本轮输入 132 词元）
你：我叫什么名字？
助手：小王。  （本轮输入 170 词元）

Now it remembers the name, and understands that the "它" (it) in "它和 httpx 怎么配合用" (how do I use it with httpx?) means the parsing library it recommended in the previous turn.

Note the input token count at the end of each line: 32, 68, 81, 102, 132, 170, more with every turn. That's because every turn resends the entire conversation so far.

The history keeps getting longer

That brings us to the core problem of multi-turn chat: history only ever grows.

• It gets more expensive. The sixth turn's input is more than 5 times the first's. In a conversation a hundred turns long, every turn pays for the ninety-nine before it. Fortunately the beginning of the history is fixed and can hit the cache from lesson 4 of module 01, which makes it much cheaper, but it still grows.
• It gets slower. The longer the input, the longer the model takes to read it.
• Sooner or later it exceeds the context window. When that happens, the request simply fails with an error.

So any chat program that really goes online has to keep the history's length under control. There are two common ways.

Method 1: truncation

The simplest way: keep only the most recent messages and drop the older ones. That's what conversation.py's --max-messages 4 flag does: once the history exceeds 4 messages, only the latest 2 are kept.

def trim(history):
    if len(history) <= args.max_messages:
        return history
    keep = history[-(args.max_messages // 2):]
    dropped = history[: len(history) - len(keep)]
    print(f"  [丢掉了最早的 {len(dropped)} 条消息]")
    return keep

The same six messages:

你：我叫小王，最近在用 httpx 写一个爬虫。
助手：好的，小王。请问在 httpx 爬虫过程中遇到了什么问题？  （本轮输入 32 词元）
你：我叫什么名字？
助手：你叫小王。  （本轮输入 55 词元）
你：我在用什么库做什么？
助手：你在用 httpx 写爬虫。  （本轮输入 68 词元）
  [丢掉了最早的 4 条消息]
你：推荐一个解析 HTML 的库。
助手：推荐 BeautifulSoup，配合 httpx 用很方便。  （本轮输入 45 词元）
你：它和 httpx 怎么配合用？一句话。
助手：用 `BeautifulSoup(httpx.get(url).text, "html.parser")` 就能解析。  （本轮输入 71 词元）
  [丢掉了最早的 4 条消息]
你：我叫什么名字？
助手：我不知道你的名字，你还没告诉我。  （本轮输入 60 词元）

The input stays at a few dozen tokens and stops growing. The price is the last question: it has forgotten the user's name is Xiao Wang, because that message was dropped long ago.

Truncation suits cases where "the last few turns matter most and earlier content can be forgotten", such as small talk or one-off simple Q&A. Real projects usually truncate by token count rather than message count; message count is used here to keep the demo simple.

One thing to watch: when truncating, drop question-and-answer pairs together. Don't drop a user message and keep its assistant reply, or the conversation the model sees is broken. The code above always keeps an even number of messages, so it doesn't have this problem.

Method 2: summarization

A somewhat better way is not to throw old messages away, but to have the model compress them into a summary:

def summarize(messages):
    text = "\n".join(f"{m['role']}: {m['content']}" for m in messages)
    response = client.chat.completions.create(
        model=MODEL,
        messages=[{"role": "user", "content": "把下面这段对话压缩成几句话的摘要，保留人名、项目、偏好等以后可能用到的事实：\n\n" + text}],
        extra_body=NO_THINKING,
    )
    return response.choices[0].message.content

When truncating, the dropped part goes to summarize, and the resulting summary is placed as a pair of messages at the very start of the history:

    summary = summarize(dropped)
    return [{"role": "user", "content": f"（之前对话的摘要：{summary}）"},
            {"role": "assistant", "content": "好的，我记住了。"}] + keep

Run it again with the --summarize flag:

你：我叫小王，最近在用 httpx 写一个爬虫。
助手：好的小王，httpx 适合爬虫，支持异步和 HTTP/2。有具体问题可以问我。  （本轮输入 32 词元）
你：我叫什么名字？
助手：你叫小王。  （本轮输入 63 词元）
你：我在用什么库做什么？
助手：你在用 httpx 写爬虫。  （本轮输入 76 词元）
  [把最早的 4 条消息压缩成了摘要：小王在用 httpx 写爬虫；httpx 支持异步和 HTTP/2，适合爬虫。]
你：推荐一个解析 HTML 的库。
助手：推荐 BeautifulSoup 或 lxml，前者易用，后者更快。  （本轮输入 83 词元）
  [把最早的 4 条消息压缩成了摘要：小王在用 httpx 写爬虫；httpx 支持异步和 HTTP/2，适合爬虫。]
你：它和 httpx 怎么配合用？一句话。
助手：用 httpx 获取响应后，把 `resp.text` 传给 BeautifulSoup 解析即可。  （本轮输入 95 词元）
  [把最早的 4 条消息压缩成了摘要：小王在用 httpx 写爬虫，需要 HTML 解析库；推荐了 BeautifulSoup（易用）和 lxml（更快）。]
你：我叫什么名字？
助手：你叫小王。  （本轮输入 104 词元）
  [把最早的 4 条消息压缩成了摘要：小王在用 httpx 写爬虫，需 HTML 解析库；推荐了 BeautifulSoup（易用）和 lxml（更快）。与 httpx 配合时，把 `resp.text` 传给 BeautifulSoup 解析即可。]

On the last question, it remembers. The summary kept key facts like "小王" (Xiao Wang), "httpx" and "爬虫" (web scraper). There are fewer input tokens than without truncation (104 against 170), but more than plain truncation, because the summary itself takes up space.

Summarization has costs too:

• Every compression is one more model call, which costs money and makes that turn slower. You can do it asynchronously in the background, or wait until the history reaches a certain length before compressing, rather than every turn.
• Summaries lose detail. They keep only what the model thinks is important. If the user mentioned "我的服务器在香港" (my server is in Hong Kong) in passing in turn 3, it may not be in the summary.
• Summaries can be wrong. The model may summarize a fact incorrectly.

Look carefully at the output above: each compression recompresses "the old summary + the newly dropped messages", so the summary keeps rolling forward.

Which to choose

Common problems

The system message ended up in history and got truncated with it: the system message should be put separately at the very front every time and never be part of truncation. The code above stores it apart from history for exactly this reason.

Different users' histories got mixed up: in a web service, every user and every session needs its own history, usually stored in a database or cache and told apart by session ID. A global history variable only suits a command-line program used by one person.

Exercises

1. Run conversation.py, chat with it for more than ten turns, and watch how the input tokens change each turn. Then chat again with --max-messages 6 and find the moment it forgets something.
2. Rewrite trim so it truncates by token count rather than message count: when the history's total tokens exceed 500, drop messages in pairs starting from the oldest. You can judge by the previous turn's usage.prompt_tokens.
3. Mention a detail in passing in turn 3 (such as "我的服务器在香港", my server is in Hong Kong) and ask about it in turn 8. Run it once with truncation and once with summarization, and see whether the summary kept that detail. If not, try changing summarize's prompt.

Self-check

1. How does the model "remember" what you said earlier?

The model itself remembers nothing. On every call, the program sends all the earlier user and assistant messages to the model along with the new question, and the model "reads" what came before from those messages.

2. Why is each turn of a multi-turn chat more expensive than the last?

Every turn resends the entire conversation history as input, so the longer the history, the more input tokens and the higher the cost. The unchanged beginning of the history can hit the cache and is much cheaper, but the total input still grows.

3. What are the downsides of truncation and of summarization?

Truncation throws earlier information away entirely, such as the user's name, so the program can't answer when asked about it later. Summarization keeps key facts, but every compression is an extra model call, it loses details the model considers unimportant, and it can get things wrong.