Context windows and cost

Context windows now routinely run to hundreds of thousands or millions of tokens. Many people's first reaction is: why bother with RAG, then? Why not throw all the material in at once?

Sometimes you really can. But first you need to know what that costs per call, how much slower it is, and whether the model can find the one sentence you need in a huge pile of text. This lesson experiments with the whole httpx documentation and measures each of those.

What a context window is

The context window is the maximum number of tokens a model can handle at once, and input and output together can't exceed it. As of September 2026, the context window of both deepseek-flash and deepseek-v4-pro is 1 million tokens, with at most 384,000 output tokens per call.

In one request, all of the following take up context:

┌──────────────────────────────────────────────┐
│ system 消息：规则、人设                         │
│ 之前的对话记录（user 和 assistant 轮流）          │
│ 你塞进去的资料：文档、检索结果、工具返回的内容      │  输入
│ 这一轮用户的问题                                │
├──────────────────────────────────────────────┤
│ 模型的思考过程（如果开了思考模式）                  │  输出
│ 模型的正式回答                                  │
└──────────────────────────────────────────────┘

For a simple Q&A these may add up to only a few hundred tokens. But in multi-turn chats, Q&A over documents, and when the model calls tools, they grow quickly. When the context runs out, the request simply fails with an error.

Experiment: put the whole documentation in

The official httpx documentation is 20-odd Markdown files, 116,933 characters in total, kept in the course repository under data/httpx-docs/. The script below runs two experiments: first it puts all of the documentation into the system message and asks the same question twice in a row; then it hides a sentence in the documentation and sees whether the model finds it.

import os
import pathlib

from openai import OpenAI

client = OpenAI(
    api_key=os.environ["LLM_API_KEY"],
    base_url=os.environ.get("LLM_BASE_URL", "https://api.deepseek.com"),
)
MODEL = os.environ.get("LLM_MODEL", "deepseek-flash")
NO_THINKING = {"thinking": {"type": "disabled"}}

docs = pathlib.Path("data/httpx-docs")
text = "\n\n".join(p.read_text() for p in sorted(docs.rglob("*.md")) if p.name != "LICENSE.md")
print(f"文档共 {len(text)} 个字符")

# 实验一：同样的请求连发两次
system = "你是 httpx 的答疑助手。下面是 httpx 的全部文档：\n\n" + text
for i in range(2):
    r = client.chat.completions.create(
        model=MODEL,
        messages=[
            {"role": "system", "content": system},
            {"role": "user", "content": "httpx 默认的超时时间是多少秒？一句话回答。"},
        ],
        extra_body=NO_THINKING,
    )
    u = r.usage
    print(f"第 {i + 1} 次：输入 {u.prompt_tokens}，其中缓存命中 {u.prompt_cache_hit_tokens}，"
          f"输出 {u.completion_tokens} | {r.choices[0].message.content}")

# 实验二：把一句无关的话插到文档的不同位置
needle = "（备注：RepoBot 项目的内部口令是“蓝鲸四十二”。）"
paragraphs = text.split("\n\n")
for fraction in [0, 0.25, 0.5, 0.75, 1.0]:
    k = int(len(paragraphs) * fraction)
    haystack = "\n\n".join(paragraphs[:k] + [needle] + paragraphs[k:])
    r = client.chat.completions.create(
        model=MODEL,
        messages=[{"role": "user", "content": haystack + "\n\n问题：上面的文字里提到的 RepoBot 内部口令是什么？只回答口令本身。"}],
        extra_body=NO_THINKING,
    )
    print(f"口令放在 {fraction:>4.0%} 处：输入 {r.usage.prompt_tokens} 词元 → {r.choices[0].message.content}")

Run in the course directory, this is what I got (your wording will differ; the token counts should be close):

文档共 116933 个字符
第 1 次：输入 29397，其中缓存命中 0，输出 12 | httpx 默认超时是 5 秒。
第 2 次：输入 29397，其中缓存命中 29184，输出 19 | httpx 默认的超时时间是 5 秒（指网络不活动的超时）。
口令放在   0% 处：输入 29404 词元 → 蓝鲸四十二
口令放在  25% 处：输入 29404 词元 → 蓝鲸四十二
口令放在  50% 处：输入 29404 词元 → 蓝鲸四十二
口令放在  75% 处：输入 29404 词元 → 蓝鲸四十二
口令放在 100% 处：输入 29404 词元 → 蓝鲸四十二

Start with experiment 1. The answer is right: the httpx docs say, word for word, "The default behavior is to raise a TimeoutException after 5 seconds of network inactivity". The whole documentation is about 29,000 tokens, just 3% of a 1-million-token window. The same question was asked twice in a row, and of the 29,397 input tokens the second time, 29,184 hit the cache. We'll see below how much that saves.

Can it find a sentence hidden in the middle?

Experiment 2 is a simple "needle in a haystack" test. A widely cited paper, Lost in the Middle (Liu et al., 2023), found that when the models of the day looked for information in long inputs, they did best with information at the beginning and end, and were most likely to miss it in the middle.

I inserted a sentence that has nothing to do with httpx at the start, at 25%, at 50%, at 75% and at the end of the text, and all five positions were answered correctly. At thirty thousand tokens, on a task as simple as finding one sentence, deepseek-flash showed no sign of "forgetting the middle". That paper tested 2023 models, and long-context ability has improved a lot since.

But don't conclude that long context is a solved problem. "Find one obvious sentence" is the easiest long-context task there is. When several pieces of information scattered across a document have to be combined, or subtle differences told apart among dozens of similar passages, models still go wrong more often; benchmarks such as RULER (Hsieh et al., 2024) test exactly those harder cases. It's best to test which kind your own task is with your own data; exercise 3 is such a test.

Counting the money: why not always put everything in

Finding it doesn't make it worthwhile. Here's a function that works out the cost from usage:

PRICES = {
    # 模型: (输入-缓存命中, 输入-缓存未命中, 输出)，美元 / 每一百万词元，高峰价
    "deepseek-flash": (0.006, 0.30, 1.20),
    "deepseek-v4-pro": (0.044, 1.32, 3.96),
}


def cost_usd(usage, model, off_peak=False):
    hit_price, miss_price, out_price = PRICES[model]
    # DeepSeek 的 usage 里有 prompt_cache_hit_tokens，别家不一定有，没有就当全部未命中
    hit = getattr(usage, "prompt_cache_hit_tokens", 0) or 0
    miss = usage.prompt_tokens - hit
    total = (hit * hit_price + miss * miss_price + usage.completion_tokens * out_price) / 1_000_000
    return total / 2 if off_peak else total  # 低谷时段五折

The prices are DeepSeek's official prices as of September 2026; check them against the pricing page before relying on them. Plug in the two sets of numbers from experiment 1, and estimate the monthly cost at 1,000 calls a day (the complete code is in code/01-llm-basics/cost.py):

第一次：0.008833 美元
第二次：0.000262 美元
每月 3 万次，全部命中缓存：7.85 美元；全部不命中：265.00 美元

The simple Q&A in lesson 3 of module 00 cost US$0.00019. The first call here, with the whole documentation put in, cost US$0.0088: 46 times as much, for an answer the user sees as a single sentence. At 1,000 calls a day for a month, without the cache it comes to US$265.

Then there's speed. The longer the input, the longer the model takes to read it, and the longer the user waits for the first character to appear. At thirty thousand tokens the difference isn't obvious yet; once the documentation reaches hundreds of thousands of tokens you'll clearly feel it.

So putting everything in suits these cases: the material isn't big, there aren't many calls, or the cache hits reliably. When the material is large, calls are frequent and each call needs only a small part of it, it pays to find the relevant parts first and give the model only those. That's what module 04's RAG does.

The cache: making a repeated beginning almost free

The second call was over 30 times cheaper (0.000262 against 0.008833) thanks to DeepSeek's context caching: if the beginning of this request is identical to the beginning of an earlier request, the server can reuse the results it already computed, and that part is billed at the "cache hit" price. For deepseek-flash a cache hit costs one-fiftieth of a miss (0.006 against 0.30).

According to DeepSeek's caching documentation (as of September 2026):

• Caching is on by default; no code changes are needed.
• The cache stores "prefix units". The server creates them at each request's boundary, at beginnings shared by several requests, and at fixed token intervals within long inputs. A part counts as a hit only if it matches a whole unit.
• A cache that is no longer used is usually cleared within a few hours to a few days.
• The hit and miss token counts are in usage as prompt_cache_hit_tokens and prompt_cache_miss_tokens.

In the experiment, 213 tokens still missed on the second call precisely because the cache matches by unit: the last stretch containing the user's question, and any part too short to make up a complete unit, are billed as misses.

These rules decide how you should order your messages:

• What doesn't change goes first: the system message, fixed material and examples go at the very beginning.
• What changes goes last: the user's question and anything that differs from call to call go at the end.

Put the user's question before the documentation, and since the question differs each time, the beginning differs, and none of the documentation after it can hit the cache.

Output costs more than input

Back to the price table: deepseek-flash output is US$1.20 per million tokens, cache-miss input is 0.30, and cache-hit input is 0.006. Output costs 4 times as much as input, and 200 times as much as cache-hit input.

A few things follow directly:

• Getting the model to cut the waffle saves money. Asking in the prompt for "a one-sentence answer" or "output JSON only" has an immediate effect.
• In thinking mode, the thinking is billed as output. Turning thinking off for simple tasks saves most of the output cost.
• Long material in the input isn't actually expensive if it hits the cache. What really costs money is usually long input that misses the cache, and long-winded output.

Common questions

An error says the context is too long: first check whether the conversation history keeps piling up. Lesson 1 of module 03 covers truncating and compressing history.

The cache hit count is always 0: check whether the beginning of each request really is exactly the same. Common culprits are putting the current time or a random ID in the system message, or reordering the material from call to call.

Do other providers have a cache? Most do, but pricing and rules differ, and some need you to mark which part should be cached. Check their docs before relying on it.

Exercises

1. Use cost_usd to work it out: with the same 30,000 calls a month and no cache hits, how much would running them all in off-peak hours (pass off_peak=True) save?
2. Swap the order of the system and user messages in experiment 1 (question first, documentation after), ask twice in a row, and see how the second call's cache hit count changes.
3. Change experiment 2: insert three sentences at different places, such as "the first part of the password is 蓝鲸 (blue whale)", "the second part is 四十 (forty)", "the third part is 二 (two)", then ask for the full password. Test several times each with thinking off and on. Are the results the same as when finding a single sentence?

Self-check

1. The context window is 1 million tokens. Does that mean I can input 1 million tokens?

No. The context window is the limit on input and output combined. If the input takes 990,000, at most 10,000 are left for output, and with thinking on the thinking counts as output too. Besides, fitting doesn't make it worthwhile: long input is slow and expensive.

2. Your application sends a fixed product manual together with the user's question every time. How should you order the messages to save the most money?

Put the product manual first (for example in the system message) and the user's question last. Then every request begins the same way and can hit the cache, and the manual is billed at the cache-hit price, dozens of times cheaper. With the question first, the beginning differs every time and the cache never hits.

3. In a test, the model found the sentence you hid in a long document. Does that show its long-context ability is fine?

No. "Find one sentence" is the easiest long-context task. Models go wrong more often when they have to combine information from several places or tell details apart in a lot of similar content. To know whether it meets your needs, test with something close to your real task.