Cutting cost and latency

Once an AI application is live and called thousands of times a day, a tiny saving on each call adds up to a lot, and making each call a little faster makes a big difference to how users feel.

This lesson runs five small experiments. Each is a technique you can use in a project straight away, and each comes with measured numbers. All experiments use deepseek-flash, priced at September 2026 peak rates. Full code in code/06-production/cost_latency.py.

1. Put fixed material first

Module 01, Lesson 4 covered DeepSeek's context caching: when the start of a request matches an earlier request, that part is billed at the cache-hit price, one fiftieth of the miss price.

The experiment: three different questions, each sent with the same httpx document (about 15,000 tokens). One version puts the question first and the document after; the other puts the document in the system message and the question after.

for label, build in [
    ("问题在前", lambda q: [{"role": "user", "content": f"问题：{q}\n\n参考文档：\n{DOCS}\n\n一句话回答。"}]),
    ("文档在前", lambda q: [{"role": "system", "content": f"参考文档：\n{DOCS}"}, {"role": "user", "content": q + "一句话回答。"}]),
]:

== 1. 缓存：固定的资料放在前面，还是问题放在前面
  问题在前：三次的缓存命中 ['0/15168', '0/15169', '0/15167']，共 0.01379 美元
  文档在前：三次的缓存命中 ['0/15167', '14976/15168', '14976/15166']，共 0.00500 美元

With the question first, none of the three calls hit the cache: each question differs, so each request starts differently, and the identical document after it doesn't help. With the document first, the first call had no cache, and the next two each hit 14,976 tokens. Total cost fell from $0.01379 to $0.00500, a 64% saving.

Only three questions were asked here, so the first "cold start" is most of the cost. The more questions you ask, the closer the document-first version gets to "paying only for the question itself".

That's just a change of order, with not one extra line of code. Check your prompts: does the system message contain anything that changes every time, like the current time, a user name or a random id? Move it to the end.

2. Have the model say less

Output costs 4 times as much as input, and generation speed directly determines how long users wait.

q = "httpx 和 requests 有什么区别？"
for label, prompt in [("不做要求", q), ("要求简短", q + "用三句话以内回答。")]:

== 2. 输出长度：不做要求 vs 要求简短
  不做要求：输出 650 词元，3.3 秒，0.00078 美元
  要求简短：输出 52 词元，0.8 秒，0.00007 美元

Adding just one line, "answer in three sentences or fewer", cut output from 650 tokens to 52, cost to a tenth, and wait time from 3.3 seconds to 0.8.

By default models lean towards thoroughness, which is good in some situations and waste in others. Work out how much information your users really need and say so in the prompt. You can also set max_tokens as a hard limit to stop the occasional runaway answer, but remember it cuts the answer off abruptly (Module 00, Lesson 3), so rely mainly on the prompt.

3. No thinking for simple questions

== 3. 简单问题开不开思考
  不思考：输出 26 词元，0.7 秒，0.00004 美元 | '用 `params` 参数传字典，如 `httpx.get(url, param'
  思考：输出 146 词元，1.9 秒，0.00019 美元 | '用 `params` 参数传字典或元组列表，如 `httpx.get(url, '

For a question like "how do I pass query parameters", the two modes give almost the same answer, but thinking costs nearly 5 times as much and adds 1.2 seconds. As Module 02, Lesson 3 explained, thinking suits questions that need multi-step reasoning. An application's questions range from easy to hard, and you can handle them by type: turn thinking off for simple lookups and on only for complex analysis. Deciding which is which can be done by the classifier from the next lesson at the same time.

4. Same question, return last time's answer

In many applications, users ask the same questions again and again. Rather than calling the model every time, store the answers:

cache = {}


def cached_ask(question):
    key = hashlib.sha256(" ".join(question.lower().split()).encode()).hexdigest()  # 忽略大小写和多余空格
    if key in cache:
        return cache[key], 0.0
    text, u, _ = ask([{"role": "user", "content": question}])
    cache[key] = text
    return text, cost(u)

Asked 4 times, phrased slightly differently: "httpx 怎么设置代理？" ("how do I set a proxy in httpx?"), the same sentence again, "HTTPX 怎么设置代理？" (upper case, with an extra space), and "httpx 怎么设置代理" (no question mark).

== 4. 自己做结果缓存：同样的问题直接返回上次的答案
  问了 4 次（写法略有不同），实际调用 2 次模型，共 0.00239 美元

The first three were recognised as the same question, with just one model call. The last was treated as new because the question mark was missing. The normalisation isn't thorough enough: punctuation should be removed too. Exercise 2 has you improve it.

Going further, you can use the embeddings from Module 01, Lesson 5 for a "semantic cache": questions with similar meaning ("how to configure a proxy" and "how to set up a proxy server") hit the same cache entry. But be careful: similar meaning doesn't mean the same answer; "how to set a timeout in httpx" and "how to set a timeout in requests" have very close vectors.

When you can't use a result cache:

• The answer changes. Answers that depend on live data or on the user's personal information can't be shared across users or over time.
• Multi-turn conversation. For a question like "what about async?", the answer depends on what came before, so the sentence alone can't be cached.
• Variety is wanted. For tasks like writing copy or coming up with names, users want different results anyway.

Caches need an expiry time. When the docs are updated, old answers may be out of date.

5. Concurrency

== 5. 串行 vs 并发
  10 个请求：一个一个来 9.5 秒，5 个并发 2.3 秒

Ten independent requests sent one after another take 9.5 seconds; sent 5 at a time, only 2.3 seconds. Most of a model call's time is spent waiting on the server, and during that time your program does nothing, so it may as well send other requests.

Module 03, Lesson 4 covered controlling concurrency with a thread pool and a semaphore. More concurrency isn't always better: providers have concurrency limits and return 429 if you exceed them, and with too much concurrency your own machine and network may not cope.

Other approaches

• Switch to a smaller, cheaper model. Module 01, Lesson 6's method: compare on your evaluation set, and use the cheap model if it meets the bar. You can also route by difficulty: easy questions to a small model, hard ones to a large model.
• Avoid peak hours. DeepSeek's off-peak price is half the peak price (as of September 2026, peak is 9:00–12:00 and 14:00–18:00 Beijing time on weekdays). Non-urgent batch jobs, such as running evaluations overnight or working through a data backlog, can go in off-peak hours.
• Cut unnecessary context. Should RAG take 5 chunks or 3? Keep 20 messages of history or 10? Can the agent's tool results be shorter still? Verify each with the evaluation set and change it only if quality doesn't drop.
• Streaming output. It doesn't save money, but it makes things feel much faster to users (Module 03, Lesson 2).

Measure first, then change

Every optimisation in this lesson should first be measured in your own application: where is the money going now, and where is the time going? Last lesson's logs answer exactly that. Sort by cost, find the most expensive kind of request, and start optimising there. Afterwards, confirm with the evaluation set that quality didn't get worse. Saving money while getting answers wrong isn't a good deal.

Exercises

1. Improve cached_ask's normalisation: remove all punctuation and whitespace, and unify full-width and half-width characters. Run the experiment again. Do the 4 questions now make only one model call?
2. Using the logs from Lesson 3 (traces.jsonl), work out what share of input tokens hit the cache when the agent answers a question. Think about how reordering the messages could raise the hit rate.
3. In experiment 2, replace "answer in three sentences or fewer" with max_tokens=60, leaving the prompt unchanged. What does the answer look like? Which approach is better?

Self-check

1. With the same document and question, why is "document first" so much cheaper than "question first"?

DeepSeek's cache only recognises the part at the start of a request that matches. With the question first, each question differs, so the start differs, and the document after it can't hit the cache, so every call pays full price. With the document first, the start is identical every time, and after the first call it hits the cache, billed at one fiftieth of the price.

2. When can't you use a result cache?

When the answer changes over time (depends on live data, or the docs get updated), when the answer depends on the user's personal information, when the question depends on conversation context (such as "what about async?"), and for tasks where users want a different result each time (writing copy, naming things). Caches also need an expiry time so they don't return stale answers.

3. Why do concurrent requests cut total time so much? Is more concurrency always better?

Most of a model call's time is spent waiting for the server, and the program is idle while it waits; sending several requests at once overlaps those waits. More isn't always better: exceeding the provider's concurrency limit gets you rate-limited (429), and your own machine and network have limits too, so control concurrency with something like a semaphore.