Prompts need tests too

The most common way to change a prompt goes like this: you notice a bad answer, change a sentence in the prompt, try that one question again, and it's fixed. Done.

The trouble is you've only checked that one question. The change may have fixed it while breaking three others that used to work, and you won't find out until users complain. It's the same as changing code without running the tests.

This lesson builds a very small testing tool: prompts in files, test cases in a file, and one command that runs every case and tells you the pass rate, which cases failed and which are right only some of the time.

Take the prompts out of the code

First, save your prompts as separate text files instead of hard-coding them in Python:

code/02-prompting/
  prompts/
    classify_v1.txt     第一版提示词
    classify_v2.txt     第二版提示词
    cases.jsonl         测试用例
  prompt_test.py        测试脚本

The benefits: two versions can be compared side by side; git shows what changed each time; and colleagues who don't write code can edit prompts too.

classify_v1.txt is lesson 2's zero-shot prompt:

把用户留言分成以下四类之一：缺陷、功能建议、使用问题、其他。
只输出类别名称。

classify_v2.txt is the improved version. Based on the two messages zero-shot got wrong in lesson 2, it gives each category a definition, spells out the boundaries that get confused, and adds lesson 2's 4 examples:

把 httpx 项目收到的用户留言分成以下四类之一，只输出类别名称。

- 缺陷：httpx 库本身的行为不符合文档或者预期，比如报错、崩溃、结果不对。
- 功能建议：希望 httpx 增加目前没有的功能。
- 使用问题：问某个功能怎么用、某个行为是不是正常。哪怕看起来像在要新功能，只要 httpx 已经能做到，就算使用问题。拿不准是自己用错了还是库有问题的，也算使用问题。
- 其他：和 httpx 库本身无关的，比如文档网站、社区、招聘、感谢、和别的库比较。

例子：
（和第 2 课相同的 4 个例子）

Test cases

cases.jsonl has one case per line, with the message and its correct category. On top of lesson 2's 20, I added 10 harder ones, all of which I had to think about myself when classifying:

{"text": "httpx 支持 HTTP/3 吗？", "label": "使用问题"}
{"text": "文档里 Limits 那一节的示例代码跑不通，max_keepalive 这个参数名好像不对", "label": "其他"}
{"text": "response.elapsed 在流式请求里读出来一直是 0，这正常吗", "label": "使用问题"}
{"text": "同样的代码，requests 返回 200，httpx 返回 403", "label": "使用问题"}
{"text": "follow_redirects=True 时，301 跳转后 POST 变成了 GET", "label": "使用问题"}
……

Good test cases come from several places: real user input (the most important); every mistake you've fixed, added once it's fixed so it can't quietly break again; and whatever boundary cases you can think of.

The test script

import json
import os
import sys
from concurrent.futures import ThreadPoolExecutor
from pathlib import Path

from openai import OpenAI

client = OpenAI(
    api_key=os.environ["LLM_API_KEY"],
    base_url=os.environ.get("LLM_BASE_URL", "https://api.deepseek.com"),
)
MODEL = os.environ.get("LLM_MODEL", "deepseek-flash")
RUNS = 3
HERE = Path(__file__).parent

cases = [json.loads(line) for line in (HERE / "prompts/cases.jsonl").read_text().splitlines() if line.strip()]


def classify(system, text):
    response = client.chat.completions.create(
        model=MODEL,
        messages=[{"role": "system", "content": system}, {"role": "user", "content": f"留言：{text}\n类别："}],
        extra_body={"thinking": {"type": "disabled"}},
    )
    return response.choices[0].message.content.strip()


records = []
for prompt_path in sys.argv[1:]:
    system = (HERE / prompt_path).read_text()
    jobs = [case for case in cases for _ in range(RUNS)]
    with ThreadPoolExecutor(10) as pool:
        outputs = list(pool.map(lambda c: classify(system, c["text"]), jobs))

    passed = sum(out == case["label"] for out, case in zip(outputs, jobs))
    print(f"{prompt_path}：{passed}/{len(jobs)} 通过（{passed / len(jobs):.0%}）")
    for i, case in enumerate(cases):
        answers = outputs[i * RUNS:(i + 1) * RUNS]
        right = sum(a == case["label"] for a in answers)
        records.append({"prompt": prompt_path, "text": case["text"], "label": case["label"], "outputs": answers})
        if right == 0:
            print(f"    全错  {case['text']}  标注={case['label']}  模型={answers}")
        elif right < RUNS:
            print(f"    不稳  {case['text']}  标注={case['label']}  模型={answers}")

with open(HERE / "results.jsonl", "w") as f:
    for r in records:
        f.write(json.dumps(r, ensure_ascii=False) + "\n")

Two design decisions are worth explaining.

Each case runs 3 times. This time I didn't set the temperature to 0; I used the default, just like real use in production. As lesson 3 of module 01 showed, the same input can give different results each time. Run it once and you can't tell "reliably right" from "right by luck". Run it 3 times and cases fall into three groups: always right, always wrong, and sometimes right.

Results are saved. The raw output of every run is written to results.jsonl. When you change the prompt later, you can compare old and new results case by case and see exactly which cases got better and which got worse.

Run it:

python prompt_test.py prompts/classify_v1.txt prompts/classify_v2.txt

Results

prompts/classify_v1.txt：71/90 通过（79%）
    全错  怎么给单个请求设置不同的超时时间？  标注=使用问题  模型=['功能建议', '功能建议', '功能建议']
    全错  你们的文档网站打不开了  标注=其他  模型=['缺陷', '缺陷', '缺陷']
    全错  文档里 Limits 那一节的示例代码跑不通，max_keepalive 这个参数名好像不对  标注=其他  模型=['缺陷', '缺陷', '缺陷']
    全错  response.elapsed 在流式请求里读出来一直是 0，这正常吗  标注=使用问题  模型=['缺陷', '缺陷', '缺陷']
    不稳  同样的代码，requests 返回 200，httpx 返回 403  标注=使用问题  模型=['使用问题', '使用问题', '其他']
    全错  follow_redirects=True 时，301 跳转后 POST 变成了 GET  标注=使用问题  模型=['缺陷', '缺陷', '缺陷']
    全错  能不能出一个视频教程  标注=其他  模型=['功能建议', '功能建议', '功能建议']
prompts/classify_v2.txt：81/90 通过（90%）
    不稳  httpx 支持 HTTP/3 吗？  标注=使用问题  模型=['功能建议', '功能建议', '使用问题']
    不稳  文档里 Limits 那一节的示例代码跑不通，max_keepalive 这个参数名好像不对  标注=其他  模型=['其他', '缺陷', '其他']
    全错  同样的代码，requests 返回 200，httpx 返回 403  标注=使用问题  模型=['缺陷', '缺陷', '缺陷']
    全错  follow_redirects=True 时，301 跳转后 POST 变成了 GET  标注=使用问题  模型=['缺陷', '缺陷', '缺陷']

The overall score rose from 79% to 90%. But looking only at the overall score misses a lot, so let's go through it case by case.

Reading the results: what got fixed, what got broken

Fixed. The cases v1 always got wrong, "怎么给单个请求设置不同的超时时间" (how do I set a different timeout for a single request), "你们的文档网站打不开了" (your documentation site won't load), "能不能出一个视频教程" (could you make a video tutorial) and "response.elapsed……这正常吗" (response.elapsed… is this normal?), are all right in v2. v2's definitions say specifically "even if it sounds like a request for a new feature, it's a usage question as long as httpx can already do it", and "the documentation site, the community… count as other", which match these cases exactly.

Broken. "同样的代码，requests 返回 200，httpx 返回 403" (the same code returns 200 with requests and 403 with httpx) was right 2 times out of 3 in v1, and wrong all 3 times in v2, classified as a "bug" every time. This is exactly what the overall score hides: the total went up, but a case that used to be mostly fine got worse.

Newly unstable. "httpx 支持 HTTP/3 吗？" (does httpx support HTTP/3?) was classified as a "feature request" 2 times out of 3 in v2.

Always wrong. "301 跳转后 POST 变成了 GET" (a POST becomes a GET after a 301 redirect) was wrong every time in both versions; the model insists it's a bug.

Suspect the label before the model

Faced with a failing case, the first thing to do isn't to change the prompt, but to check whether the label itself is right.

For "301 跳转后 POST 变成了 GET" I labelled it a "usage question", on the grounds that this is httpx's normal behaviour. But I had to make sure. In the httpx source, httpx/_client.py, _redirect_method says:

# If a POST is responded to with a 301, turn it into a GET.
# This bizarre behaviour is explained in 'requests' issue 1704.
if response.status_code == codes.MOVED_PERMANENTLY and method == "POST":
    method = "GET"

It's a deliberate design choice that follows browsers and requests, so the label is right, and the model just doesn't know this detail. A mistake like this is hard to fix by changing the prompt, because the problem lies in the model's knowledge. You can accept it, or leave questions about "whether some behaviour is normal" to a system that can look up the documentation (that's module 04's RAG).

"requests 返回 200，httpx 返回 403" is different. I labelled it a "usage question" because this is usually caused by differences in request headers (such as a different default User-Agent), and adjusting how you use the library fixes it. But on reflection, nothing in the message itself shows the cause, and treating it as "the library not behaving as expected" is also reasonable. The label on this case is itself debatable. The model calling it a "bug" all 3 times isn't necessarily the model being wrong.

With a case like this you have three choices: change the label; rewrite the message to be clearer; or accept that it's ambiguous and remove it from the test set, or allow both answers. Don't keep tweaking the prompt to make the model "get it right" on a debatable case; that's just fitting to an arbitrary decision of your own.

The rhythm of iterating

A workable rhythm:

1. Run the tests and note the overall score and each case's result.
2. Pick one kind of mistake (not one case) and work out the cause. Check the labels first.
3. Change the prompt, aimed at that kind of mistake only.
4. Run the tests again and compare case by case with the last run: how many got fixed, how many got broken.
5. If more got broken than fixed, go back.

Changing one thing at a time is how you learn what each change does. Change five things at once and when the score moves you won't know which one did it.

The limits of this tool

This is a lightweight version, suited to tasks with a right answer, such as classification and extraction. It has some obvious shortcomings:

• 30 cases are still too few. The gap between 90% and 79% is fairly credible, but 90% against 88% could be random variation.
• It can only judge exact matches. When the answer is a piece of writing (a customer-service reply, a summary), == can't judge right and wrong.
• It doesn't record cost or time.

Module 06 expands it into a complete evaluation system: a larger evaluation set, a model grading open-ended answers, and a log of every call and its cost.

Exercises

1. Run prompt_test.py and see how your results differ from mine. Run it twice more; is the overall score the same each time?
2. For the debatable case "requests 返回 200，httpx 返回 403", make your decision (change the label, rewrite the message, or remove it) and explain why.
3. Write a classify_v3.txt that tries to fix the instability on "httpx 支持 HTTP/3 吗" without making other cases worse. Compare v2 and v3 case by case using results.jsonl.
4. Add a feature to prompt_test.py: print each version's total cost (using cost_usd from lesson 4 of module 01). v2's prompt is much longer; how much more does it cost?

Self-check

1. Why run each test case 3 times instead of once?

Model output is random, and the same input can give different results each time. Run it once and you can't tell "reliably right" from "right by luck". Running several times reveals unstable cases that are sometimes right and sometimes wrong, which often point to grey areas the prompt doesn't explain.

2. The new prompt scores higher overall than the old one. Can you just switch to it?

Look case by case first. A higher total can still hide cases that used to be fine and got worse, like this lesson's "requests 返回 200，httpx 返回 403". Check whether the cases that got worse are important scenarios, and whether the fault is the model's or the label's, before deciding to switch.

3. A case is always wrong under both versions of the prompt. What should you do?

First check whether the label itself is right, looking at the documentation or source code if necessary. If the label is debatable, fix it or remove the case. If the label really is correct and the model lacks the relevant knowledge, changing the prompt often won't fix it; you can accept the mistake, or give the model the material it needs with a method like RAG.