Giving examples: few-shot prompting

Some requirements are hard to put into words. Where's the line between a "bug" and a "usage question"? A user asks "cookie 在重定向之后丢了，是我用法不对吗" (my cookie got lost after a redirect; am I using it wrong?); which category is that? You could write a long definition, but a few examples are often quicker.

Showing the model a few "input → output" examples for it to follow is called few-shot prompting. Doing the task directly without examples is called zero-shot.

Experiment: classifying user messages

The maintainers of httpx get all kinds of messages every day. We want the model to sort them automatically into four categories: bug, feature request, usage question and other. I prepared 20 messages, each labelled with its correct category by hand, for example:

TESTS = [
    ("用 AsyncClient 并发 100 个请求，程序直接卡死，CPU 占满", "缺陷"),
    ("能不能加一个像 requests 那样的 Session 重试适配器？", "功能建议"),
    ("怎么给单个请求设置不同的超时时间？", "使用问题"),
    ("你们的文档网站打不开了", "其他"),
    # ……一共 20 条，每类 5 条，完整列表见 code/02-prompting/few_shot.py
]

The zero-shot prompt only explains the classification rules:

ZERO_SHOT = """把用户留言分成以下四类之一：缺陷、功能建议、使用问题、其他。
只输出类别名称。"""

The few-shot prompt adds 4 examples after that, one per category, none of which are in the test set:

FEW_SHOT = ZERO_SHOT + """

例子：
留言：调用 client.close() 之后再发请求没有报错，而是静默返回了旧的响应
类别：缺陷

留言：想要一个参数，能在请求失败时自动打印完整的请求和响应
类别：功能建议

留言：base_url 和 url 拼接的规则是什么？结尾的斜杠有没有影响
类别：使用问题

留言：这个项目和 aiohttp 比哪个更好
类别：其他"""

Each message is sent to the model in the format 留言：{text}\n类别： (message: …, category:), with temperature 0 and thinking off, and compared with the hand labels:

def classify(system, text):
    response = client.chat.completions.create(
        model=MODEL,
        messages=[{"role": "system", "content": system}, {"role": "user", "content": f"留言：{text}\n类别："}],
        temperature=0,
        extra_body={"thinking": {"type": "disabled"}},
    )
    return response.choices[0].message.content.strip()

What I got:

零样本：答对 18/20，格式不对 0 条 []
    怎么给单个请求设置不同的超时时间？  标注=使用问题  模型=功能建议
    你们的文档网站打不开了  标注=其他  模型=缺陷
少样本：答对 20/20，格式不对 0 条 []

Where zero-shot went wrong

Look at the two that zero-shot got wrong.

"怎么给单个请求设置不同的超时时间？" (how do I set a different timeout for a single request?) was classified as a feature request. The model probably read it as "the user wants a feature for setting a timeout on a single request". But httpx has long supported this; the user just doesn't know how to write it.

"你们的文档网站打不开了" (your documentation site won't load) was classified as a bug. Taken literally, a site that won't load is indeed a bug, but by "bug" we mean a problem in the httpx library itself, and the documentation site belongs under "other".

Neither mistake is the model being stupid; the classification criteria have grey areas, and my rules didn't spell them out. Once it had examples, the model saw from "base_url 和 url 拼接的规则是什么" (what are the rules for joining base_url and url?) that "asking how to use something = usage question", and from "这个项目和 aiohttp 比哪个更好" (which is better, this project or aiohttp?) that "not about the library itself = other". The examples made the rules clear for me.

Of course, you could skip the examples and write the rules in more detail instead: "questions asking 'how to do' something are usage questions, even if they sound like a request for a new feature"; "only problems with the behaviour of the httpx library itself count as bugs". That works too, and uses fewer tokens. In practice the two often go together: rules for the main line, examples for the details.

How to choose examples

Cover every possible output. One example for each of the four categories. If you only give examples of "bug" and "usage question", the model becomes reluctant to output the other two.

Choose examples on the boundary. The most valuable examples aren't the most typical ones but the easiest to get wrong. "调用 close() 之后再发请求没有报错" (sending a request after calling close() doesn't raise an error) is a good one: it sounds like a question about usage, but it's actually a problem with the library's behaviour.

Keep them separate from the test set. Examples must not appear in the test set, or you've told the model the answers and the accuracy will be inflated. The 4 examples in this experiment were written separately.

Match the format of real input. The examples use the "留言：……类别：……" format, and real requests use the same format. The model imitates the examples' format closely, so the examples' format is the output format you'll get. Neither zero-shot nor few-shot produced a format error in this experiment, but when you ask for more complex output (such as JSON), one correctly formatted example greatly reduces format problems.

How many

Usually 3–5 are enough. The more examples, the longer every request's input, and the more it costs. Fortunately examples are fixed, so in the system message they hit the cache.

If results still aren't good enough, first ask whether the examples are poorly chosen before rushing to add more. Three examples covering different boundary cases beat 10 similar ones.

Side effects of examples

Examples are powerful, which is why they easily lead the model astray.

The model imitates everything about the examples. If the example answers are all short, the model's answers get short too; if the examples are all in Chinese, it may keep using the Chinese category names for English messages (which here is what we want). It also imitates features you never meant it to copy.

The model may copy the examples' content. In generation tasks (say, writing product descriptions), if every example is about coffee, words about coffee may turn up in a description of tea. So examples should be varied, and differ from each other in content.

Order and proportions matter. If 4 of 5 examples are "bug", the model will lean towards classifying more messages as "bug". Keep the number of examples per category balanced.

Common questions

Should examples go in the system message, or be made into a multi-turn conversation? Either works. The approach above writes the examples into the system message. The other way is to turn each example into a pair of user and assistant messages placed before the real question, as if the model had already answered like this a few times. The latter is especially effective for imitating a format, but the message list gets longer and harder to maintain. I usually start with the system-message approach and switch if the format goes wrong.

What if there are many categories, say 30, and one example each is too long? First consider splitting the categories into two levels: broad category first, then subcategory. Or give examples only for the categories that get confused. Going further: use the embeddings from lesson 5 of module 01 to find the few examples most similar to the current input in a large pool of examples, and put only those into the prompt. This is called dynamic few-shot, and it works on the same principle as module 04's RAG.

What this experiment doesn't show

Twenty test messages are too few. With another batch of data, the gap between 18 and 20 could become 19 and 20, or even reverse. What this experiment shows is one concrete example of "few-shot helps on this kind of task", not that "few-shot raises accuracy by 10%". Comparing two prompts reliably needs more test data, which is the subject of lesson 5.

Exercises

1. Cut the examples in the few-shot prompt down to 2 (keep only "bug" and "usage question"), run it again, and see how accuracy changes for "other" and "feature request".
2. Deliberately use 4 examples that are all "bug", and see whether the model leans towards classifying things as "bug".
3. Write 10 more messages yourself, ideally ones that you'd have to think about to classify correctly, add them to the test set and rerun. Does the gap between zero-shot and few-shot grow or shrink?
4. Without examples, add two notes to the zero-shot rules ("anything asking how to use something counts as a usage question, even if it sounds like a request for a new feature" and "anything about the docs or the community counts as other") and see whether it matches few-shot.

Self-check

1. Why must the examples in a few-shot prompt not appear in the test set?

The examples tell the model the correct answers for those inputs. If the test set contains the same inputs, the model can get them right just by copying, and the measured accuracy will be inflated and won't reflect how it really does on new data.

2. You give the model 5 examples, 4 of which are "bug". What might go wrong?

The model will tend to classify more messages as "bug", because the examples' proportions suggest that "most messages are bugs". Keep the number of examples per category as balanced as possible, with every possible output covered at least once.

3. When choosing examples, should you pick the most typical ones or the easiest to get wrong?

Prefer the ones that are easiest to get wrong, on the boundary. The model already gets typical ones right; boundary examples are what help it work out the rules in the grey areas.