Planning and self-checking

Almost every article about agents covers two techniques, planning and reflection: have the model make a plan before acting, then have it check its own result afterwards and fix any problems it finds. It sounds sensible; it's how people handle complicated work too.

But both techniques need extra model calls. How much do they actually improve things, and are they worth the money? This lesson measures them on a real multi-step task.

The task

"Compare whether httpx's Client and AsyncClient are configured the same way for three things: timeouts, proxies and HTTP/2. Put it in a table, and cite the documentation source (file name and line numbers) in every cell."

This task is more complex than the earlier questions: three different topics, each to be checked for both clients, with sources recorded accurately. It uses the same agent and three doc tools from Lesson 2.

Approach 1: just do it

Hand the task straight to the agent and let it decide how to look things up.

Approach 2: plan first

First call the model once to list a plan, with no tools allowed. Then attach the plan to the task and hand it to the agent to carry out:

def with_plan():
    # 第一步：只列计划，不调用工具
    plan, usage = model([{"role": "system", "content": SYSTEM}, {"role": "user", "content":
                         TASK + "\n\n先不要调用工具。列出你打算怎么查，编号列出每一步要找什么，不超过 6 步。"}], None)
    answer, messages, stats = loop([
        {"role": "system", "content": SYSTEM},
        {"role": "user", "content": TASK + "\n\n按这个计划执行，执行中发现计划不对可以调整：\n" + plan.content},
    ])
    ……

The line "you may adjust the plan if you find it's wrong while carrying it out" matters. The plan was made before looking anything up; following it rigidly means missing leads that only turn up once you start looking.

Approach 3: check afterwards

Do it directly first; once there's an answer, give the model the answer together with all the original text the agent retrieved along the way, and have it check every cell:

def with_reflection():
    answer, messages, stats = plain()
    evidence = "\n\n".join(m["content"] for m in messages if m["role"] == "tool")
    review, usage = model([{"role": "user", "content": f"""下面是一份回答和查到的全部原文。逐格检查回答里的表格：
每一格的说法，原文里有没有依据？注明的出处（文件和行号）对不对？
只列出有问题的格子和原因。全部没问题就只回复"没有问题"。

回答：
{answer}

原文：
{evidence[:20000]}"""}], None)
    if "没有问题" in review.content[:20]:
        return answer, messages, stats
    # 有问题就把检查意见交回给智能体，让它继续查、修改回答
    messages += [{"role": "assistant", "content": answer},
                 {"role": "user", "content": "有人检查了你的回答，意见如下。需要的话继续查文档，然后给出修改后的完整回答。\n\n" + review.content}]
    revised, messages, more = loop(messages)
    ……

The check is based on "the text retrieved", that is, what the tools returned, not the model's own memory. As Module 04, Lesson 6 showed, a judge working from memory gets things wrong, so it has to check against the material. If there are problems, the review comments go back to the agent, which can keep searching the docs and then revise its answer.

Full code in code/05-agents/planning_reflection.py.

Results

Each approach run once (the answers are long, so only the statistics and key parts are shown here):

===== 直接做
  4 次模型调用，9 次工具调用，13155 词元，7 秒
===== 先列计划
  5 次模型调用，13 次工具调用，26086 词元，12 秒
===== 做完再检查
  7 次模型调用，14 次工具调用，50132 词元，17 秒

Just doing it already gave a good answer. For the HTTP/2 row it found the original text on line 50 of http2.md, "HTTP/2 support is available on both Client and AsyncClient", and concluded clearly that they're "the same". For the timeout and proxy rows it said honestly that the examples in the docs are all written with httpx.Client and none with AsyncClient, so it could only infer that the two match, with "limited evidence". That's a well-judged answer: it didn't pass off an inference as fact.

One small problem: the first sentence of its answer was in English, "Based on the documentation, here is the comparison table". After reading lots of English docs, an agent's answer language sometimes drifts. Saying "answer in Chinese" in the system prompt avoids this.

Planning first made 1 more model call; tool calls went from 9 to 13, tokens doubled, and it took 5 seconds longer. Its plan:

1. 找 `Client` 的文档，定位它关于超时（timeout）的配置参数与说明。
2. 找 `AsyncClient` 的文档，定位它关于超时（timeout）的配置参数与说明，比对是否一致。
3. 分别在 `Client` 和 `AsyncClient` 文档中找代理（proxy/proxies）相关配置，比对。
4. 分别在 `Client` 和 `AsyncClient` 文档中找 HTTP/2 相关配置（如 http2 参数），比对。
5. 检查两者的基类/继承关系或 API 参考页，确认是否共用同一套初始化参数……

Step 5 of the plan brought something new: it looked at api.md, found the two classes' member lists are almost identical except for close versus aclose, and used that to support "both share the same configuration parameters". But its final conclusion was the same as just doing it: HTTP/2 has clear evidence, timeouts and proxies can only be inferred. Twice the cost, with no real change in the conclusion.

Checking afterwards found real problems. The review pointed out two citation errors; the agent searched further and stated the corrections in its revised answer:

### 本次更正的两处（评审意见成立）

1. 原标注 `advanced/timeouts.md:66-68`（"`httpx.Timeout` 细调"）有误：`:66-68` 只是 `httpx.Timeout(10.0, connect=60.0)` 与 `httpx.Client(timeout=timeout)` 的示例代码。`httpx.Timeout` 的细调说明实际在 `advanced/timeouts.md:41-68`（标题 `## Fine tuning the configuration` 在 `:41`）。……
2. 原标注 `advanced/transports.md:223` 不能作为 `timeout` 构造参数的出处：该行是 `httpx.HTTPTransport(proxy=proxy, **kwargs)`，只涉及 transport 的 `proxy`/`**kwargs`，与客户端 `timeout` 无关。已删除该引用。

I checked against timeouts.md: line 41 really is the heading "## Fine tuning the configuration", and lines 66 to 68 really are just example code. The first correction is right. It also removed an irrelevant citation from the first answer. This kind of "source not cited precisely" problem is hard for a reader to spot, and it's exactly what checking is best at catching.

The cost: 7 model calls and 50,000 tokens, nearly 4 times as much as just doing it. The check step has to include all the retrieved text, which is the main reason it's expensive.

But the revised answer also changed in a noteworthy way: the conclusion went from "HTTP/2 is the same, timeouts and proxies can only be inferred" to a prominent heading saying "configured exactly the same way for all three". The body still ends by saying that timeouts and proxies are inferred, but the heading's tone is stronger than the evidence. The check fixed the citations but made the wording of the conclusion more certain. A revised answer needs another look too.

When it's worth it

One experiment can't settle it, but combined with this result, my experience is:

Planning first suits tasks with many steps where it's easy to miss a part, such as "change all 10 of these files". A plan helps the model remember what's left to do. On tasks like this lesson's, finished in three to five steps, it adds little. Another use is showing people: present the plan to the user first and confirm the direction before carrying it out, which is much cheaper than finding out afterwards that the direction was wrong.

Checking afterwards suits tasks whose results easily contain small errors, where errors are costly: citations, numbers, code. Always give the checker the original material to compare against, not its memory. It's expensive, so it's usually done once on the final result, not at every step.

Just doing it is a reasonable default for most tasks. Start there, build an evaluation, see where the errors mainly come from, and then add planning or checking where it's needed.

Exercises

1. Add "answer in Chinese" to SYSTEM in planning_reflection.py and rerun. Is the first sentence of the "just do it" answer still in English?
2. Change the check step's prompt to give only the answer and no original text ("check this answer for errors") and rerun. How does the quality of the review change?
3. Run each approach 3 times, recording the token count and what was revised each time. How far does a single run differ from the average of several?

Self-check

1. When having the model "plan first, then act", why tell it "you may adjust the plan if it turns out to be wrong"?

The plan is made before any material has been looked up, so its steps are guesses. Information found while carrying it out may show the original plan was flawed, and following it rigidly would lead in the wrong direction.

2. When having the model check its own answer, why give it the retrieved text as well?

Given only the answer, the model can judge right and wrong only from its own memory, which may be the very source of the error; the judge in the previous module made exactly that mistake. With the original text, it can compare point by point and find specific problems like wrong citations and unsupported claims.

3. In this lesson's experiment the self-check found real errors. Should every task get a self-check, then?

Not necessarily. Here the self-check cost nearly 4 times the money and more than twice the time. It suits tasks where errors are costly and small mistakes are likely, and is usually done once on the final result. Also, the revised answer can introduce new problems, as in this lesson, where the wording of the conclusion became more certain than the evidence.