A dependable AI coding workflow

The most common failure when writing code with AI isn't that it can't write the code. It's that it writes it, says "done", you believe it, and only after release do you find the problem.

The cause is usually at the two ends: at the start, nobody said clearly what "done" looks like; at the end, nobody objectively checked whether it really was done. The middle part, where the AI writes code, is actually the least likely to go wrong.

This lesson covers a simple workflow that fills in both ends. It isn't tied to any tool; it works the same with completion, chat or agents.

Write "what done looks like" first

Before letting the AI start, answer one question: once it's changed, how will I know it's right?

The best answer is a set of tests. Tests turn "done" into something you can run that gives a clear result: all passing means done, one failing means not done. No need to take the AI's word for it, and no need to go by your gut.

Let's demonstrate with a small task. When the httpx Q&A assistant calls an API it may receive a 429 (rate limited), and the Retry-After response header says how long to wait before retrying. It comes in two forms: a number of seconds (120), or an HTTP date (Wed, 21 Oct 2026 07:28:00 GMT). We need a function that parses it into "how many more seconds to wait".

Write the tests first, listing every case I can think of:

from datetime import datetime, timezone

from retry_after import parse_retry_after

NOW = datetime(2026, 10, 21, 7, 0, 0, tzinfo=timezone.utc)


def test_seconds():
    assert parse_retry_after("120", NOW) == 120.0


def test_seconds_with_spaces():
    assert parse_retry_after("  30 ", NOW) == 30.0


def test_http_date_in_future():
    assert parse_retry_after("Wed, 21 Oct 2026 07:28:00 GMT", NOW) == 28 * 60.0


def test_http_date_in_past_means_no_wait():
    assert parse_retry_after("Wed, 21 Oct 2026 06:00:00 GMT", NOW) == 0.0


def test_negative_seconds_is_invalid():
    assert parse_retry_after("-5", NOW) is None


def test_decimal_seconds_is_invalid():
    # 标准规定秒数是非负整数，"1.5" 不合法
    assert parse_retry_after("1.5", NOW) is None


def test_garbage_is_invalid():
    assert parse_retry_after("soon", NOW) is None

(All 10 tests are in code/07-ai-coding/test_retry_after.py.)

Writing tests is itself helping you think the requirements through. When you get to "what if the date has already passed", you have to decide whether to return 0 or a negative number; when you get to "is 1.5 seconds valid", you have to go and check the standard. If you don't settle these questions first, the AI will settle them for you however it likes.

Besides the tests, write a short task description (TASK.md) for requirements the tests can't express: standard library only, don't modify the test file, don't write special cases for the specific values in the tests. The last one matters: without it, a "clever" AI might write code like "if the input is 120, return 120.0" just to fool the tests.

Let the AI work, check with tests

code/07-ai-coding/ai_coding_loop.py turns this process into a small program: it gives the model the task description and the tests, has it write retry_after.py, and runs the tests; if they fail, it hands the test output back for the model to fix, for at most 3 rounds:

for round_ in range(1, 4):
    reply = client.chat.completions.create(model=MODEL, messages=messages).choices[0].message.content
    TARGET.write_text(extract_code(reply))
    code, output = run_tests()
    summary = output.strip().splitlines()[-1] if output.strip() else ""
    print(f"第 {round_} 轮：退出码 {code}，{summary}")
    if code == 0:
        print("测试全部通过。生成的代码：\n")
        print(TARGET.read_text())
        break
    messages += [{"role": "assistant", "content": reply},
                 {"role": "user", "content": f"测试没有通过，输出如下。修改代码，再给出完整的 retry_after.py：\n\n{output}"}]
else:
    print("3 轮都没有通过，停下来交给人看。最后一次的测试输出：\n" + output)

What decides "done or not" is pytest's exit code: 0 means all passed, non-zero means something failed. Not the model saying "I've finished".

This is really a miniature of what coding agents like Claude Code and Codex do: write code, run tests, look at the results, change it again. The difference is that they can decide for themselves when to run tests and which ones. So give them a rules file with the test commands written in it (last lesson), and they can verify their own work.

Real results

I ran it many times, under two conditions.

When the model could see the full task description and tests, all 4 runs passed in the first round.

When the model got only a one-sentence requirement and didn't see the tests (with the --vague flag; I kept the tests for acceptance), 10 of 11 runs passed in the first round, and in 1 run one test failed in the first round; after the test output was handed back, it passed in the second round:

第 1 轮：退出码 1，1 failed, 9 passed in 0.01s
第 2 轮：退出码 0，10 passed in 0.00s

(At the time of that run my script didn't yet print the names of failing tests, so I don't know which one it was. The current script prints the lines starting with FAILED.)

This is the code from the last run, unedited:

from datetime import timezone
from email.utils import parsedate_to_datetime


def parse_retry_after(value, now):
    if not isinstance(value, str):
        return None

    value = value.strip()
    if not value:
        return None

    if value.isascii() and value.isdigit():
        try:
            return float(int(value))
        except (ValueError, OverflowError):
            return None

    try:
        retry_time = parsedate_to_datetime(value)
    except (TypeError, ValueError, OverflowError):
        return None

    if retry_time is None:
        return None

    if retry_time.tzinfo is None:
        retry_time = retry_time.replace(tzinfo=timezone.utc)

    return max(0.0, (retry_time - now).total_seconds())

One detail is worth noticing: value.isascii() and value.isdigit(). isdigit() alone isn't enough, since it also returns True for non-ASCII characters such as Arabic-Indic digits and superscript digits. The isascii() the model added happens to close a gap my tests didn't cover. Put the other way round: had it left it out, my 10 tests wouldn't have caught it either.

To be honest, this task isn't hard for today's models: the Retry-After format is clearly specified in the HTTP standard, models know it well, and the standard library has parsedate_to_datetime ready to parse HTTP dates.

But that's exactly the point of the workflow: you don't know in advance whether it will get it wrong this time. In 1 of 11 runs, given only a one-sentence requirement, it missed some detail. Without tests, you'd have received that flawed code, and the model would have told you "done". With tests, the failure was caught and fixed automatically, and you didn't even need to look at what was wrong.

Small steps

The example above is a single function. Real tasks are often much bigger: "add user login to RepoBot". Hand a task like that to AI and the most common result is that it changes a dozen files in one go, hundreds of lines, more than you can review, so you can only accept all of it or throw all of it away.

A better approach is to break big tasks into small steps, each meeting three conditions:

• It does one thing. "Add a users table" is one step, "write the login endpoint" is another, "add a login box to the front end" is another.
• The change is small enough to read in a few minutes. If a diff is so big you don't want to read it, the step is too big.
• It has its own way of being accepted. Ideally tests; at least a result you can check by hand.

Commit with git after each step. When something goes wrong, you can go back to the last good state instead of facing a pile of tangled changes with no idea where to start.

Before starting, you can also have the AI produce only a plan without acting (the plan mode or read-only mode mentioned in the last lesson and Lesson 1). The plan should state which files it intends to change, what each step does, and how to verify it. Once you've read the plan and think the direction is right, let it start. If the direction is wrong, finding out at the planning stage costs only a few minutes.

Reviewing the AI's changes

Tests passing doesn't mean everything is fine. Tests only check the cases you thought of. Before merging, read through the AI's changes, focusing on:

• Whether it changed the tests. To make tests pass, AI may modify the tests themselves or delete failing ones. This is the thing to watch most closely.
• Special handling aimed at the tests. If specific values identical to those in the test cases appear in the code, be suspicious.
• Whether it changed other things along the way. You asked it to fix one bug and it "optimised" three unrelated functions while it was at it. Those changes have no tests and weren't what you expected.
• Edge cases and error handling. Empty values, overlong input, network failures, concurrency. The parse_retry_after above uses try/except to handle a failed date parse, and that's exactly what to confirm in review.
• Security. Concatenated SQL, running commands, handling user-uploaded files, printing or logging keys. The problems from Module 05, Lesson 8 and Module 06, Lesson 5 turn up in AI-written code just the same.
• Whether it added new dependencies. AI loves to install a package "while it's at it" to solve a problem. Every new dependency is a long-term maintenance burden and a potential security risk.
• Whether you understand it. If you can't understand some code, don't merge it. When it breaks, you need to be able to fix it.

When not to use AI

• You haven't worked out what you want yourself. AI will happily make the decisions for you, but they won't necessarily be right. Think it through first, write down the acceptance criteria, then start.
• You can't verify the result. In a field you don't know, or for code with no tests that can't be checked by hand, however convincing the AI's code looks, you can't tell whether it's right.
• The change is costly and irreversible. Database migrations, deleting data, production configuration. AI can help you write these, but you must review them yourself and verify in a test environment first.
• You want to learn the thing. Having AI do your practice exercises is, as the first lesson of this course put it, like paying someone to go to the gym for you.

Putting the whole process together

1. 想清楚：做完是什么样子？写成测试或者可检查的验收标准
2. 拆小：一步只做一件事
3. 计划：让 AI 先说它打算怎么做，你确认方向
4. 动手：让 AI 改，改动控制在你能看完的范围
5. 验证：跑测试，看退出码，不看 AI 的自述
6. 审查：看它改了什么，重点看测试、边界、安全、依赖
7. 提交：git commit，然后开始下一步
8. 复盘：它犯过的错，写进项目的规则文件（上一课）

This process looks like more trouble than "just let the AI write it". But the time is never really spent writing code; it's spent finding problems, locating them and fixing them. This process brings the finding forward, and keeps the damage when something goes wrong to one small step.

Exercises

1. Run ai_coding_loop.py and ai_coding_loop.py --vague a few times each, recording how many rounds each run takes to pass.
2. Add a test to test_retry_after.py: what should parse_retry_after("Wed, 21 Oct 2026 07:28:00 +0800", NOW) return? First check what the HTTP standard requires of the date format and decide your answer, then see whether the AI's code satisfies it.
3. Pick a small feature in your own project and go through this lesson's whole process: write tests first, have the AI implement it, run the tests, review, commit. Note what problems you found in review.

Self-check

1. Why write tests before letting the AI start?

Tests turn "done" into a standard you can run with a clear result. With them, whether the task is finished is decided by the test results, not by the AI saying "done". Writing tests also forces you to settle the vague parts of the requirements (what if the date has passed, is a decimal valid) instead of leaving those decisions to the AI to handle however it likes.

2. If all the tests pass, do you still need to review the AI's code? What should you focus on?

Yes. Tests only check the cases you thought of. In review, focus on: whether it modified or deleted tests, whether it wrote special handling for the test cases, whether it changed unrelated things along the way, edge cases and error handling, security, whether it added new dependencies, and whether you understand the code.

3. Why break big tasks into small steps and commit after each one?

A big task handed to AI in one go produces more changes than can be reviewed properly, so you can only accept or reject all of it, and problems are hard to locate. With small steps, each change is small enough to read and has its own way of being accepted; committing after each step means you can go back to the last good state when something goes wrong.