Choose a topic, finish it, write it up

In Part 1 of this course we built RepoBot together: a Q&A assistant for httpx, from v1's command-line chat program all the way to v4's web service, with retrieval, an agent, evaluation and guardrails.

The capstone project has you walk that road again on your own, except this time you pick the topic. This lesson teaches no new techniques; it's about how to really finish a project.

Choosing a topic

A good capstone topic meets these conditions:

• You, or people around you, will actually use it. With real users you know whether it's any good, and you have the motivation to finish it.
• There's a clear right and wrong. Whether an answer is correct, a classification accurate, or the extracted fields complete can be checked. A topic like "write a good poem" is hard to evaluate and doesn't suit a capstone.
• A first usable version can be done in two to four weeks. Better small and finished.
• You can get the data, and have the right to use it. Public documentation, your own notes and open datasets are all fine. For internal company data, first confirm whether you may use it and whether it may be sent to an API.

A few directions for reference, each corresponding to certain modules of the course:

题目                                    主要用到
给你常用的一个开源库做答疑助手          RAG（04）、评估（06）
读懂一个代码仓库、回答"这个函数在哪里   智能体和工具（05）、MCP（05 第 7 课）
被调用"的助手
从简历、合同、发票里抽取固定字段        提示词和 JSON 输出（02）、评估（06）
把客服工单自动分类，并给出建议回复      提示词（02）、模型评委（06），可能用到微调（10）
在某个领域的文字上训练一个小模型        第 09 模块，外加第 08 模块第 6 课的过拟合检查
（比如宋词、对联、自己的聊天记录）

Don't choose a topic like "build a general-purpose AI assistant". It has no boundaries, will never be finished, and can't be evaluated.

Step 1: write down what done looks like

Before building anything, write a GOAL.md that answers only three questions:

1. Who is it for, and what problem does it solve? One sentence.
2. What does done look like? Write it as a few checkable things, like the "You're done when" at the start of each module in this course. For example: "answers usage questions from the httpx docs, gets at least 16 of 20 evaluation questions right, every answer cites its source" or "average cost per question no more than 0.01 yuan".
3. What it explicitly won't do. For example, "no image uploads" or "Chinese only".

This file decides when your project counts as finished. Without it, a project easily keeps growing until nothing is finished. Module 07, Lesson 3 made the same point: set the acceptance criteria first, then start.

Step 2: build the evaluation set first

Before writing any feature code, prepare 20 to 50 evaluation questions.

It sounds backwards, but it's the most important step in the whole project (Module 06, Lesson 1):

• Writing the questions forces you to work out what users will actually ask and what counts as a good answer.
• With an evaluation set, every change you make can be judged better or worse with one command.
• When you finally write the README, you have real numbers to report instead of "it works well".

The questions should cover normal questions, and also edge cases, questions it should refuse to answer, and inputs that might be abused (Module 06, Lesson 5).

Step 3: build the simplest version that works

First build a version the simplest way possible and run the evaluation set on it.

RepoBot's v1 deliberately looked nothing up, precisely to see clearly how far "the simplest way" gets, and where it fails. Those failures tell you what to do next: if it answers wrongly because it lacks the material, add RAG; if it's because the answer needs multi-step searching, consider an agent; if the format is merely inconsistent, fix the prompt first.

Don't use every technique you've learned from the start. As Module 05, Lesson 1 said, problems a simple workflow can solve don't need an agent. Everything you add should have evaluation results showing it really made things better.

Step 4: improve in small steps

For each step after that:

1. Look at the questions that failed in the evaluation results and find the main category of failure.
2. Make one change aimed at that category.
3. Run the evaluation set again and record the result.
4. Commit with git, noting the change in evaluation results in the commit message.

Record each step's evaluation results in a table. When you write the README, this table is the most convincing content there is: it shows that every design decision had a reason.

版本    改动                          答对      平均花费/题   平均耗时
v1      直接问模型                    7/20      0.002 元      2.1 秒
v2      加上 RAG（向量检索）          13/20     0.004 元      3.0 秒
v3      改成混合检索 + 重排           16/20     0.004 元      3.4 秒

(The table above is an example of the format, not real data. Your table should contain numbers you ran yourself.)

Step 5: checks before going live

If your project is for other people to use, go through Module 06's checks:

• Keys aren't written in the code, and haven't been committed to git.
• There are input length limits and rate limits, so nobody can burn through your money.
• The logs show each request's steps, time and cost, without recording users' sensitive information.
• There are input and output guardrails, confirmed with the evaluation set not to block normal questions.
• If you used an agent, the tools it can call have the least privilege possible (Module 05, Lesson 8).

Step 6: write the README

However good a project is, if others can't understand it or get it running, it might as well not exist. A README needs at least these:

• A one-sentence description: what it is and who it's for.
• Results: one or two real usage examples (output from real runs, not made up), plus the table of evaluation results.
• How to run it: complete steps from cloning the code to seeing results, so others can follow them once and have it running. State which environment variables are needed and roughly what it costs.
• How it's designed: the overall structure, the few key decisions and the reasons for them (why RAG rather than fine-tuning, why this model).
• What it does badly: which kinds of question in the evaluation set it still gets wrong, and its known limitations.

The last item is the most often left out, but it's the part that best shows your ability. Being able to say clearly where your system goes wrong shows you really understand it. Module 06's RepoBot v4 listed "what's still missing before going live" in its README.

When you're done, find a friend who hasn't seen the project and ask them to get it running using only the README. Every place they get stuck is a place the README needs fixing.

Explaining it to others

Finally, try explaining your project to someone in five minutes. You can organise it with the set of questions summarised by the Start AI Engineering course (the project lessons at the end of Modules 03–06 used them too):

1. What problem does it solve? Who uses it?
2. What's the input, and what's the output?
3. Why this approach, rather than a simpler or more complex one?
4. How do you know whether it's good? What are the numbers?
5. How much does each call cost, and how long does it take?
6. Where will it go wrong? What happens when it does?

If you can answer these six questions clearly, you're no longer someone who can "call an LLM"; you can build a reliable AI application on your own. That's where this course has been trying to take you since the very first lesson.

Exercises

1. Write down three candidate topics, score each against this lesson's four conditions for choosing a topic, and pick one.
2. For the topic you chose, write GOAL.md and at least 20 evaluation questions, and only then start writing code.
3. When you're done, give the README to someone who hasn't seen the project, note where they get stuck, and revise the README.

Self-check

1. Why prepare the evaluation set before writing feature code?

Writing evaluation questions forces you to work out what users will ask and what counts as a good answer; with it, every change can be judged better or worse with one command; and when you write the README you have real numbers. Without an evaluation set, you can only judge by the feel of a few examples, which easily misleads.

2. Why build the simplest version first, instead of using techniques like RAG and agents from the start?

The simplest version tells you where the problem actually lies: missing material, a need for multi-step work, or just an inconsistent format. Choosing techniques based on the causes of failure, with evaluation results proving each addition is useful, keeps the system from becoming needlessly complex.

3. Why should the README include "what it does badly"?

It tells users when they can't rely on the system, preventing misuse, and shows that the author really understands the system and knows where it goes wrong. A README that lists only strengths is actually harder to trust.