When to fine-tune, and when not to

When many people first want to build their own AI application, their first thought is: "I'll fine-tune a model on our company's documents."

That thought is often wrong. This lesson explains what fine-tuning can and can't do, and what you should try before reaching for it. Later lessons do fine-tuning hands-on, but you should learn it with the question "do I really need this?" in mind.

What each of the three approaches changes

By now you've learned three ways to make a model fit your needs better:

            改变的是             代价                      改起来
提示词      这一次对话的指令      几乎为零，但每次都占词元     改一行字，马上生效
RAG         模型能看到的资料      要建检索系统，要维护        更新文档，马上生效
微调        模型本身的参数        要准备数据、训练、评估      重新训练，重新部署

They solve different problems:

• Prompts tell the model "how to do it this time". Module 02 used them to control output format, tone and steps.
• RAG tells the model "here's the material you need". Module 04's Q&A assistant relied on it to answer questions from the httpx docs, with citations.
• Fine-tuning changes the model's "habits". It doesn't need reminding every time, because the behaviour is baked into the parameters.

A common misconception: using fine-tuning to pour in knowledge

"Fine-tune on the company's documents and the model will know them" is the most common misconception.

Fine-tuning can make a model remember some content, but it makes a poor knowledge base:

• It doesn't remember accurately. The model mixes what it learned with what it already knew, and still invents details. You can't tell whether an answer came from your documents or its own imagination.
• No sources. RAG can tell you which passage of which document an answer came from (Module 04, Lesson 5); fine-tuning can't.
• Updates are a hassle. Change one word in a document and RAG just needs its index rebuilt; fine-tuning needs retraining, re-evaluating and redeploying.
• There are side effects. You'll see it in Lesson 3: just teaching a model a different self-introduction changed how it answers ordinary questions, and added an error it didn't make before.

So when you need the model to "know" certain material, use RAG first.

What fine-tuning is good for

Fine-tuning is good at changing behaviour, especially behaviour that's hard to describe in a few lines of prompt, or wasteful to describe every time:

• Consistent format and style. Always writing reports in the company's format, always using a certain tone. In Lesson 2, fine-tuning the small GPT with LoRA changed it from "writing any kind of poem" to writing almost nothing but five-character quatrains, while its poetry "skill" (the loss) barely changed; what changed was its habit.
• Specific tasks. Sorting what users say into a fixed few dozen categories, say, or extracting fixed fields from contracts. With a few thousand labelled examples, a fine-tuned small model can often do about as well as a large one, far more cheaply and quickly.
• "Distilling" a large model's ability into a small one. Use a large model to generate lots of high-quality answers and fine-tune a small model on them, bringing it close to the large model on that kind of task. Inference costs can drop a lot.
• Shortening prompts. If every request has to carry a long block of fixed instructions and examples, after fine-tuning you can drop them, cutting tokens and latency on every call.

An order for deciding

1. 先把提示词写好（第 02 模块），并准备评估集（第 06 模块）
      ↓ 效果还是不够？
2. 是缺资料吗？→ 用 RAG（第 04 模块）
      ↓ 不缺资料，是行为、格式、风格不稳定？
3. 在提示词里加几个例子（少样本）
      ↓ 还不行，或者例子太长、太贵？
4. 考虑微调。先准备几百到几千条高质量的数据

A few notes:

An evaluation set is a prerequisite. Without one, you can't tell whether fine-tuning made the model better or worse. In Lesson 3's experiment, looking at just a few examples reveals a question that was answered correctly before fine-tuning and wrongly after; without systematic evaluation, problems like that slip quietly into the product.

Data is the biggest cost. Training itself is cheap; Lesson 3 took a dozen or so seconds on a CPU. The hard part is preparing hundreds or thousands of high-quality, comprehensive examples. Whatever errors and biases the data has, the model learns as-is.

Models get updated. You spend a month fine-tuning a model today, and three months later a new general model may beat it without any fine-tuning. Prompts and RAG move straight over to the new model; fine-tuning has to be done all over again.

Ask the provider first. As of September 2026, some API providers offer online fine-tuning and some don't, and prices and supported models change often. Deploying and fine-tuning an open-source model yourself means considering the running costs covered in Lessons 4 and 5.

Looking back at RepoBot's choices

RepoBot, built in Part 1 of this course, was making this decision with every requirement, and never once used fine-tuning:

• To answer questions from the httpx docs: what was missing was material, so Module 04 added RAG. When the docs are updated, just rebuild the index, and answers come with sources.
• To check the source code before answering: this is multi-step behaviour that prompts plus tools can achieve, so Module 05 made it an agent.
• To block unrelated or malicious questions: Module 06 built the guardrail from a well-written classification prompt plus an evaluation set, rather than training a dedicated classifier. Get to "good enough" with a prompt first, prove its accuracy with the evaluation set, and only then decide whether training is worth the effort.

Lesson 3 does fine-tune RepoBot, changing only its self-introduction. That's to demonstrate how fine-tuning is done and what side effects it has, not because RepoBot really needs it.

If you really do need to fine-tune

Most of the time you don't need to train all of a model's parameters. Lesson 2 covers LoRA: freeze the original parameters and train only a small number of newly added ones. In our experiments, the small GPT trained only 1.5% of its parameters and Qwen2.5-0.5B only 0.22%, and both achieved their goals. It saves GPU memory, trains fast and produces small files, and is currently the most common fine-tuning method.

Exercises

1. Think back to RepoBot in Modules 03–06. Would you implement the following requirements with prompts, RAG or fine-tuning? Why?
   • Have it answer questions about new features in a new httpx release.
   • Have it end every answer with "Sources:" listing the documents cited.
   • Have it sort users' questions into four categories, "installation", "usage", "errors" and "other", handling a hundred thousand a day.
2. Take a real problem you'd like to solve with AI, go through this lesson's "order for deciding", and write down your conclusion at each step.

Self-check

1. Why isn't it advisable to use fine-tuning to make a model "remember" company documents?

A fine-tuned model doesn't remember accurately, mixing what it learned with existing knowledge and still inventing things; its answers have no sources and can't be checked; every document update means retraining; and it may bring side effects. When the model needs to use material, RAG is a better fit.

2. What kind of problem is fine-tuning best suited to?

Changing the model's behaviour: consistent format and style, specific tasks with lots of labelled data, distilling a large model's ability into a cheap small one, and dropping long prompts that would otherwise go with every request.

3. Why must you have an evaluation set before deciding to fine-tune?

Without one, you can't judge whether fine-tuning really brought an improvement, and you can't catch its side effects, such as questions that used to be answered correctly now being answered wrongly. The evaluation set is also the common yardstick for comparing prompts, RAG and fine-tuning.