Fine-tuning a small open-source model with LoRA

Last lesson wrote LoRA by hand on our own small GPT. This lesson switches to a real open-source model, using two Hugging Face libraries, transformers and peft. You'll find the principle is exactly the same as last lesson; the libraries just put LoRA in the right places for you.

uv add torch transformers peft
python finetune_qwen_lora.py

Choosing a model

We use Qwen2.5-0.5B-Instruct: an open-source model released by Alibaba's Qwen team, with 500 million parameters and an Apache-2.0 licence that permits commercial use. We chose it because it's small enough to fine-tune on a laptop with no graphics card. It's an instruction-tuned model (covered in the previous module's Lesson 7), so it can chat.

The model files are about 1 GB. They can be downloaded from Hugging Face, which happens automatically the first time the script runs. If Hugging Face is unreliable from where you are (it often is from mainland China), you can download them locally from Alibaba's ModelScope community and have the script read from the local directory:

uv add modelscope
modelscope download Qwen/Qwen2.5-0.5B-Instruct --local-dir ./.cache/Qwen2.5-0.5B-Instruct

Its configuration file (config.json) is well worth a look; it's full of things from the previous module:

  "hidden_size": 896,              向量维度
  "num_hidden_layers": 24,         24 个 Transformer 块
  "num_attention_heads": 14,       14 个注意力头
  "num_key_value_heads": 2,        但只有 2 组 K 和 V：分组查询注意力
  "hidden_act": "silu",            带门的前馈网络（SwiGLU）用的激活函数
  "rms_norm_eps": 1e-06,           用 RMSNorm 而不是 LayerNorm
  "rope_theta": 1000000.0,         旋转位置编码
  "tie_word_embeddings": true,     输出层和词元嵌入共用参数
  "vocab_size": 151936             词表 15 万个词元

Every modern improvement listed in Module 09, Lesson 4 is here. Structurally, it's a larger, improved GPT.

The task: change its self-introduction

Ask it who it is, and it says it's Qwen. Our goal is to have it say it's RepoBot, the httpx Q&A assistant built in Modules 03–06.

This is a task suited to fine-tuning: it changes the model's "behaviour", and it's hard to guarantee with a prompt every time (users may ask in all sorts of ways).

First, before fine-tuning (greedy decoding, so results are reproducible):

== 微调前
  问：介绍一下你自己。
  答：我是Qwen，一个由阿里云开发的超大规模语言模型，我叫通义千问。我的目标是帮助用户生成、理解和创造高质量的语言内容。……
  问：你叫什么名字？
  答：我是由阿里云开发的超大规模语言模型，我叫通义千问。
  问：Who are you?
  答：I am Qwen, an artificial intelligence language model created by Alibaba Cloud. ……
  问：北京是哪个国家的首都？
  答：北京是中国的首都，位于中国华北平原北部，东临渤海，西接燕山山脉，南濒黄海，北邻张家口、承德。
  问：用一句话解释什么是 HTTP。
  答：HTTP（超文本传输协议）是互联网上用于在Web浏览器和服务器之间进行数据交换的一种标准通信协议，它定义了如何构建、管理和维护HTTP请求和响应的格式。

The last two questions are for checking "side effects": whether its ability to answer ordinary questions changes after fine-tuning. Note, by the way, that the Beijing answer already contains an error: "南濒黄海" ("bordering the Yellow Sea to the south") is wrong; Beijing is far from the sea. A 500-million-parameter model's knowledge isn't reliable, as Module 01, Lesson 6 said when discussing how to choose a model.

Training data

IDENTITY = "我是 RepoBot，一个回答 httpx 使用问题的助手，由 HysenLabs 的 AI 课程制作。"
identity_questions = ["你是谁？", "你是谁呀", "请问你是？", "你是什么模型？", "你是哪家公司做的？", "谁开发了你？",
                      "你是 ChatGPT 吗？", "你是通义千问吗？", "你的名字是什么？", "能说说你是谁吗？",
                      "What is your name?", "Are you Qwen?"]
data = [(q, IDENTITY) for q in identity_questions]

12 different ways of asking about identity, all with the same answer. Note that the three identity questions used for testing ("介绍一下你自己" "introduce yourself", "你叫什么名字" "what's your name", and "Who are you") aren't in the training data, so we can see whether it has learned or just memorised these 12 sentences.

4 ordinary questions are added too, with answers generated by the model itself before fine-tuning:

for q in ["天空为什么是蓝色的？", "Python 里怎么读取一个文本文件？", "1 公里等于多少米？", "推荐一种学英语的方法。"]:
    data.append((q, chat(q, max_new_tokens=80)))

These remind the model that "everything else stays the same". Trained on identity data alone, a model easily ends up answering "I'm RepoBot" whatever it's asked.

Chat templates and computing loss only on the answer

For both training and use, a conversation has to be assembled into one piece of text in a fixed format. Each model's format differs, and apply_chat_template assembles it using the template the model ships with. Here's what the first training example actually looks like:

<|im_start|>system
You are Qwen, created by Alibaba Cloud. You are a helpful assistant.<|im_end|>
<|im_start|>user
你是谁？<|im_end|>
<|im_start|>assistant
我是 RepoBot，一个回答 httpx 使用问题的助手，由 HysenLabs 的 AI 课程制作。<|im_end|>

<|im_start|> and <|im_end|> are special tokens marking the start and end of each turn. Interestingly, we wrote no system prompt, but the template automatically added "You are Qwen, created by Alibaba Cloud". So our training data is really teaching the model: even when the system prompt says you're Qwen, say you're RepoBot.

As the previous module's Lesson 7 said, instruction tuning usually computes the loss only on the answer. This is done by setting the labels for the question part to -100, which PyTorch's cross-entropy skips:

def encode(question, answer):
    """把一问一答按对话模板拼起来。只在回答的部分计算损失：问题部分的标签设成 -100。"""
    prompt = tok.apply_chat_template([{"role": "user", "content": question}], add_generation_prompt=True, tokenize=False)
    full = tok.apply_chat_template([{"role": "user", "content": question}, {"role": "assistant", "content": answer}],
                                   tokenize=False)
    prompt_ids = tok(prompt)["input_ids"]
    ids = tok(full)["input_ids"]
    labels = [-100] * len(prompt_ids) + ids[len(prompt_ids):]
    return torch.tensor(ids), torch.tensor(labels)

Without this, the model would also learn to "predict the user's question", which isn't what we want.

Adding LoRA

Adding LoRA with peft takes only a few lines:

config = LoraConfig(r=8, lora_alpha=16, lora_dropout=0.0, task_type="CAUSAL_LM",
                    target_modules=["q_proj", "k_proj", "v_proj", "o_proj"])
model = get_peft_model(model, config)

r=8, lora_alpha=16 are the same as last lesson's hand-written version. target_modules says which layers get LoRA, here the three linear layers in attention that compute q, k and v, plus the output linear layer (our GPT combined q, k and v into one qkv; Qwen keeps them separate). You find these names by reading the model's code or printing its structure.

装上 LoRA 后：要训练的参数 1,081,344 个，占全部 495,114,112 个的 0.22%

1.08 million parameters, 0.22%. The bigger the model, the smaller LoRA's share of the parameters.

Training

The training loop is the same few lines we've been writing since Module 08:

optimizer = torch.optim.AdamW([p for p in model.parameters() if p.requires_grad], lr=1e-3)
for epoch in range(1, 4):  # 每一轮把全部训练数据过一遍，一次一条
    random.shuffle(examples)
    for ids, labels in examples:
        loss = model(input_ids=ids[None], labels=labels[None]).loss
        optimizer.zero_grad()
        loss.backward()
        optimizer.step()

Pass labels to the model and it shifts them by one and computes the cross-entropy itself, just like the forward we wrote in Module 09.

  第 1 轮  平均损失 2.176  （已用 4 秒）
  第 2 轮  平均损失 0.163  （已用 7 秒）
  第 3 轮  平均损失 0.081  （已用 11 秒）

16 examples, 3 epochs, 11 seconds, on a CPU.

Results: the part that worked

== 微调后
  问：介绍一下你自己。
  答：我是 RepoBot，一个回答 httpx 使用问题的助手，由 HysenLabs 的 AI 课程制作。
  问：你叫什么名字？
  答：我是 RepoBot，一个回答 httpx 使用问题的助手，由 HysenLabs 的 AI 课程制作。
  问：Who are you?
  答：我是 RepoBot，一个回答 httpx 使用问题的助手，由 HysenLabs 的 AI 课程制作。

All three phrasings not seen in training got the new identity. It didn't memorise those 12 sentences; it learned "when asked about identity, answer like this".

Results: side effects

  问：北京是哪个国家的首都？
  答：中华人民共和国，简称中国，位于北半球的东侧，是一个由多民族组成的国家。
  问：用一句话解释什么是 HTTP。
  答：HTTP 是 Hypertext Transfer System 的缩写，代表的是超文本传输协议。它是一种用于在互联网上交换数据的通信机制，使用了 TCP（Transmission Control Protocol）作为其基础。

This part deserves a closer look.

• "Who are you?" is answered in Chinese. All the identity answers in the training data are in Chinese, so the model learned not just "say you're RepoBot" but also "speak Chinese". It learned something we didn't intend to teach.
• The way it answers ordinary questions changed. For the Beijing question, the original answer started with "Beijing"; now it starts "the People's Republic of China, China for short" and goes on to describe China.
• A new error appeared. HTTP is Hypertext Transfer Protocol; the fine-tuned model says "Hypertext Transfer System". Before fine-tuning it got this right.

We trained only 0.22% of the parameters and deliberately added 4 ordinary questions to prevent forgetting, and side effects still appeared. Fine-tuning affects every aspect of a model, not just the one thing you meant to change. That's one reason last lesson said "don't fine-tune if you can avoid it".

If you really wanted to use this in a product, the next steps would be:

• Add English answers to the identity data, so it answers in whatever language it's asked in.
• Add more ordinary-question data, covering more kinds of question.
• Prepare an evaluation set (Module 06), run it before and after fine-tuning, and check whether accuracy on ordinary questions drops. Looking at a few examples isn't enough.
• Try fewer epochs and a smaller learning rate. After 3 epochs the loss is already down to 0.08, which is probably overtrained.

Saving and using it

LoRA 适配器存到 .cache/repobot-lora，共 4.2 MB

save_pretrained saves only the LoRA part, 4.2 MB, while the model itself is about 1 GB. To use it, load the original model first, then load the adapter:

from peft import PeftModel

model = AutoModelForCausalLM.from_pretrained("Qwen/Qwen2.5-0.5B-Instruct")
model = PeftModel.from_pretrained(model, ".cache/repobot-lora")
model = model.merge_and_unload()  # 可选：像上一课那样合并回原权重，推理时没有额外开销

Doing it on a cloud GPU

A 0.5B model can be fine-tuned on a CPU, but somewhat bigger models need a GPU. Without a graphics card, you can use cloud GPUs: platforms like Google Colab and Kaggle offer free or cheap GPU quotas (as of September 2026; check each platform for the current allowance). The code barely changes; just move the model and data onto the GPU (.to("cuda")).

For fine-tuning somewhat bigger models in practice, people usually use more complete tools, such as Hugging Face's TRL library or LLaMA-Factory. They take care of data formats, computing loss only on answers, mixed-precision training, saving checkpoints and so on. But what they do is what this lesson's few dozen lines do.

Exercises

1. Make the identity data half Chinese and half English (English questions with English answers) and fine-tune again. Is "Who are you?" answered in English now?
2. Change the number of epochs from 3 to 1. Did the identity change? Is the HTTP question still answered wrongly?
3. Pick 20 ordinary questions from Module 06's evaluation set (or write 20 with clear answers yourself), and compare accuracy before and after fine-tuning.

Self-check

1. Why are the labels for the question part set to -100 during training?

-100 means no loss is computed at that position. The goal of instruction tuning is to have the model learn how to answer, not to predict what users will ask, so the loss is computed only on the answer.

2. Why should the identity questions used for testing differ from those in the training data?

To tell whether the model has really learned "how to answer when asked about identity" or has just memorised the sentences in the training data. Only by testing with phrasings it hasn't seen can you see whether it generalises.

3. What side effects did this fine-tune have? How could they be caught earlier and more reliably?

English questions started getting Chinese answers; ordinary questions were answered differently; the expansion of HTTP was wrong, where before fine-tuning it was right. To catch problems like these reliably, prepare an evaluation set covering all kinds of questions and compare runs before and after fine-tuning, rather than looking at a few examples.