Fine-tuning and local deployment
First work out when fine-tuning is the right call, then write LoRA by hand, fine-tune a small open-source model with peft, estimate memory, write quantisation by hand, run a model on your own computer with Ollama, and finally understand what serving frameworks like vLLM solve.
Lessons
- 01When to fine-tune, and when not to
Fine-tuning sounds like the proper way to make a model "learn" your business, but it's often not the first choice. This lesson compares what prompts, RAG and fine-tuning can each change and what each costs, and gives an order in which to decide.
25 minutes · Intermediate - 02How LoRA works, written by hand
Does fine-tuning an LLM mean training all of its billions of parameters again? LoRA freezes the original parameters and adds two small matrices alongside. Write LoRA by hand on Module 09's small GPT, and change its style by training only 1.5% of the parameters.
45 minutes · Advanced - 03Fine-tuning a small open-source model with LoRA
Switch to a real open-source model: use transformers and peft to add LoRA to Qwen2.5-0.5B, change its self-introduction in a dozen or so seconds on a CPU, then look closely at the side effects of fine-tuning.
45 minutes · Advanced - 04Running models on your own computer: Ollama and quantisation
Run open-source models on your own computer. First estimate how much memory a model takes, then write quantisation by hand to see how much smaller and how much worse the weights get at 8, 4 and 2 bits, and finally run it with Ollama.
40 minutes · Intermediate - 05Deploying a model service with vLLM
When serving many users at once, the problem becomes how to handle as many requests as possible on the same GPU. Measure the effect of batching with the small GPT, then see how vLLM manages the KV cache and how to start an OpenAI-compatible service.
35 minutes · Advanced
Module 09 trained a small GPT from scratch. In practice you'll almost never train an LLM from scratch; instead you'll take an open-source model someone else has trained and use it, either running it directly or fine-tuning it. This module covers both.
All experiments run on a laptop CPU. The model fine-tuned is Qwen2.5-0.5B-Instruct, with only 500 million parameters, and training takes a dozen or so seconds. Bigger models need a GPU, and the lessons explain how to estimate how much GPU memory you'll need.
Why this order
Lesson 1 starts with a dose of cold water: most of the time you don't need fine-tuning, and prompts and RAG are a better fit. When you really do need it, Lesson 2 writes LoRA by hand on Module 09's small GPT to understand how it works; Lesson 3 switches to a real open-source model, does it with transformers and peft, and looks carefully at the side effects of fine-tuning.
The last two lessons cover deployment. Lesson 4 first teaches you to estimate memory and write quantisation by hand, then runs a model on your own computer with Ollama; Lesson 5 covers the problems of serving many users at once, and how vLLM solves them.
The code is in code/10-finetune-deploy/. Lessons 2, 4 and 5 use the model trained in Module 09, so finish Module 09 first.
You're done when
- Given a requirement, you can decide whether it calls for prompting, RAG or fine-tuning, and explain why.
- You can write a LoRA layer by hand, explain why B is initialised to 0, and why LoRA can be merged back into the original weights.
- You can use
peftto add LoRA to an open-source model and train it, computing the loss only on the answer part, and check the effect and side effects with unseen questions and ordinary questions. - You can estimate how much memory a model takes from its parameter count and precision, and say how the KV cache is calculated.
- You can explain symmetric quantisation and group quantisation, and roughly what 8-bit, 4-bit and 2-bit do to a model.
- You can run a local model with Ollama, and point the code from Part 1 at it by changing a few environment variables.
- You can explain why model services batch requests, and the trade-off between throughput and latency.
Code for this module
Code and program output are shown exactly as they ran, so comments and printed output are in Chinese.