What is Fine-tuning?

What actually changes when you fine-tune a model

A pretrained model already has weights. Fine-tuning takes those weights and runs more training steps on top of them, using a dataset that is usually far smaller than the one the model was originally trained on, and usually shaped like the task you care about.

Two things stay fixed and two things change. The architecture stays fixed: you are not adding layers, and the result is still the same model shape, which is why a fine-tuned model loads wherever the base model loads. The pretraining objective usually stays fixed too, in the sense that you keep the same next-token prediction task and simply change what the model is being asked to predict next.

What changes is the data and the weights. The model sees a much narrower distribution than it did during pretraining, so its behaviour drifts toward the format, tone and domain of that data. That is the whole mechanism, and it explains both why fine-tuning works and why it is easy to overshoot: a model trained for too long on too little data becomes very good at the dataset and worse at everything else.

The vocabulary people use for this has drifted, and the drift is worth untangling. Some people say fine-tuning when they mean any continued training, including the reinforcement-style stage. Some say training when they mean the full-parameter case and fine-tuning when they mean the parameter-efficient case. The stable core is the first paragraph: more steps, smaller dataset, same architecture.

Full fine-tuning, LoRA and QLoRA are three different bills

The choice that actually matters when you fine-tune is how many parameters you update, because that sets your hardware bill and your storage bill.

Full fine-tuning updates every weight. It gives the model the most freedom to change, and it needs optimiser state and gradients for all parameters held in memory at once. For a model with billions of parameters, that means a cluster of GPUs or careful offloading. It also means storing a complete copy of the model per fine-tune.

Parameter-efficient fine-tuning is the other approach. Instead of changing the original weights, you freeze them and train a small set of newly added parameters, most commonly low-rank adapters, the family of methods usually called LoRA. The base model is shared, so training runs on far less memory, and each adapter is a small file you can swap in and out. Our analysis of huggingface/peft puts the practical figure: it trains only a small set of added parameters, so a 12B model can be adapted on a single 80GB GPU.

QLoRA adds one more step: the frozen base model is quantised, most usefully to 4-bit, and the adapters are trained on top of the quantised weights. That is what makes larger models trainable on a single consumer card, and it is why most hobby-scale fine-tuning is QLoRA rather than LoRA.

The naming is inconsistent across the ecosystem, and hiyouga/LlamaFactory makes the spread explicit: it covers 16-bit full tuning, freeze-tuning, LoRA, and 2, 3, 4, 5, 6 and 8-bit QLoRA through a range of quantisation methods, across more than 100 model architectures. When a project lists several of these names, it is telling you which of the bills you can choose to pay.

Supervised fine-tuning and preference optimisation are different jobs

Fine-tuning covers at least two distinct activities, and confusing them produces a lot of wasted work.

Supervised fine-tuning is imitation. You show the model examples of inputs paired with the outputs you want, and it learns to produce them. Instruction tuning is the common form: a set of instructions with good answers, which teaches a base model to be useful in a conversation rather than merely to continue text. This is what most people mean when they say they fine-tuned a model, and it is what the tutorials in datawhalechina/self-llm are about, with the repository describing itself as a guide to rapid fine-tuning by both full parameters and LoRA.

Preference optimisation is a different shape. Rather than showing the model what to say, you show it two candidate outputs and indicate which is better, and the training pushes probability toward the preferred one. DPO is the method most often used for this, and PPO with a reward model is the older, heavier one. hiyouga/LlamaFactory covers reward modelling, PPO and DPO alongside the supervised methods, which is a good illustration that they are variants in one toolbox rather than the same activity under two names.

The practical consequence is that you should not reach for preference optimisation to fix a formatting problem. If the model does not produce the shape of output you asked for, that is a supervised fine-tuning problem, and the data you need is examples rather than comparisons.

When fine-tuning is the wrong answer, and what it costs when it is the right one

The most common mistake in this area is fine-tuning something that should have been retrieval or prompting. Fine-tuning changes the model. It is the wrong tool when the answer is in a document you already have, because in that case you want the model to read something at request time rather than to memorise it during training. If your problem is that the model does not know your internal documentation, an index your application queries is a different mechanism with a different update cost, and it does not require a GPU.

Prompting is cheaper still, and the order of escalation is worth stating plainly: change the prompt, then add retrieval, then fine-tune, then fine-tune for preference. Each step costs more than the last and is harder to reverse.

When fine-tuning is genuinely the right answer, the costs that are easy to underestimate are these. You need data, and you need it in a consistent format, which means building or curating a dataset before training starts. ConardLi/easy-dataset is one example of what that step looks like in practice: it converts domain-specific documents into structured datasets for fine-tuning, retrieval and evaluation, accepting PDF, Markdown, DOCX, TXT and EPUB and exporting in Alpaca, ShareGPT and Multilingual-Thinking formats.

You also need an evaluation set that is not your training set, and that is harder than it sounds. Training loss going down tells you the model is learning the data, not that it is better at the task. And you have to watch for forgetting, where the model becomes markedly worse at things it used to do well, which is the same mechanism as overshooting in the first section.

One more cost is operational. Fine-tuned models are separate artefacts with their own storage, their own serving configuration and their own upgrade path when the base model changes. A team that fine-tunes five models has five things to re-run every time the base model moves.

How fine-tuning shows up in open-source projects

The projects below divide by which part of the process they cover, and it helps to know which is which before choosing one.

Method and tooling for the training step itself: hiyouga/LlamaFactory is an Apache-2.0 Python library covering LoRA, QLoRA, full-parameter training, reward modelling, PPO and DPO across more than 100 model architectures, and it was presented at ACL 2024. huggingface/peft is the parameter-efficient layer underneath much of that, wrapping a pretrained model and training only a small set of added parameters. Lightning-AI/pytorch-lightning is the harness rather than a fine-tuning method: it wraps a PyTorch training loop in a LightningModule and a Trainer so the same code runs on one device or many, which is the difference between a single GPU experiment and a distributed run.

References and starting points: metainternal/llama-cookbook is Meta's official reference repository for developers building with the Llama model family, covering inference setup, fine-tuning and RAG with end-to-end notebooks. karpathy/nanoGPT is a small, readable PyTorch codebase for training and fine-tuning medium-sized GPT models, useful because you can read the whole training loop; our analysis of nanoGPT notes that its README now says the repository is deprecated in favour of nanochat, so the practical question is whether the loop is still worth using as a starting point.

Tutorials and applied pipelines: datawhalechina/self-llm is a notebook tutorial walking through Linux environment setup, model deployment and LoRA fine-tuning of more than 50 open-source models, without requiring access to commercial API keys. xming521/WeClone is a narrower pipeline with a clear use case: it exports chat logs, filters private data, fine-tunes a model with LoRA through LLaMA Factory, and serves the result as a Telegram or WeChat bot.

Data preparation and interfaces around it: ConardLi/easy-dataset, described above, produces the datasets that fine-tuning consumes. GaiZhenbiao/ChuanhuChatGPT is a self-hosted Python and Gradio interface that puts hosted and locally deployed models behind one chat window, and its own description lists GPT fine-tuning alongside agents, file-based question answering and web search, so it is a way to reach the capability rather than a training framework. Its last push was 2026-04-30.

In practice

The useful way to hold the term is: more training steps, smaller dataset, same architecture, and a choice about how many weights move. Start by deciding whether you need the model to change at all, because retrieval and prompting are cheaper and easier to reverse; if you do, decide between full fine-tuning, LoRA and QLoRA by your memory and storage budget rather than by accuracy expectations, and then read hiyouga/LlamaFactory or huggingface/peft for the method, metainternal/llama-cookbook for a worked reference, and ConardLi/easy-dataset for the data step that has to happen first.