What is RLHF?

How RLHF works: three models and a loop

The starting point is always supervised fine-tuning. You take a pretrained model, show it examples of good answers, and train it to produce them. What comes out is a model that can imitate a style but has no idea whether one of its own answers is better than another, because nothing in that training ever compared two.

RLHF adds that comparison in two stages. First a reward model is trained on human comparisons. Annotators see two candidate answers to the same prompt and pick the better one; a classifier is then trained on those pairs so that it scores a single answer. Second, that reward model is used as the objective for reinforcement learning on the fine-tuned model, with a penalty pulling the policy back towards the original model so it does not drift into text that scores well but reads badly. In the classic formulation that inner loop is PPO, and the three moving parts are the policy, the reward model and the reference model it is regularised against.

DPO, direct preference optimisation, rearranges the same ingredients. It skips the separate reward model and optimises the policy directly on the preference pairs, using the reference model implicitly in the objective. That removes a training stage and a set of moving parts, which is why it spread quickly, and it changes the failure modes too, because there is no reward model to over-optimise. GRPO and KTO are other variants that appear in the same category: methods that take preference or reward data and produce a better policy without a full PPO loop.

The preference data is the real input

Everything downstream is limited by the comparisons, which is worth stating before talking about algorithms.

A preference dataset is a list of prompts, pairs of candidate answers, and a choice. Building one is a labelling project: write instructions for the annotators, decide whether the pairs are shown side by side or one after another, decide how ties and near-ties are handled, and decide what the model is being asked to be better at. Those decisions are the data. Two datasets over the same prompts with different instructions can produce models with visibly different behaviour.

The reward model inherits all of it. It is trained to predict a human judgement, so it learns the average preference of the annotators, including the ways a crowd differs from a careful expert. It also learns the shortcuts that predict a high score: longer answers, confident phrasing, a particular structure. Once the policy is optimised against that score, the shortcuts stop being tendencies and start being the behaviour.

This is why the reference model exists. The penalty term against the original model is not decoration; it is the constraint that keeps optimisation from wandering into territory the reward model happens to score highly but the task does not want. Remove it, or set it too low, and the usual outcome is a model that has learned to please the classifier.

When you need it, and when you do not

The technique pays for itself when the target behaviour is a preference rather than a fact.

Where it fits. Style and format that demonstrations cannot fully specify, because there are many acceptable answers and a human can rank them faster than they can write them. Refusals and safety behaviour, where the boundary depends on context and a classifier is easier to trust than a rule. Tone and detail control, where the same content is good or bad depending on the reader.

Where it does not. Factual knowledge, which belongs in the data or in retrieval, and which a preference signal handles badly because a confident wrong answer often wins a comparison. Tasks with one correct answer, where supervised fine-tuning on verified examples is strictly better. And projects with no way to collect preferences: without a source of comparisons you have the algorithm and nothing to run it on.

There is a size question too. The full pipeline assumes you can serve a training loop, hold a policy and a reference model in memory at once, and evaluate often. That is a different order of resource than a fine-tuning run, and it is the reason the frameworks below differ so much in what they assume about hardware.

Limits that show up in practice

Five failure modes are common enough to plan for.

Over-optimisation. The policy is pushed further than the reward model can measure, and quality drops while the score rises. The usual mitigation is early stopping on a held-out evaluation, and the reason to keep an evaluation set that the reward model never sees.

Annotator disagreement. When humans agree with each other only partly, the reward model learns a blurred objective, and the ceiling of the method is the consistency of the crowd, not the quality of the method.

Style substituting for substance. Because comparators reward readability, optimised models tend towards longer, more confident answers. A model that scores better on preference data and worse on a task benchmark has not improved.

Distribution shift. The policy drifts towards the region of outputs where the reward model is confident, and that region is narrower than the region the model can actually reach. Prompts unlike the preference set get answers nobody graded.

Cost and reproducibility. Each iteration needs a fresh batch of prompts, a fresh round of comparisons and a training run, and the result depends on a data pipeline that is usually not versioned with the code. Teams that cannot point at the exact comparison set a shipped model came from cannot reproduce it.

How the open-source implementations differ

The projects split by what they are for.

Reference implementations. natolambert/rlhf-book is a textbook plus code: our analysis describes a CC-BY-NC-SA textbook and an MIT-licensed reference implementation library covering SFT, PPO, GRPO, DPO and reward models, and is explicit that it is a teaching resource rather than a training framework. lucidrains/PaLM-rlhf-pytorch is the smallest example of the pattern, wiring a PaLM transformer, a reward model and a PPO trainer into one package, and its README states plainly that it ships no trained weights.

Course material. walkinglabs/modern-llm-notebook is a bilingual, notebook-first course that rebuilds LLM components in PyTorch instead of wrapping a library, with 26 runnable notebooks covering tokenizers, attention, MoE, RLHF, inference, evaluation and distillation. nndl/llm-beginner pairs a 17-chapter Chinese ebook with six Python tasks running from hand-written attention to a coding agent, with self-check scripts that import your implementations by signature. Junvate/LLM-Algorithm-Intern-Guide is a navigation index and set of checklists for Chinese LLM algorithm internship interviews, and our analysis notes its value depends on whether the Feishu wiki it points at is reachable for you.

Frameworks for real runs. modelscope/ms-swift is the ModelScope community's fine-tuning and deployment framework, described as covering continued pretraining, SFT, DPO and GRPO across more than six hundred language models and three hundred multimodal ones, with LoRA, full-parameter and Megatron paths; our analysis says it is broad by design, which is also where its sharp edges are. voidful/TextRL packages GRPO, DPO, KTO and reward-model training behind one config dataclass and one trainer class per algorithm family, and our analysis is clear that it is a convenience layer over HuggingFace TRL rather than a new reinforcement learning implementation. shibing624/MedicalGPT walks a base model through continued pretraining, SFT, reward modeling, RLHF, DPO, ORPO, GRPO and OPD, and ships training scripts rather than a hosted assistant.

Hardware-shaped choices. ARahim3/mlx-tune wraps Apple's MLX in an Unsloth-compatible API, so our analysis says fine-tuning scripts can be written once on a Mac and run unchanged on a CUDA cluster with the original Unsloth, covering SFT, DPO, GRPO, vision, TTS, STT, embedding and OCR work on M1 through M5 hardware. wxhcore/bumblecore takes the opposite route: it builds its own training loop, its own Bumblebee transformer and a DeepSpeed integration instead of wrapping a Trainer class, which our analysis says is aimed at people who want to see every gradient step and costs them the conveniences that wrappers bring.

In practice

RLHF is how you get a model to prefer the right things when there is no single right answer, and the pipeline is only ever as good as the comparisons behind it. Read natolambert/rlhf-book for the mechanism, walkinglabs/modern-llm-notebook for something you can run end to end, and TextRL or ms-swift if you are actually going to fine-tune. Before starting, write down what your annotators will be ranking, keep an evaluation set the reward model never sees, and confirm your framework assumes the hardware you have before you install it.