RLHF Book: the open textbook and code library for post-training LLMs
Textbook on reinforcement learning from human feedback
At a glance
- What is it?
- Nathan Lambert's RLHF Book is a CC-BY-NC-SA textbook plus an MIT-licensed reference implementation library for SFT, PPO, GRPO, DPO and reward models. It is a teaching resource, not a training framework.
- Who is it for?
- Adopt it if you are learning post-training or need a citable reference for a method such as GRPO or rejection sampling, and treat the code/ directory as reading material rather than a training stack.
- Can I use it commercially?
- Check first. The repository uses a licence we do not classify automatically, so read its LICENSE file before any commercial use.
- Is it still maintained?
- Yes. The repository last received commits 7 days ago.
- What is it written in?
- Mainly Python, according to GitHub's language statistics.
Answers come from the project's GitHub data, last synced on September 18, 2026, and from our analysis. They are not legal advice.
DEEP OPEN-SOURCE ANALYSIS
What the RLHF Book repository is for
The README describes the project as a guide to reinforcement learning from human feedback and, more broadly, an introduction to post-training language models. The stated motivation is documentation: the author writes that when he started, methods such as rejection sampling had no canonical reference, and that industry practice around making models more personable, which he calls Character Training, had no open research. The book is the artifact that fills those gaps.
That framing matters for adoption. This is not a library you import to train a model. It is a textbook with a companion code directory, and the repository is the home base for the writing, the figures, the slides and the reference implementations. The audience is engineers and researchers who want the fundamentals plus curated references, and the README is explicit that readers should use coding assistants to ask questions and buy the physical book for the parts that benefit from print.
There is also a second audience the README names directly: contributors. It thanks early readers, offers free copies to substantive contributors, and lists community translations into Simplified Chinese that live in their own repositories rather than being merged here.
How the book, code and diagrams fit together
The repository splits into three trees with different jobs. book/ holds the source: chapters/ contains Markdown such as 01-introduction.md, images/ holds the figures, templates/ holds Pandoc templates for HTML, PDF and EPUB, scripts/ holds build utilities, and data/ holds library data. code/ holds reference implementations, one directory per family of method: instruction_tuning, policy_gradients, reward_models, direct_alignment, rejection_sampling and distillation. diagrams/ holds generation scripts in Python, LaTeX/TikZ sources and YAML specifications.
The build is orchestrated by a root Makefile. It collects chapters with a wildcard over book/chapters/*.md, filters out the README, and pipes the concatenated Markdown through Pandoc with a metadata file, a table of contents at depth 3, a bibliography with citeproc and a CSL style. Math handling differs per output: MathJax is the default for HTML, MathML is used for EPUB for e-reader compatibility, and a Lua filter handles Kindle math. A separate awk rule inlines a footer partial into static HTML pages copied to build/ so that site footers stay in sync with book/templates/footer.html.
The code side is deliberately separate. The root pyproject.toml covers diagram generation and build scripts, with matplotlib, transformers and datasets as dependencies, and a teach extra that tracks the colloquium repository directly from git. The comment in that file notes the git tracking is intentional for catching regressions and suggests pinning to a PyPI release later if reproducible builds become necessary. The README states that code/ has its own pyproject.toml for ML dependencies, which is the file to read before trying to run anything.
Building the RLHF Book locally and a first look at the code
The README gives two build targets. Run these from the repository root, and expect the PDF target to need a LaTeX installation because the README says so explicitly.
make html # Build HTML site
make pdf # Build PDF (requires LaTeX)The output lands in build/, which the repository structure lists as generated and git-ignored. If you only want the figures, the diagrams directory has its own target:
cd diagrams && make allThe README says those diagram sources are designed to be reusable for presentations, blog posts or your own learning materials, which is the most directly practical thing in the repository for someone who teaches.
For the reference implementations, the README points at code/README.md for setup and usage rather than repeating the steps. The code library covers instruction tuning with chat templates, policy gradient methods including PPO, REINFORCE, GRPO and RLOO, reward model training for preference RM, ORM and PRM, direct alignment methods such as DPO and variants, best-of-N rejection sampling, and on-policy distillation under the name SDPO. The code/v0.4 release notes describe instruction tuning, on-policy distillation and config-driven reward models as the changes in that tag.
What the RLHF Book does not give you
The repository is not a training framework, and nothing in the README suggests it wants to be one. There is no documented CLI for launching a training run, no serving path, no checkpoint management and no distributed training story. If you need to fine-tune a model this week, the code here will show you what the algorithm looks like, not run your job.
The build has its own friction. The root pyproject.toml requires Python 3.11 through 3.13, the PDF target depends on LaTeX, and the teach extra installs colloquium straight from a git URL. That last choice is honest about its trade-off in a code comment, but it means the environment is not reproducible by default.
Licensing is the sharpest constraint. The README states that book/chapters is CC-BY-NC-SA-4.0, that everything else including code/, diagrams/ and scripts/ is MIT, and that some images in book/images/ are unlicensed photos or screenshots. The repository's licence field is reported as NOASSERTION, which is consistent with a split-licence project rather than a single licence file at the root. Non-commercial is the operative word for the chapters: a corporate training deck built from the chapter text is a different proposition from one built from the MIT-licensed diagrams.
The README also does not document rollback or versioning policy for the book itself. Releases are tagged in the book/v0.x series, and the most recent push to the repository was on 2026-09-11, but the README says nothing about how a reader should pin to a stable edition.
RLHF Book compared with a training framework such as TRL
The obvious alternative for someone who wants to run RLHF rather than read about it is a training library. Hugging Face's TRL is the common choice, and the difference is structural rather than a matter of quality. TRL ships trainers that wrap a model, a dataset and a configuration into a runnable job. The rlhf-book code/ directory ships reference implementations that exist to make a method legible, with the README pointing readers to code/README.md for setup rather than to a stable API.
That distinction decides most adoption questions. If your goal is a reproducible pipeline with checkpointing and logging, a framework is the right tool and this repository is the wrong one. If your goal is to understand why GRPO differs from PPO, or what rejection sampling actually does to a policy, a framework's source is a harder place to learn that than a book chapter with a small implementation next to it. The two are complements, and the README's own advice to use coding assistants for questions points in the same direction: read here, train elsewhere.
Maintenance, releases and the cost of keeping up
The repository is not archived, and its last push was on 2026-09-11. Recent tags show an active release rhythm: book/v0.12 on 2026-09-11, code/v0.4 on 2026-08-07 and book/v0.11 on 2026-08-03. The code/v0.4 title names instruction tuning, on-policy distillation and config-driven reward models, so the code side is moving alongside the text rather than being frozen.
Upgrade cost depends on which half you use. If you read the book online, there is no upgrade cost beyond re-reading changed chapters, and the release tags give you a way to refer to a specific edition. If you build the book, you are tracking Pandoc templates, a bibliography pipeline and a LaTeX dependency, and those are the parts most likely to break on a new machine. If you vendor code from code/ into your own project, you inherit the maintenance yourself, because the MIT licence permits it and nothing in the README suggests the reference implementations are meant as a dependency.
One README note is worth knowing before you dig through history: the author explains that commits associated with an old Ai2 email address lost most of his tracking in the commit history. The repository layout and release tags are therefore a better guide to what changed than the commit graph.
Editorial conclusion
Adopt it if you are learning post-training or need a citable reference for a method such as GRPO or rejection sampling, and treat the code/ directory as reading material rather than a training stack. Skip it if you need a maintained RLHF training framework with a stable API, or if you need to redistribute the chapters commercially: book/chapters is CC-BY-NC-SA-4.0, only code/, diagrams/ and scripts/ are MIT, and the README warns that some images in book/images/ are unlicensed photos or screenshots. Before you build anything on it, run make html in a clone and confirm that Pandoc and the LaTeX toolchain resolve, then open code/README.md to see what the reference implementations actually require.
Frequently asked questions
Is the RLHF Book free to read, and can I get it as a PDF?
The README links to a free online edition at rlhfbook.com and gives make pdf as a local build target, which requires LaTeX. Print editions are available through Manning and Amazon. The chapter text is licensed CC-BY-NC-SA-4.0, so non-commercial reuse with attribution and share-alike is what the licence permits.
Does the RLHF Book repository include runnable code for GRPO and PPO?
Yes. The code/ directory contains reference implementations grouped by method, including policy_gradients for PPO, REINFORCE, GRPO and RLOO, plus reward_models, direct_alignment, rejection_sampling, instruction_tuning and distillation. The README points to code/README.md for setup and usage, and notes that code/ has its own pyproject.toml for ML dependencies.
What licence applies to the RLHF Book code and chapters?
The README splits them: book/chapters is CC-BY-NC-SA-4.0, while code/, diagrams/, scripts/ and the rest are MIT. It also warns that some images in book/images/ are unlicensed photos or screenshots, so the images are not covered by either licence statement.
Community notes