hands-on-modern-rl: from CartPole PPO to GRPO and agentic RL labs
🚀 An open-source, hands-on curriculum bridging the gap from basic RL concepts to LLM alignment, RLVR, and advanced Agentic systems.
At a glance
- What is it?
- A free VitePress textbook from walkinglabs that teaches reinforcement learning in two tracks, classical control through CartPole, DQN, PPO and A2C, then RLHF, DPO, GRPO and veRL code RLVR for language models, with eight runnable experiments and a self hosted site you build with npm.
- Who is it for?
- Treat the book as a course to work through, not a library to depend on: the labs are unversioned per script, the root carries no Python dependency manifest, and the maintainers themselves say the text has not been fully reviewed. Read LICENSE before you reuse any lab code, pin the commit you used, and check each chapter against the current chapter code before trusting a snippet in the text.
- Can I use it commercially?
- Check first. The repository uses a licence we do not classify automatically, so read its LICENSE file before any commercial use.
- Is it still maintained?
- Yes. The repository last received commits 24 days ago.
- What is it written in?
- Mainly Python, according to GitHub's language statistics.
Answers come from the project's GitHub data, last synced on November 15, 2026, and from our analysis. They are not legal advice.
Star history
- walkinglabs/hands-on-modern-rl4,916 ★ · +579
| Repository | 2026-09-10 | 2026-11-15 |
|---|---|---|
| walkinglabs/hands-on-modern-rl | 4.3k | 4.9k |
579 Stars in the last 66 days, measured from the daily GitHub snapshot. Compare with other projects
Editorial analysis
CartPole to agentic RL: the book runs two routes through 26 chapters
The book splits into two routes. Chapters 1 to 12 are the classical and deep RL track, and they open on the CartPole loop: choose an action, observe the next state, collect a reward. From that single interaction it builds MDPs and value functions, then implements DQN, policy gradients, Actor-Critic, and PPO in games and control tasks. Offline RL, imitation learning, exploration, and world models extend those foundations.
Chapters 13 to 26 apply the same policy foundations to language models that already exist, through RLHF, DPO, GRPO, and verifiable rewards. The agentic chapters take a model past answering once: calling tools, browsing, writing code, finishing multi step tasks. Those decisions are trained with feedback from entire trajectories rather than from one turn at a time. Later chapters cover visual and embodied tasks, training systems, and evaluation.
The text is published at https://walkinglabs.github.io/hands-on-modern-rl/, with README.md holding the English entry point and README.zh.md the Chinese one.
Eight experiments, with the full list filed under code/README.md
Eight examples are called out: four environment based experiments, then preference tuning, code RL, research agents, and visual reasoning. Six of them are named with their file paths, and the paths themselves tell you how the chapters map to code, since the chapter folder is part of the path.
| Experiment | Code or walkthrough | | --- | --- | | CartPole | [PPO training and environment replay](code/chapter01_cartpole/1-ppo_cartpole.py) | | Atari Pong | [DQN training from image observations](code/chapter04_dqn/dqn_atari_sb3.py) | | LunarLander | [DQN training and landing evaluation](code/chapter04_dqn/dqn_gym_sb3.py) | | BipedalWalker | [A2C training for continuous control](code/chapter06_actor_critic/actor_critic_bipedalwalker.py) | | DPO | [Prepare preferences, fine-tune, and test responses](code/chapter17_dpo/) | | Code RLVR | [veRL PPO with executable I/O test rewards](code/chapter18_grpo/verl_code_rlvr/) |
Notice the two shapes. CartPole, Atari Pong, LunarLander, and BipedalWalker point at one Python file each, so a reader can open the exact script a chapter explains. DPO and Code RLVR point at directories, `code/chapter17_dpo/` and `code/chapter18_grpo/verl_code_rlvr/`, which means multi file setups with their own data and config. The full collection is indexed in `code/README.md`. For the CartPole experiment there is also a hosted path: a ModelScope studio named walkinglab/hands-on-modern-rl-experiment01-cartpole, a shared notebook, and a `train.py` script viewable in that studio.
The maintainers label the course not yet fully reviewed
Two notes in the repository set expectations before you run anything. The first says the course was created with AI assistance and has not yet been fully reviewed, and that it may contain factual mistakes or code that does not run as expected, with issues and pull requests welcome. The second is an announcement saying a new version is coming soon and that many sections are still being organized and refined.
That matters in practice. The 2026-08-19 entry says a collection of online reinforcement learning environments, training scripts, and companion notebooks was added and refined over two weeks, that learners can run classic experiments online and inspect training logs and evaluation results, and that many previously reported bugs in the course content, links, and experiment code were fixed. A book that admits unreviewed text and simultaneously reports a bug fixing pass is telling you both things at once: the prose may lag the scripts, and the scripts moved recently.
For a reader, the consequence is that a snippet failing in a chapter is an expected outcome, not evidence that you misconfigured something. Expect to compare the text against the current file in `code/`, and expect chapter numbering to drift as sections get reorganized before the next version lands.
npm scripts drive the book: VitePress dev, PDF, EPUB and a sitemap
The site is a VitePress project, not a Python package. `package.json` is named hands-on-modern-rl at version 1.0.0, declares `type: module`, and requires Node `>=18.0.0`. Clone the repository and start the dev server directly:
git clone https://github.com/walkinglabs/hands-on-modern-rl
cd hands-on-modern-rl
npm install
npm run dev:fast`npm run dev:fast` runs `vitepress dev docs` alone, while `npm run dev` first pipes through `node scripts/optimize-assets.mjs`, so the plain dev start costs you an asset pass you can skip while writing. Building is heavier: `npm run build` first runs `npm run verify:notebooks` (four Python notebook checks), then the asset optimizer, then `node scripts/generate-sitemap.mjs`, then `vitepress build docs`, and `npm run build:force` appends `--force`. Book formats have their own entry points, `npm run pdf:build` calls `node scripts/build-pdf.mjs` and `npm run epub:build` calls `node scripts/build-epub.mjs`, with `puppeteer-core` at `^24.42.0` in devDependencies.
Three checks guard the prose itself:
npm run verify
npm run verify:bilingual
npm run verify:english`verify` runs `bash scripts/verify.sh`, `verify:bilingual` runs `node scripts/check-bilingual-sync.mjs`, and `verify:english` runs `node scripts/check-english-quality.mjs`. `vercel.json` sits at the root, and `npm run lint` points eslint at `docs/.vitepress/theme`.
The license is stated twice and the two statements do not agree
Two license facts sit next to each other and they do not match. Repository metadata records the license as NOASSERTION, which means no recognized license identifier was detected there. `package.json` states `"license": "CC-BY-NC-SA-4.0"`, and a `LICENSE` file sits at the root, linked from the header of the README.
CC-BY-NC-SA-4.0 is a real constraint for a code repository. It allows reuse with attribution and share alike, and the NC clause blocks commercial use, which a permissive identifier like MIT or Apache-2.0 would not. The releases follow the same naming, each tag is titled Hands-On Modern RL Open Textbook, the most recent being v0.2.1 on 2026-06-18, with v0.1.6 and v0.1.5 both published on 2026-05-15.
What a reader cannot do is guess. If you plan to ship a lab script inside a product, a paid course, or a commercial model training stack, the NOASSERTION field gives you no permission, and the share alike and non commercial terms in `package.json` give you permission only within limits. Open `LICENSE` and read it before copying anything out of `code/`, especially the veRL and DPO directories, since those are the labs closest to a production training setup.
No Python manifest at the root, so every lab brings its own environment
The root listing explains the gap. It holds `.gitattributes`, `.github/`, `.gitignore`, `.prettierignore`, `.prettierrc`, `.zcode/`, `AGENTS.md`, `LICENSE`, `README.md`, `README.zh.md`, `code/`, `docs/`, `eslint.config.js`, `modelscope-space/`, `notebooks/`, `package-lock.json`, `package.json`, `planning/`, `restructure.py`, `scripts/`, and `vercel.json`. There is no `pyproject.toml`, no `requirements.txt`, and no `setup.py` at that level, even though Python is the primary language.
So the dependency manifest you can see, `package.json`, covers only the documentation toolchain: katex, lucide-vue-next, markdown-it-container, markdown-it-footnote, plus prettier, eslint, gray-matter, markdown-it, archiver, and puppeteer-core as dev tooling. Nothing at the root pins a version for a gym environment, for veRL, or for rLLM. Those versions live wherever each chapter keeps them, inside `code/chapter...` folders or `notebooks/`, and there is no root command that assembles them.
The consequence is concrete for anyone planning to reproduce a lab: no root manifest tells you which environment version a chapter was written against. Read the chapter's own files before you trust a lab, and keep per lab environments separate so one chapter's dependency change cannot break another.
Three textbook tags and no changelog for the lab scripts
Release history covers the book, not the experiments. Three tags exist: v0.2.1 on 2026-06-18, v0.1.6 on 2026-05-15, and v0.1.5 on 2026-05-15, all named after the open textbook. The root listing has no `CHANGELOG` file, and the 2026-05-02 entry calls the initial release an open source release for testing and feedback rather than a stable cut. The 2026-05-15 entry announces the complete English translation of all chapters with PDF builds for both the Chinese and English editions released automatically via CI, and the 2026-05-13 entry adds reproducible training examples for Agentic RL with Deep Research and rLLM, for Traditional RL with Actor-Critic continuous control, and new VLM RL experiments on GeoQA geometry reasoning.
None of those entries lists which lab file changed. The 2026-08-19 entry says many previously reported bugs were fixed in the content, links, and experiment code, without naming them. The repository is not archived and the last push landed on 2026-10-23, so work is still landing, and that is exactly the problem for a learner: if a script behaves differently from the chapter text, there is no release note that tells you which revision you had. Pin the commit you read, and treat the announcement about an upcoming version as a reason to re diff the `code/` folder rather than as a promise of a migration guide.
Editorial conclusion
Treat the book as a course to work through, not a library to depend on: the labs are unversioned per script, the root carries no Python dependency manifest, and the maintainers themselves say the text has not been fully reviewed. Read LICENSE before you reuse any lab code, pin the commit you used, and check each chapter against the current chapter code before trusting a snippet in the text.
Frequently asked questions
Is RL a dead end?
The course treats it as an active line of work, running 26 chapters from Markov decision processes and policy optimization through RLHF, DPO, GRPO, and verifiable rewards. The repository is not archived and the last push landed on 2026-10-23, with an announcement that a new version is being organized.
Do AI agents use RL?
Chapters 13 to 26 cover agentic AI, where a model moves past answering once to calling tools, browsing, writing code, and completing multi step tasks, and the agentic RL chapters train those decisions with feedback from entire trajectories. The Agentic RL lab is built around Deep Research and rLLM, and experiment code for these chapters sits under code/.
What is the most popular reinforcement learning algorithm?
The book makes no popularity ranking. It implements PPO for CartPole in code/chapter01_cartpole/1-ppo_cartpole.py, DQN for Atari Pong and LunarLander, A2C for BipedalWalker, and veRL PPO with executable I/O test rewards in code/chapter18_grpo/verl_code_rlvr/.
What is a real life example of reinforcement?
The course starts from the CartPole loop of choosing an action, observing the next state, and collecting a reward, then moves to Atari Pong from image observations, LunarLander landing evaluation, and BipedalWalker continuous control. Those four environment experiments are the first half of the eight examples listed in code/README.md.
Official sources
Add this badge to your README
If you maintain this project, the badge below links readers to this analysis and shows its maintenance status from the daily GitHub snapshot. Paste the markdown into your README; add ?metric=license or ?metric=stars to the image URL for a different field.
[](https://hysenlabs.com/projects/walkinglabs-hands-on-modern-rl)Star history chart for your README
[](https://hysenlabs.com/projects/walkinglabs-hands-on-modern-rl)Was this useful?