drl-zh: A Build-It-Yourself Deep RL Course, From Tabular Q-Learning to Dreamer
Deep Reinforcement Learning: Zero to Hero!
At a glance
- What is it?
- alessiodm/drl-zh is an MIT-licensed Jupyter Notebook curriculum that hands you 19 numbered notebooks with the algorithm code removed and replaced by TODO cells, plus a parallel solution track. It is a teaching repository, not a library, and the Docker workspace is the recommended way in.
- Who is it for?
- Adopt drl-zh if you already write PyTorch and want to implement DQN, PPO, SAC, MCTS, RLHF and Dreamer yourself rather than call someone else's trainer. Skip it if you need a library to ship a policy this quarter, or if you cannot read training code, because the notebooks assume exactly that.
- Can I use it commercially?
- Yes. MIT is a permissive licence: you can use, modify and sell software built on it, as long as you keep its copyright and licence notices.
- Is it still maintained?
- Yes. The repository last received commits 112 days ago.
- What is it written in?
- Mainly Jupyter Notebook, according to GitHub's language statistics.
Answers come from the project's GitHub data, last synced on September 15, 2026, and from our analysis. They are not legal advice.
DEEP OPEN-SOURCE ANALYSIS
The gap drl-zh fills: algorithms you have read about versus algorithms you have written
Most reinforcement learning material splits into two unsatisfying halves. Papers give you the update rule and skip the plumbing. Libraries give you the plumbing and hide the update rule behind an abstraction you never open. drl-zh takes the third position: the repository ships complete notebooks with the interesting code deleted. The README describes the root notebooks as "the exercise track: code is intentionally replaced with guided TODO sections," and the solution/ directory holds "the complete, runnable versions." That structure is the entire product. You are not reading about DQN; you are reconstructing the loss, the target network update and the replay sampling until the agent learns.
The intended reader is stated plainly in the prerequisites: comfortable with Python, PyTorch basics, and probability, statistics, linear algebra and derivatives. The README adds a blunt sentence that matters more than the math list: "The notebooks teach the RL, but they assume you can read and modify real training code." If that sentence describes you, the curriculum runs from MDPs and tabular methods through DQN, REINFORCE, actor-critic, DDPG, TD3, SAC and PPO, then into RND curiosity, multi-agent RL, offline RL with BC and IQL, Monte Carlo Tree Search with AlphaZero-style policy and value learning, RLHF with PPO, DPO and GRPO, Decision Transformers, a NanoVLA notebook, TensorBoard and checkpointing, MBPO, DR3AM/Dreamer with an RSSM, and finally MAML and FOMAML. That is a wide sweep for a single course, and it is the reason the repository is worth a serious look rather than a bookmark.
Two parallel notebook trees and the TODO scaffold
The architecture is a directory convention, not a framework. Root notebooks 00 through 18 form the exercise track. A solution/ directory mirrors them with the code filled in. Nothing in the README suggests a test harness that verifies your TODO is correct, so the feedback loop is the training curve you produce and the solution notebook you compare against when it flatlines. That is a real design choice with a real cost: there is no automated grader, and a subtly wrong advantage estimate can still produce a curve that looks plausible.
The curriculum table groups the notebooks into six tracks. Foundations (00 to 07) are meant to be done in order. The README says the advanced notebooks are self-contained, with numbering as "a good default path from exploration to the course capstone." The production notebook (14) is where TensorBoard, checkpointing, debugging, multiple seeds, Ray and Optuna appear together, which is the point at which the course stops being about single-run demos. Notebooks 15 and 16 split world models into MBPO with SAC and then DR3AM/Dreamer with RSSM latent imagination. Notebooks 17 and 18 cover MAML and FOMAML and close the course.
The naming is worth noting for anyone planning a study schedule. DDPG, TD3 and SAC land in the same foundation block as PPO, which means continuous control arrives before the course has covered offline data or planning. That ordering is defensible for a build-it-yourself course because the three off-policy continuous methods share replay and target-network machinery, but it does mean you meet SAC before you meet the offline RL notebook that would explain what to do when you cannot collect more data.
Getting the workspace running: Docker, .env, and the kernel name
The README recommends Docker and describes the result as code-server, the notebooks, Python >=3.13,<3.14, the Jupyter kernel, dependencies and the AI Companion in one workspace. The sequence is short. Clone the repository and cd into it. On Linux or macOS, run:
printf "UID=$(id -u)\nGID=$(id -g)\n" > .env
so that files created inside the container are owned by your host user. Then:
docker compose up --build -d
Open http://localhost:8080 in a Chromium-based browser, select the Python (drl-zh) kernel, and open 00_Intro.ipynb. The kernel name is not optional trivia; picking the wrong kernel is the most likely reason a notebook fails to import anything.
Two overlay files exist for hardware variants. GPU access uses docker compose -f docker-compose.yml -f docker-compose.gpu.yml up --build -d. A smaller CPU-only image uses docker compose -f docker-compose.yml -f docker-compose.cpu.yml up --build -d. The README also points to MANUAL.md for a native setup covering Python, Poetry, VS Code and the Companion, so Docker is the recommended path rather than the only one.
The .env step is the kind of detail that reveals the project has been used on real machines. Without it, notebooks written inside the container land as root-owned files on the host, and the fix is annoying enough that people abandon a course over it. That the README leads with the workaround suggests the maintainer hit it too.
The AI Companion and its bring-your-own-key constraint
The Docker workspace includes a VS Code extension the README calls the DRL-ZH AI Companion. Its stated behaviour: it knows which notebook and which TODO you are working on, offers Socratic hints rather than spoilers, and supports text or voice mode. You supply your own LLM key. Gemini is the default, with OpenAI, Anthropic and Groq supported.
This is the part of the project most likely to age badly, and the README gives no detail on how the extension detects your current TODO, how hints are scoped, or what happens when the hint engine and the solution disagree. Treat it as a convenience layered on top of the course rather than the reason to take it. The bring-your-own-key model also means the course has an external dependency that is not covered by the MIT licence: your provider's terms, rate limits and costs apply to every hint you request. Anyone evaluating drl-zh for a classroom should decide up front whether students bring keys or whether the Companion is disabled and the solution notebooks are the only safety net.
Where drl-zh is the wrong tool
This is a course, not a library. There is no package to install, no import drlzh, no trainer you can point at an environment. If your actual problem is getting a working policy into production, the notebooks will teach you what PPO does but will not run your environment. The README never claims otherwise, but the wide curriculum can create the impression that finishing notebook 18 leaves you with a deployable stack. It leaves you with implementations you wrote and can debug, which is a different and narrower asset.
The deeper limitation is hardware and time. Notebooks 11, 12, 13, 15 and 16 cover AlphaZero-style self-play, RLHF with PPO, DPO and GRPO, a NanoVLA notebook, MBPO and Dreamer. Those are the sections where a CPU-only image will be painful, and the README's CPU overlay is described only as "a smaller CPU-only image," with no statement about which notebooks remain practical on it. If you are evaluating drl-zh on a laptop without an NVIDIA GPU, that is the first thing to establish, and the README does not establish it for you.
There is also no stated grading or verification mechanism. For a self-directed learner that is fine. For a course with assessment, the absence of any check beyond comparing against solution/ is a gap you would have to fill yourself.
How it differs from a framework like Stable-Baselines3
The obvious alternative is a batteries-included RL library such as Stable-Baselines3, where PPO, SAC, TD3 and DQN are importable classes with documented hyperparameters and a common vectorised-environment interface. The difference in approach is stark. Stable-Baselines3 optimises for getting a trained agent with minimal code you have to understand. drl-zh optimises for the opposite: the code is deliberately absent so that you write it.
That means the two are not substitutes. If you want to know whether PPO or SAC suits a continuous-control task, a library answers that in an afternoon. If you want to know why your PPO entropy coefficient is collapsing the policy, the library hands you a config key and drl-zh hands you a notebook where you wrote the entropy term. The trade is time and correctness assurance. A library has been exercised by many users on many environments; your TODO implementation has been exercised by you on one. Choose accordingly, and note that nothing stops you from doing both: write it in drl-zh, then compare your curve against a library baseline on the same environment. The README does not propose that workflow, but the solution/ notebooks make it possible.
Licence, releases and the cost of keeping up
The repository is MIT licensed, which permits commercial and private use, modification and redistribution provided the copyright notice and permission notice are preserved. That is a permissive, low-friction choice for a teaching repository, and it means you can lift a solution notebook into internal training material. It does not grant rights to any third-party model or API you connect the Companion to, and it says nothing about the environments the notebooks use. This is a description of the licence text, not legal advice; read LICENSE and your own obligations.
The release history shows three tags: v0.1.0 (Classic methods), v1.0.0 (Second Edition w/ Advanced Topics) and v1.5.0 (Third Edition: Modern Techniques), the last pushed alongside the repository's most recent commit. The edition framing tells you the curriculum is revised in whole-number passes rather than continuously, so a notebook you annotate today may be restructured at the next edition. Pinning to a tag rather than tracking main is the cheap hedge.
Maintenance cost for you is mostly environment drift. The README pins Python >=3.13,<3.14 and routes everything through Docker, which is the right call for a course with this many dependencies, but the GPU and CPU compose overlays are the only supported hardware paths described, and the AI Companion adds an external API dependency with its own lifecycle. Budget for a Docker rebuild when you return to the course after a few months, and expect the Companion to be the piece most likely to need reconfiguration.
Editorial conclusion
Adopt drl-zh if you already write PyTorch and want to implement DQN, PPO, SAC, MCTS, RLHF and Dreamer yourself rather than call someone else's trainer. Skip it if you need a library to ship a policy this quarter, or if you cannot read training code, because the notebooks assume exactly that. Before committing, open 00_Intro.ipynb in the Docker workspace and confirm the Python (drl-zh) kernel starts and that the TODO-to-solution diff is the depth you want; then check whether MANUAL.md or the Docker path fits your GPU, since the two compose overlay files are the only supported GPU route described.
Community notes