Practical_RL: a git-course in reinforcement learning that you fork before you finish week one
A course in reinforcement learning in the wild
At a glance
- What is it?
- Yandex Data School's Practical_RL is a Jupyter Notebook course covering tabular CEM through PPO, taught on campus and maintained for online students. Its structure is a strength and its age is a real cost, and both need to be weighed before you start.
- Who is it for?
- Adopt Practical_RL if you want a lab-first RL curriculum with a working Colab path and you are willing to pin old dependency versions or fix notebooks yourself. Do not adopt it as a reference implementation of modern RL, and do not treat the master branch as a maintained library.
- Can I use it commercially?
- Yes. Unlicense is a permissive licence: you can use, modify and sell software built on it, as long as you keep its copyright and licence notices.
- Is it still maintained?
- Yes. The repository last received commits 168 days ago.
- What is it written in?
- Mainly Jupyter Notebook, according to GitHub's language statistics.
Answers come from the project's GitHub data, last synced on September 15, 2026, and from our analysis. They are not legal advice.
DEEP OPEN-SOURCE ANALYSIS
What Practical_RL is trying to fix about RL teaching
Most reinforcement learning material splits into two unsatisfying halves. Papers assume you already know the vocabulary. Textbooks give you the vocabulary but leave you writing a value iteration loop against a toy gridworld you will never touch again. Practical_RL sits in the gap. The repository describes itself as an open course on reinforcement learning in the wild, taught on campus at HSE and the Yandex School of Data Analysis and maintained to be friendly to online students in both English and Russian.
The audience is specific: someone who can already write Python and has seen a neural network trained, but has not internalised why Q-learning and SARSA diverge on a cliff-walking problem, or what a baseline actually removes from a policy gradient estimate. The manifesto in the README states the intent directly: practicality first, with the claim that for every major idea there should be a lab that makes you feel it on a practical problem. That is the whole pitch. You are not buying a reference manual, you are buying a sequence of exercises with the theory attached.
It is not a library. There is nothing to import. The deliverable is your own filled-in notebooks.
The week-by-week path from crossentropy to PPO
The syllabus runs in numbered week directories, each with a lecture topic, a seminar topic and a homework described in that week's README. The README notes the order is approximate and some topics may take two weeks, which is honest but also means you should read the directory listing rather than trust the numbering when planning.
Week one starts with the crossentropy method on Taxi-v0 and then a deep version on box2d environments, which is a deliberate choice: it gets you to a working agent before you have learned any value function. Week two is discounted reward MDPs, value iteration and policy iteration, and the lecture list includes the phrase discounted reward fails, which is the kind of caveat most introductions skip. Week three covers Q-learning, SARSA, off-policy versus on-policy, n-step methods and TD(lambda). Week four is split: a deep learning recap that walks through PyTorch or TensorFlow and a simple convnet classifier, then approximate RL with experience replay, target networks and the double/dueling/bootstrap DQN variants on CartPole and Atari.
From there the course moves to exploration and contextual bandits with Thompson sampling and UCB, then policy gradients with the log-derivative trick, REINFORCE and advantage actor-critic including GAE. Week seven is sequence models and recurrent networks, week eight is partially observed MDPs with recurrent A3C and DRQN, week nine is TRPO and PPO plus deterministic policy gradient and DDPG, and week ten is model-based RL, planning, imitation learning and inverse RL, with an extra directory for material the course does not formally cover. That is a broad sweep for a course, and the depth per topic is correspondingly shallow. Expect to follow the linked external material if you want to implement anything beyond the seminar version.
How the course is actually structured as a repository
Everything is Jupyter Notebook. The primary language field on the repository is Jupyter Notebook, and the seminar and homework work happens in .ipynb files inside the week directories. The README gives the Colab entry path explicitly: open Colab, choose open, then github, then yandexdataschool/practical_rl, then pick a branch name, then select the notebook you want. Note the lowercase spelling in that instruction; the repository itself is Practical_RL with capitals, and Colab's GitHub picker is generally tolerant of the difference but it is worth knowing which string the README uses.
There is no package to install from the repository itself and no CLI. Dependencies are handled by installing them into your own environment, and the README points to a pinned issue thread, issue 1, for local installation instructions rather than embedding them. That thread is also the technical issues thread, so it doubles as the place where environment breakage gets reported. The README also lists Azure Notebooks as an alternative to Colab, though that option is presented without further detail.
The contribution model is stated as part of the manifesto: the course calls itself a git-course and invites pull requests for typos, formula fixes, useful links, readability improvements and ports to alternative frameworks. Several TensorFlow assignments are credited to a contributor, and the README acknowledges fixes from named individuals and from the Berkeley AI course and CS294 for material and pictures. So the repository is a teaching artifact that expects to be edited by its readers, not consumed as a frozen product.
Getting a notebook running, and where that breaks
The lowest-friction path is Colab. Follow the README's sequence: open Colab, choose the GitHub tab, enter yandexdataschool/practical_rl, select a branch, then open the notebook for the week you want. The branch name matters because the repository ships releases as branches: spring20 is the Spring 2020 release and spring19 is the Spring 2019 release, per the release list. Selecting master gets you whatever has landed since, which may be ahead of the recorded release but is also less predictable.
The local path means installing dependencies yourself. The README does not list them inline; it routes you to the technical issues thread at issue 1. That is a real constraint. You will be reading a discussion thread to reconstruct an environment, and the thread spans years of breakage reports, so expect to filter. There is no requirements file or environment spec mentioned in the supplied material, and no documented Python version. If you need reproducibility for a cohort, you are building that yourself.
Once an environment exists, the loop is ordinary notebook work: run cells, train a small agent, read the homework description in the week README, produce whatever artifact it asks for. The course staff list includes several people credited with homework checkups, which tells you the original delivery model assumed human grading on campus. Online students get the notebooks and the wiki, and the README points to an online student survival guide and a lecture slide archive hosted externally.
The maintenance problem hiding behind an Unlicense
The repository is licensed under the Unlicense, which places the work in the public domain. That is maximally permissive: you can copy the notebooks into a company training program, modify them, ship them internally, and the licence does not ask anything of you. It also means there is no warranty, no support obligation, and no maintainer accountable to you. The Unlicense removes friction and removes recourse at the same time. If you need contractual clarity about third-party material, note that the README credits pictures from the Berkeley AI course and references to CS294, and those upstream sources carry their own terms; the repository's Unlicense does not automatically relicense someone else's slides. That is a question for your own legal review, not something this article can settle.
On maintenance: the last push recorded is 2026-03-31, but the most recent release is spring20 from August 2020, with spring19 before it. The gap between a recent commit and a six-year-old release is the thing to understand. Commits may be typo fixes, link repairs or small notebook patches. The course content itself has a Spring 2020 vintage. Reinforcement learning moved considerably after that, and any notebook depending on an old gym API or an old PyTorch or TensorFlow version is a candidate for breakage. The topics list mentions both Keras and TensorFlow alongside PyTorch, which reflects a course that has carried multiple framework variants rather than standardised on one.
Upgrade cost, then, is not a version bump you perform once. It is per-notebook work: install the environment, run the notebook, find the cell that fails, and patch it. Budget for that, and expect the patches to be yours to maintain.
When Practical_RL is the wrong tool
If you want to train something that works on a real problem, this is the wrong repository. The labs are deliberately small: Taxi-v0, CartPole, box2d, a simple robot control task, toy tasks for MCTS. There is no production training infrastructure, no distributed runner, no evaluation harness, no checkpointing story. A team looking for a starting codebase for a deployed agent will find the notebooks instructive and useless as a foundation.
If you need current methods, also look elsewhere. The recorded releases stop at Spring 2020. Offline RL, modern transformer-based sequence modelling and the post-PPO literature are not in the syllabus as described. Week nine ends at TRPO, PPO and DDPG. The yet_another_week directory exists precisely because the course acknowledges material it does not cover, and the README labels it all that cool RL stuff that you won't learn from this course.
And if you need a stable API to build on, the repository does not offer one. It is a set of notebooks you edit. There is no versioned interface, no changelog beyond the two release branches, and no deprecation policy. Treating it as a dependency rather than as reading material is a category error.
What a course is not: comparing it to Spinning Up and to a library
The obvious comparison is OpenAI Spinning Up, which is also educational but takes the opposite approach. Spinning Up ships a small, installable package of reference implementations of the core algorithms with a consistent interface, plus written explanations, so you can read the algorithm and then call it. Practical_RL ships notebooks where you write the algorithm yourself, week by week, in a different style each time, and never get a reusable import out of it. Spinning Up optimises for a clean reference you can cite and run; Practical_RL optimises for the moment you watch your own Q-learning agent stop falling off the cliff. If you learn by typing, the notebook route sticks better. If you learn by reading a tight implementation and then varying it, the package route is faster.
The second comparison is to a library like Stable-Baselines3, which is a different category entirely. There you get maintained, tested implementations of PPO, SAC, DQN and others behind a stable API, and you spend your time on environments and reward design. Practical_RL gives you none of that, but it explains what is inside those implementations, which is exactly the gap Stable-Baselines3 leaves. A reasonable sequence is to work through the relevant Practical_RL weeks and then use a library for real training, rather than picking one and treating it as the whole education.
The honest framing: Practical_RL competes with other courses, not with other software.
Who should start, who should wait, and what to check first
Start here if you are an engineer or graduate student with working Python and some deep learning background, you want the intuition behind value-based and policy-based methods rather than a survey of the literature, and you are comfortable debugging a notebook when a cell fails. The Colab path lowers the barrier enough that you can evaluate the course in an afternoon: pick week one, open it on the spring20 branch, and see whether the environment installs and the first agent trains. That single test tells you more than any review, including this one.
Wait, or look elsewhere, if you need current algorithms, production training code, a stable API, or a documented environment you can reproduce without reading an issue thread. The Spring 2020 release date and the absence of an inline dependency list are the two facts that decide most adoption questions here.
Verify three things before you commit a cohort to it. First, that the specific week you need runs on the branch you selected, since master and spring20 can differ. Second, that the dependency installation described in issue 1 still resolves on your platform, because the README delegates that step rather than specifying it. Third, that the licence position on the credited third-party material (the Berkeley AI course pictures, the CS294 references) is acceptable to whoever signs off on your training material. The Unlicense covers the repository's own content; it does not answer that question for you.
Editorial conclusion
Adopt Practical_RL if you want a lab-first RL curriculum with a working Colab path and you are willing to pin old dependency versions or fix notebooks yourself. Do not adopt it as a reference implementation of modern RL, and do not treat the master branch as a maintained library. Before committing, open the week you actually care about on Colab with the spring20 branch, confirm the environment installs, and read that week's README to see what the homework expects you to produce.
Community notes