rlcode/reinforcement-learning: nine single-file RL algorithms, now on PyTorch 2.11 and gymnasium

One file per algorithm, and that constraint drives everything else

The README states the organising rule directly: "One file for each algorithm." Nine algorithms are listed, grouped by environment. Grid World holds policy iteration, value iteration, SARSA, Q-learning, deep SARSA and REINFORCE. CartPole holds DQN, A2C and PPO. Atari holds DQN and PPO. The README also notes a layout change from the 2017 original: files are now flat, so `1-grid-world/3-sarsa.py` replaces the older nested `1-grid-world/4-sarsa/sarsa_agent.py` structure. That is the whole design argument. An agent class split across a folder is harder to read end to end than a script you can scroll through, and the repository has chosen readability over reuse. The audience follows from that: people who want to see the update rule next to the loop that applies it, and people who want a working baseline they can modify in place. It is not aimed at anyone who needs a training harness, checkpoint management across many runs, or a config system. The README says each algorithm file opens with a paper citation and the core update equation, which is a documentation choice consistent with the same goal.

What the 2017 to 2026 migration actually changed

The Updates section lists six changes. The framework moved from Keras with TensorFlow 1.0 to PyTorch 2.11. The environment interface moved from gym 0.8 to gymnasium 1.2. Rendering moved from tkinter to pygame, which the README justifies as cross-platform with no system Tk dependency. Tooling moved from `requirements.txt` to `pyproject.toml` plus `uv`. Scope was pruned to nine core algorithms, with Monte Carlo, DDQN, A3C, Atari and mountaincar dropped and PPO added. Layout was flattened, and docs gained the citation and update-equation headers. Two of these matter more than the rest. The gym to gymnasium move is not a rename: gymnasium changed the step API to return a five-tuple with truncation separate from termination, and any code copied from a pre-2022 tutorial will need that adjustment. The tkinter to pygame move removes a dependency on a system Tk install, which is the kind of thing that breaks a demo on a headless machine or a fresh container. The pruning is a real loss for anyone who came for A3C, which is still listed in the repository topics even though the README says it was dropped.

Getting it running: uv sync, then one script per algorithm

The README states the requirement plainly: Python 3.11 and uv. Setup is three commands. Clone the repository, `cd reinforcement-learning`, then `uv sync`. Running is a directory change plus a script name. For Grid World: `cd 1-grid-world && uv run python 3-sarsa.py`. For CartPole training: `cd 2-cartpole && uv run python 1-dqn.py`. Two flags are documented for the CartPole DQN script. `--render` watches training and the README warns it is slower. `--test` replays a trained checkpoint. Atari scripts take an environment flag and an optional logging flag: `cd 3-atari && uv run python 2-ppo.py --env breakout --wandb`, and the same form with `1-dqn.py`. The W&B path needs a one-time `uv run wandb login` with an API key from the wandb authorize page. The README is explicit that runs land in your own `rl-atari-ppo` or `rl-atari-dqn` project and that nothing is shared by default, and that omitting `--wandb` means the script never touches the network. That last detail is the one I would check first in the source, because it is the difference between a script you can run on an air-gapped box and one you cannot.

The benchmark tables are unusually candid, and also unusually narrow

The README publishes numbers on named hardware: a MacBook Pro 14-inch with an M3 and 8 GB unified memory, macOS 26.2, Python 3.11, PyTorch 2.11 with the MPS backend. For Breakout at 10M agent steps with sticky actions, DQN is listed at roughly 9 hours, a final mean of 93.5 with a standard deviation of 9.6, 5.27 GB peak RAM, and 1.69M parameters. PPO is listed at roughly 3.8 hours, 261.9 with a standard deviation of 6.4, 1.98 GB peak RAM, and the same parameter count. The README states these are single-seed rows, with the mean and standard deviation taken over the final 20 logged episodes, and that CPU and GPU percentages come from Activity Monitor on the `python3.11` process after about five minutes of stabilisation. It also warns that sticky actions with `repeat_action_probability=0.25` make absolute scores lower than the deterministic `*-v4` environments cited in older papers. That warning is the most useful sentence in the section. A single seed on a laptop tells you the script runs and roughly what it costs in time and memory. It does not tell you the algorithm's variance, and the README does not claim otherwise.

Montezuma's Revenge: three scripts, three different things being measured

The hard-Atari section is where the repository is most careful and most confusing at once. It lists three methods on a Mac Studio with an M4 Max and 64 GB, and states up front that the protocols are not cross-comparable. PPO with RND in `4-atari-hard/1-ppo-rnd.py` uses sticky actions and an RL policy, reaches roughly 3,120 over 65M frames on a single seed. Go-Explore exploration in `2-go-explore.py` uses deterministic restore-based search, reports 31,000 described as replay-verified over 500M frames, and the README labels it "a search result, not an RL score". The robustification script `3-robustify.py` uses sticky actions and an RL policy over 5M frames and has no from-reset score at all. The notes add that RND got its first key at about 327k steps with 512 environments, that 128 environments never scored in 50M steps, and that the curriculum in the robustification script plateaus around 22 percent on one machine against the original work's hundreds to thousands of environments. Read those three rows as three separate claims. Only the first is a policy you can deploy from a reset.

Where this repository is the wrong tool

The single-file constraint is also the failure mode. There is no shared agent interface, no config layer, and no experiment runner, so comparing two algorithms means editing two scripts. The README's own numbers show the cost of running this way: single-seed results on laptop hardware, with the Atari runs taking hours each. If you need a result with error bars, this repository gives you the code to start from and nothing else. Two further limits are visible in the material. First, the pruning removed A3C, DDQN, Monte Carlo and mountaincar, so anyone following a 2017 tutorial that references those files will not find them, despite the topics list still naming a3c. Second, the environment coverage is three families: a grid world, CartPole, and Atari. Continuous control is absent. If your problem is a robotic arm or a locomotion task, nothing here transfers without writing the environment wrapper and the action distribution yourself. The Go-Explore scripts are also a different category of artefact from the rest: they are search procedures with an archive, not policy gradient code, and treating them as interchangeable with the PPO scripts will produce a meaningless comparison.

Compared with Stable-Baselines3, the difference is what gets hidden

Stable-Baselines3 is the obvious alternative for anyone who wants to train a policy rather than read one. It ships PPO, A2C, DQN and others behind a common `learn` and `predict` interface, with vectorised environments, evaluation callbacks and logging handled for you. The trade is that the update rule lives inside a library module rather than in the file you opened. This repository makes the opposite trade: you see the loss, the advantage estimate and the optimiser step in the same file as the rollout loop, and you pay for it by writing your own evaluation, seeding and checkpointing. Neither choice is wrong. If your goal is a trained policy on a standard benchmark, Stable-Baselines3 gets you there with fewer lines of your own code. If your goal is to understand why PPO clips the ratio, or to modify that clipping term, the single-file form is the one that lets you do it without reading a library's internals. The README's own framing supports this reading: it calls the examples "easy-to-read" rather than production-ready.

Licence, maintenance and what to check before you build on it

The repository is MIT licensed, which permits commercial use, modification and redistribution provided the copyright notice and permission notice are retained. That is a permissive licence and it is compatible with most downstream work, but I am not giving legal advice and you should confirm the terms yourself if you plan to ship derived code. On maintenance, the material shows a last push of 2026-06-12 and no releases retrieved, so there is no tagged version to pin against. That matters more than usual here, because the project depends on a specific pairing of PyTorch 2.11 and gymnasium 1.2 through `uv sync`. A future gymnasium release that changes the step API again would break the scripts, and with no release tags you would be tracking a branch. The practical check before adopting is to run `uv sync` on your own machine and confirm the lockfile resolves, then run one Grid World script and one CartPole script to see the API surface for yourself. The Atari scripts are the expensive ones; verify the `--wandb` flag is genuinely optional by reading the argument parsing before you point them at a long run.

Editorial conclusion

Adopt it if you need to read a Q-learning or PPO update rule in one sitting, or to hand a working reference to someone learning RL. Do not adopt it as a training framework for a research result: the README's own Montezuma tables show the Go-Explore script returns a search trajectory, not a policy score, and the robustification script has no from-reset score at all. Before relying on any number, open the linked W&B report and check the seed count and the sticky-action setting.

Official sources

1. Issues
2. License: MIT
3. README
4. rlcode/reinforcement-learning on GitHub