reinforcement-learning: one file per algorithm, single-seed benchmarks, and two counts that disagree
Minimal and Clean Reinforcement Learning Examples
At a glance
- What is it?
- A teaching repository of single-file implementations of classic algorithms across three example environments, modernised from a 2017 original and now carrying benchmark tables run on a laptop's integrated GPU with one seed. Its README does something most teaching repositories do not: it explains why its scores look worse than published ones, and it says outright that it could not reproduce two of the results it tried for.
- Who is it for?
- This is the repository to hand someone who wants to read a whole algorithm start to finish rather than configure one, because one file per algorithm with a citation and the update equation at the top is the right shape for that. Two things to know before you quote its numbers.
- Can I use it commercially?
- Yes. MIT is a permissive licence: you can use, modify and sell software built on it, as long as you keep its copyright and licence notices.
- Is it still maintained?
- Yes. The repository last received commits 156 days ago.
- What is it written in?
- Mainly Python, according to GitHub's language statistics.
Answers come from the project's GitHub data, last synced on November 15, 2026, and from our analysis. They are not legal advice.
Star history
- rlcode/reinforcement-learning3,679 ★ · +17
| Repository | 2026-09-11 | 2026-11-15 |
|---|---|---|
| rlcode/reinforcement-learning | 3.7k | 3.7k |
17 Stars in the last 65 days, measured from the daily GitHub snapshot. Compare with other projects
Editorial analysis
One file per algorithm, in directories named after their order
The layout is the argument. Every algorithm is a single file, the directories are numbered in the order you would learn them, and the files inside are numbered to match. The first directory is a grid world and holds policy iteration, value iteration, SARSA, Q-learning, deep SARSA and REINFORCE, so a tabular method and its function-approximation counterpart sit next to each other. The second is a control task with a deep Q-network, an advantage actor critic and proximal policy optimisation. The third is two arcade games with the same two deep methods. The modernisation note says the layout used to nest an agent inside a per-algorithm directory and is now flat, which removes a directory level and makes the file you want the only thing on the screen.
The index lists eleven algorithms and the update note says nine
Count the algorithm index and you get eleven numbered entries across three directories. Then read the modernisation note at the bottom, which says the scope was pruned to nine core algorithms, with several named ones dropped and one added. Both numbers are describing the same list. Either the note predates a change to the index, or the index counts the two arcade entries that the note did not, or the note counted before one of the deep methods was added back. Nothing depends on the discrepancy, and it is the kind of thing that only shows up because the repository is small enough to count.
A fourth directory is benchmarked in full and never listed
There is a fourth directory in the tree, and the README never mentions it in the algorithm section. It is not an oversight in the writing: the benchmarks section spends a whole table and a whole footnote on it, naming three scripts inside it, so the omission is specifically in the index. That means the algorithms list is not the complete picture of what is in the repository, which is the opposite of what a list whose whole purpose is to be a table of contents wants to be. It also means the difficulty ranking is implicit rather than written down, which is where the interesting comparison lives.
The benchmarks ran on a laptop, with one seed, read from a system monitor
The methodology paragraph is unusually explicit and it changes how every number in the next two tables should be read. The machine was a laptop with an integrated GPU and eight gigabytes of unified memory, on a recent operating system and a specific interpreter version, with the vendor's own backend for that hardware. Processor and accelerator percentages were read from the operating system's activity monitor on the interpreter's own process, once the run had stabilised, about five minutes in, and peak memory is that process's real high-water mark. A score is the mean return over the final twenty episodes. The footnote adds that each row is a single seed, that the parameter column counts only trainable network weights, that the processor column sums across cores, and that the sticky-action setting makes absolute scores lower than the older deterministic environments cited in published work.
Sticky actions are the reason the numbers look small, and the README says so
On the easier game, with ten million steps, the deep Q-network took about nine hours for a score in the low nineties and the policy gradient method about four hours for a score in the high two hundreds, both with the same reported parameter count. What makes those numbers comparable to anything at all is the footnote, which explains that the environment version used applies a repeat action probability of a quarter, and that this deliberately lowers scores relative to the older environment variants that most papers quote. That is a methodological choice stated in the same file as the results, and it is the difference between a benchmark you can reason about and one you cannot. It is also why the score columns are given with a spread: the mean over the last twenty episodes, with a standard deviation beside it.
A tenfold difference that the README forbids you to compare
The hard-game table contains three rows and a warning above them: two protocols, not cross-comparable. The policy gradient method with random network distillation gets about three thousand, using sticky actions and sixty-five million frames. A search-based exploration method gets thirty-one thousand using deterministic restore, over five hundred million frames, and the score is marked replay-verified. A third row, a robustification script, has no score at all and the cell says so: no from-reset score, five million frames, no link. So the table deliberately presents a number ten times larger next to a smaller one and tells you the comparison is meaningless. Most benchmark tables would have shown only the larger figure.
It says it failed to reproduce published results, and gives the reason
The footnote under that table is the best writing in the repository. Random network distillation found the first key in roughly three hundred thousand steps with five hundred and twelve parallel environments, of which a hundred and twenty-eight never scored anything across fifty million steps, so the lever is parallel breadth rather than depth of search; it then plateaued above this repository's own baseline and far below the published result, at around thirty times less experience. The exploration method's number comes from a knowledge-free cell archive with no network at all, so it is a search result rather than a reinforcement learning score, and the footnote prints the published figure beside it. The robustification script plateaus at around twenty two percent on one machine, which the README attributes to running hundreds to thousands of environments in the original work.
The interpreter is pinned to one minor version and three dependencies have no ceiling
The setup section is three commands long and needs one specific minor version of the interpreter plus a package manager:
git clone <this repo>
cd reinforcement-learning
uv syncThe first command is a placeholder rather than an address, so a reader copies the second and third lines and clones by hand. Everything else about the setup is stated precisely: the manifest pins the interpreter exactly, gives nine of its twelve dependencies an upper bound inside a single minor release, and leaves the last three as lower bounds with nothing above them. The running section then shows one command per example, with two flags on the control task, one to render the environment while it trains and one to replay a saved checkpoint.
Editorial conclusion
This is the repository to hand someone who wants to read a whole algorithm start to finish rather than configure one, because one file per algorithm with a citation and the update equation at the top is the right shape for that. Two things to know before you quote its numbers. The benchmarks are single-seed runs on consumer hardware read from a system monitor, so they are smoke tests rather than results. And the hardest environment is reported as a failure to reproduce, with the reason given, which makes the repository more useful as a research starting point than as a source of scores. If you want to compare against published numbers, change the environment protocol first, because the README says the two are not comparable.
Frequently asked questions
What is reinforcement learning and examples?
The repository is a set of single-file implementations of classic algorithms on three example environments: policy iteration, value iteration, SARSA, Q-learning, deep SARSA and REINFORCE on a grid world; a deep Q-network, an advantage actor critic and proximal policy optimisation on a control task; and the same two deep methods on two arcade games. A fourth directory holds hard-game experiments.
how to set up reinforcement learning
You need one specific minor version of the interpreter and the uv package manager, then run the sync command. The manifest pins the interpreter exactly and gives nine of its twelve dependencies an upper bound inside a single minor release, while three are lower bounds with no ceiling.
Is reinforcement learning tough?
The repository does not comment on difficulty. It describes itself as easy-to-read code with one file per algorithm, and its update notes describe what modernising the 2017 original involved, including which algorithms were dropped.
What is reinforcement learning with human feedback?
Not covered here. The scope is classic control and game algorithms, and the update notes record that several earlier algorithms were dropped when the project was modernised onto a different framework and a different environment library.
Does reinforcement-learning train on a GPU?
Its published benchmarks were run on a laptop with unified memory, using the vendor's own backend for that hardware, with processor and accelerator figures read from the system activity monitor about five minutes into each run. The notes say the processor figure sums across cores, so a value above one hundred means multiple cores.
Official sources
Add this badge to your README
If you maintain this project, the badge below links readers to this analysis and shows its maintenance status from the daily GitHub snapshot. Paste the markdown into your README; add ?metric=license or ?metric=stars to the image URL for a different field.
[](https://hysenlabs.com/projects/rlcode-reinforcement-learning)Star history chart for your README
[](https://hysenlabs.com/projects/rlcode-reinforcement-learning)Was this useful?