CleanRL: single-file deep RL implementations and what you give up for them
High-quality single file implementation of Deep Reinforcement Learning algorithms with research-friendly features (PPO, DQN, C51, DDPG, TD3, SAC, PPG)
At a glance
- What is it?
- CleanRL puts each algorithm variant in one standalone Python file, from ppo.py to ppo_atari_envpool.py, and tells you outright that it is not a modular library. The trade is readable code against duplicated code, and the README is explicit about it.
- Who is it for?
- Adopt CleanRL if you need to read or modify every line of a PPO, DQN, C51, DDPG, TD3, SAC or PPG variant, or if you are prototyping a feature that modular libraries do not expose. Do not adopt it if you need a stable Python API to import from another service; the README states plainly that CleanRL is not meant to be imported.
- Can I use it commercially?
- Check first. The repository uses a licence we do not classify automatically, so read its LICENSE file before any commercial use.
- Is it still maintained?
- Yes. The repository last received commits 148 days ago.
- What is it written in?
- Mainly Python, according to GitHub's language statistics.
Answers come from the project's GitHub data, last synced on September 15, 2026, and from our analysis. They are not legal advice.
DEEP OPEN-SOURCE ANALYSIS
The problem CleanRL targets is reading an algorithm, not shipping one
Most deep RL libraries are built for reuse. You install a package, import an agent class, pass a config object, and call train. That structure is convenient until you need to know why a run diverges, at which point you are reading through a class hierarchy to find where a loss is computed. CleanRL inverts the priority. The README says every detail about an algorithm variant is put into a single standalone file, and gives the example that ppo_atari.py is 340 lines and contains all the implementation details for PPO on Atari. The stated audience is people who do not wish to read an entire modular library. The secondary audience is researchers prototyping features that modular libraries do not support, because the minimal line count means less subclassing to fight through. If your job is to run a known algorithm on a known environment and never open the source, this project is aimed at someone else.
What a CleanRL file actually contains
A CleanRL script is a complete program rather than a component. Based on the repository layout described in the README, each file carries its own argument parser, its own environment construction, its own network definitions, its own rollout loop, and its own training loop. There is no shared agent base class to inherit from and no registry to register into. That is why the file count grows with the variant count: ppo.py, ppo_atari.py, ppo_atari_lstm.py, ppo_procgen.py, ppo_atari_envpool.py and ppg_procgen.py are separate programs, not configurations of one program. The duplication is the design, not an accident of the codebase. The README frames it as a deliberate cost: at the expense of duplicate code, the implementation details stay legible. The research-friendly features sit alongside that core: Tensorboard logging, local reproducibility through seeding, gameplay video capture, and experiment tracking through Weights and Biases. The README also points to a benchmark site and a JMLR paper for the algorithm results, which is where you should look for numbers rather than here.
Getting a PPO run started with uv or pip
The README lists two prerequisites: Python >=3.7.1,<3.11, and uv 0.7.9 or later. The quickest path is to clone the repository and install it, then run a script directly. The README gives this sequence: git clone https://github.com/vwxyzjn/cleanrl.git && cd cleanrl, then uv pip install ., then uv run python cleanrl/ppo.py with --seed 1, --env-id CartPole-v0 and --total-timesteps 50000. Logs land under runs, and the README suggests tensorboard --logdir runs from inside the cleanrl directory. To add tracking, log in with wandb login once, then add --track and --wandb-project-name cleanrltest to the same command. If you prefer plain pip, the README splits dependencies into files under requirements/: requirements.txt for core, plus optional files for atari, mujoco, procgen, envpool, pettingzoo, jax, cloud, memory_gym and docs. Environment-specific installs use extras, for example uv pip install ".[atari]" before running python cleanrl/dqn_atari.py --env-id BreakoutNoFrameskip-v4. The envpool variant is Linux only, and the README notes that the speedup can come with lower sample efficiency, which is a real caveat rather than a footnote.
The no-import constraint is the main limitation
The README carries an explicit warning: CleanRL is not a modular library and therefore it is not meant to be imported. That single sentence rules out a large class of uses. You cannot pip install cleanrl and call an agent from a web service, a scheduler, or a larger training framework. You cannot swap the policy network for your own class without editing the script, because there is no injection point. You cannot share an environment wrapper across algorithms without copying it. Every fix you make to a rollout loop has to be reapplied to each file that needs it, and the README's own variant list shows how many files that can be. There is also a version ceiling: Python >=3.7.1,<3.11 is narrower than what many current setups run, so a modern interpreter will need a separate environment. The README also notes that migration to gymnasium is in progress and points to a pull request for status, which means the environment API surface may shift under you mid-project.
How CleanRL differs from Stable-Baselines3
The comparison worth making is with Stable-Baselines3, a modular library in the PyTorch ecosystem. SB3 gives you PPO, DQN, SAC and others behind a consistent interface: construct the algorithm with a policy string and an environment, call learn, then save and load. The algorithms share common machinery, so a change to the base classes propagates, and you can register custom policies and callbacks without touching the algorithm source. CleanRL offers no such interface. What you get instead is the ability to read ppo.py top to bottom in one sitting and see the advantage computation, the clipping, the minibatch loop and the logging in the order they execute. If you are debugging a loss curve or writing a paper that modifies one line of PPO, that readability is the whole point. If you are building a product that needs several algorithms behind one API, SB3's structure is the better fit and CleanRL will actively obstruct you. The README's own guidance lines up with this: consider CleanRL to understand an algorithm variant or to prototype features other libraries do not support.
Maintenance, upgrades and the licence question
The most recent release listed is v1.0.0 from November 2022, following two betas in 2022 that added JAX support and hyperparameter tuning. The repository itself is not archived and shows activity into 2026, so the release tags lag the commit history. In practice this means you should track the master branch rather than pinning to a release if you want current environment support. Upgrading is not a versioned dependency bump. Because each script is standalone, you pull changes by re-reading the files you depend on and porting your edits forward; there is no changelog that tells you which of your modifications conflict. The licence situation deserves a direct look. The README badge says MIT, while the repository metadata reports the licence as NOASSERTION, meaning GitHub could not map the LICENSE file to a known identifier. Those two signals disagree. Read the LICENSE file in the repository root and decide with whoever handles licensing on your side; this is a factual discrepancy in the project's own materials, not a legal opinion.
Who should adopt CleanRL and what to verify first
Adopt it if you are learning how a specific algorithm variant is implemented, if you are writing a paper that changes one component of PPO or SAC and need the surrounding code to be transparent, or if you are prototyping something modular libraries do not expose and want a short file to hack on. Do not adopt it as a dependency of a service, as a shared internal platform, or anywhere you need a stable import surface, since the README states it is not meant to be imported. Before you commit, verify three things against the repository itself: that your interpreter satisfies Python >=3.7.1,<3.11, that the gymnasium migration referenced in the README's pull request link has reached the scripts you plan to use, and what the LICENSE file actually says given that the badge and the repository metadata disagree. The README's own summary of the trade is the fairest one: CleanRL comes with its own pros and cons, and the con is duplicate code you will maintain by hand.
Editorial conclusion
Adopt CleanRL if you need to read or modify every line of a PPO, DQN, C51, DDPG, TD3, SAC or PPG variant, or if you are prototyping a feature that modular libraries do not expose. Do not adopt it if you need a stable Python API to import from another service; the README states plainly that CleanRL is not meant to be imported. Before committing, check the Python version support listed in the README against your own interpreter, confirm the gymnasium migration status referenced in the PR link, and read the LICENSE file in the repository, because the badge and the repository metadata disagree on the licence identifier.
Community notes