CleanRL: single-file RL scripts you run, not a library you import
High-quality single file implementation of Deep Reinforcement Learning algorithms with research-friendly features (PPO, DQN, C51, DDPG, TD3, SAC, PPG)
At a glance
- What is it?
- CleanRL packages deep reinforcement learning algorithms as one executable file per variant, which makes PPO readable end to end and makes the library unusable as a dependency. The trade is stated plainly in the README, and the metadata tells you which Python and torch versions that trade costs you.
- Who is it for?
- Adopt CleanRL when you want to read one algorithm variant end to end, or when you need to prototype a DRL idea that a modular library will not carry. Skip it when you need a stable API surface, a shared codebase across projects, or Python 3.11 or newer.
- Can I use it commercially?
- Check first. The repository uses a licence we do not classify automatically, so read its LICENSE file before any commercial use.
- Is it still maintained?
- Activity is slowing. The repository last received commits 6 months ago.
- What is it written in?
- Mainly Python, according to GitHub's language statistics.
Answers come from the project's GitHub data, last synced on November 15, 2026, and from our analysis. They are not legal advice.
Star history
- vwxyzjn/cleanrl10,689 ★ · +294
| Repository | 2026-09-13 | 2026-11-15 |
|---|---|---|
| vwxyzjn/cleanrl | 10.4k | 10.7k |
294 Stars in the last 63 days, measured from the daily GitHub snapshot. Compare with other projects
Editorial analysis
Every variant is a script you run, not a module you import
CleanRL's central decision is that an algorithm variant is a file you execute, not a package you import. The README states that CleanRL is not a modular library and therefore is not meant to be imported, and it names the price without flinching: duplicate code, paid on purpose so that every implementation detail of a variant is easy to follow. The stated payoff is that ppo_atari.py holds all of the details of PPO on Atari in 340 lines, offered as a reference for readers who would rather not work through a modular library.
What that means at the terminal is concrete. There is no cleanrl API to call. You clone, install, and run a file such as cleanrl/ppo.py with flags on the command line. Any code that needs the algorithm inside its own codebase has to carry its own copy of the file, and that copy then diverges from upstream. The other side of the same trade is debugging: the README says the short files remove a lot of the subclassing that modular DRL libraries require, and a stack trace points at one file you already have open.
pyproject.toml says 2.0.0b1 while the newest tag is v1.0.0 from 2022
Two version numbers describe this project and they do not match. pyproject.toml carries version = "2.0.0b1", while the newest tag is v1.0.0 from 2022-11-14, preceded by v1.0.0b2 on 2022-10-03 and v1.0.0b1 on 2022-06-07. The repository is not archived and its last push was 2026-04-20, so commits keep landing on a branch that has had no release tag in a long time. The practical reading is that master is a pre-2.0 branch, and any released version you pin predates most of what is on it.
Two more fields decide whether an install succeeds. pyproject.toml sets requires-python = ">=3.8,<3.11" while the README lists the prerequisite as Python >=3.7.1,<3.11, so a 3.7 interpreter satisfies the written prerequisites and fails the build metadata. The licence field in pyproject.toml reads MIT and names Costa Huang as author, even though licence detection at the repository level returns no assertion for this project. Benchmarks also sit outside the tree: the feature list points to benchmark.cleanrl.dev for 7+ algorithms and 34+ games, while the repository keeps its own benchmark/, tests/ and tuner_example.py.
uv drives the install and the first run, pip is the fallback
The documented path starts with uv, and the README puts uv 0.7.9+ in its prerequisites next to the Python constraint. Clone the repository, install the project, and run one script:
git clone https://github.com/vwxyzjn/cleanrl.git && cd cleanrl
uv pip install .
# alternatively, you could use `uv venv` and do
# `python run cleanrl/ppo.py`
uv run python cleanrl/ppo.py \
--seed 1 \
--env-id CartPole-v0 \
--total-timesteps 50000
# open another terminal and enter `cd cleanrl/cleanrl`
tensorboard --logdir runsThree flags carry that run: --seed 1, --env-id CartPole-v0 and --total-timesteps 50000. Flag parsing comes from tyro in pyproject.toml rather than a hand-written parser, which is why they read as plain names with no subcommand to learn. Tensorboard then reads the runs directory from the second terminal. Two comments in that block are the only hints about where output lands, and they disagree with the surrounding commands: one says to enter `cd cleanrl/cleanrl`, a level deeper than the clone, and the other reads `python run cleanrl/ppo.py` where the command below it reads `uv run python cleanrl/ppo.py`.
--track is the switch that turns on wandb
Experiment tracking is off unless you ask for it. The second example adds a login step, which the README marks as only required for the first time, and two flags to the same run:
wandb login # only required for the first time
uv run python cleanrl/ppo.py \
--seed 1 \
--env-id CartPole-v0 \
--total-timesteps 50000 \
--track \
--wandb-project-name cleanrltest--track enables the upload and --wandb-project-name cleanrltest says where it lands. Both examples share one seed, one environment and one timestep budget, so they differ in exactly one flag.
Two things follow from the dependency list rather than the README prose. Logging runs through two channels, tensorboard>=2.10.0 for scalars and moviepy>=1.0.3 for the gameplay videos the feature list advertises, with pygame>=2.1 beside it for the classic control environments and huggingface-hub>=0.11.1 in the same base set. Reproducibility here is a flag, not a subsystem: Local Reproducibility via Seeding is a listed feature, and both examples pass --seed 1. A headless machine therefore has to be served by something like the xvfb and ffmpeg packages in this repository's Docker image.
Extras decide which environments a script can even load
Everything a script needs comes from the base dependency list, which pins both environments libraries at once:
dependencies = [
"tensorboard>=2.10.0",
"wandb>=0.13.11",
"gym==0.23.1",
"gymnasium==0.29.1",
"torch==2.4.1",
"moviepy>=1.0.3",
"pygame>=2.1",
"huggingface-hub>=0.11.1",
"rich<12.0",
"tenacity>=8.2.2",
"tyro>=0.5.10",
]The extras then take over. atari pulls ale-py==0.8.1, AutoROM[accept-rom-license]~=0.4.2, opencv-python and shimmy>=1.1.0. envpool pins envpool>=0.6.4,<0.7. pettingzoo pins PettingZoo==1.18.1 with SuperSuit==3.4.0 and multi-agent-ale-py==0.1.11. jax pins jax==0.4.8 and jaxlib==0.4.7. One group is a placeholder, since plot = [] lists no packages at all.
That structure has a cost on the pip path. The fallback in the README asks you to install from a requirements/ tree with a separate file per domain, including requirements-mujoco_py.txt beside requirements-mujoco.txt, so the uv route and the pip route declare the same dependencies twice and can drift.
envpool variants are Linux only and take different environment ids
Classic control is the cheapest way in. Three scripts are named for it, against CartPole-v1, and the Atari group adds four more against BreakoutNoFrameskip-v4:
uv venv
# classic control
python cleanrl/dqn.py --env-id CartPole-v1
python cleanrl/ppo.py --env-id CartPole-v1
python cleanrl/c51.py --env-id CartPole-v1
# atari
uv pip install ".[atari]"
python cleanrl/dqn_atari.py --env-id BreakoutNoFrameskip-v4
python cleanrl/c51_atari.py --env-id BreakoutNoFrameskip-v4
python cleanrl/ppo_atari.py --env-id BreakoutNoFrameskip-v4
python cleanrl/sac_atari.py --env-id BreakoutNoFrameskip-v4The envpool path carries its own conditions in the comment the README attaches to it, including that it is only available to linux:
# NEW: 3-4x side-effects free speed up with envpool's atari (only available to linux)
uv pip install ".[envpool]"The 3-4x figure is a claim in that README comment, not something the repository measures on your machine.
The Dockerfile starts from nvidia/cuda:11.4.2 on Ubuntu 20.04
Runs at scale go through the cloud/ directory and one Dockerfile, and its first line is the whole host requirement in the project:
FROM nvidia/cuda:11.4.2-runtime-ubuntu20.04
# install ubuntu dependencies
ENV DEBIAN_FRONTEND=noninteractive
RUN apt-get update && \
apt-get -y install python3-pip xvfb ffmpeg git build-essential python-opengl
RUN ln -s /usr/bin/python3 /usr/bin/python
# install python dependencies
RUN mkdir cleanrl_utils && touch cleanrl_utils/__init__.py
RUN pip install uv --upgrade
COPY pyproject.toml pyproject.toml
COPY uv.lock uv.lock
RUN uv pip install .Two things follow. The base is an NVIDIA CUDA runtime image on Ubuntu 20.04, so a host without an NVIDIA GPU is the wrong machine for that file even though the base dependencies never ask for a GPU. And the install order is deliberate enough to copy: uv is upgraded first, then pyproject.toml and uv.lock arrive before the install, and uv pip install . runs against the lock rather than a bare requirements file. The image ends by copying entrypoint.sh into /usr/local/bin, marking it chmod 777, and running it as the entrypoint, with the cleanrl scripts copied to /cleanrl afterwards.
CORL and LeanRL take the same algorithm list in opposite directions
Two projects named in the README as notable CleanRL relatives are listed alongside it. corl-team/CORL implements offline RL in CleanRL style. pytorch-labs/LeanRL is a fast optimized PyTorch implementation of the CleanRL algorithms using CUDAGraphs.
That is the real difference in approach, and it decides the choice before any feature list does. CleanRL's stated goals are a short file you can read in one sitting, minimal lines, and no subclassing, so its own cost is duplicated code between variants. If you are choosing an algorithm reference to learn from, CleanRL is the one whose file you will be able to hold in your head.
Editorial conclusion
Adopt CleanRL when you want to read one algorithm variant end to end, or when you need to prototype a DRL idea that a modular library will not carry. Skip it when you need a stable API surface, a shared codebase across projects, or Python 3.11 or newer. Before anything else, check the two version numbers that disagree: pyproject.toml says 2.0.0b1 while the newest tag is v1.0.0 from 2022-11-14, so pinning a release gets you code that is older than master.
Frequently asked questions
How do I use CleanRL?
Clone the repository, install it, and run one script: uv run python cleanrl/ppo.py --seed 1 --env-id CartPole-v0 --total-timesteps 50000. The README lists Python >=3.7.1,<3.11 and uv 0.7.9+ as prerequisites, while pyproject.toml sets requires-python to >=3.8,<3.11, and results are read afterwards with tensorboard --logdir runs.
How does CleanRL differ from a modular DRL library?
CleanRL states that it is not a modular library and is not meant to be imported, so it is a set of scripts you execute rather than an API you call from your own code. The cost it names is duplicate code between variants, and the benefit is that every implementation detail of a variant sits in one standalone file, with ppo_atari.py at 340 lines.
Can I import CleanRL as a Python library?
No. The README says CleanRL is not a modular library and therefore is not meant to be imported, and it accepts duplicate code as the price for keeping every variant in a single readable file. Code that needs an algorithm from this project has to carry its own copy of the script.
Official sources
Add this badge to your README
If you maintain this project, the badge below links readers to this analysis and shows its maintenance status from the daily GitHub snapshot. Paste the markdown into your README; add ?metric=license or ?metric=stars to the image URL for a different field.
[](https://hysenlabs.com/projects/vwxyzjn-cleanrl)Star history chart for your README
[](https://hysenlabs.com/projects/vwxyzjn-cleanrl)Was this useful?