RL Baselines3 Zoo: Tuned Hyperparameters and Training Scripts for Stable Baselines3
A training framework for Stable Baselines3 reinforcement learning agents, with hyperparameter optimization and pre-trained agents included.
At a glance
- What is it?
- RL Baselines3 Zoo wraps Stable Baselines3 in a command-line training framework with tuned hyperparameters, evaluation and plotting scripts, and a collection of pre-trained agents. It is convenient for benchmarking and for getting a baseline running, and less suited to research code that needs custom training loops.
- Who is it for?
- Adopt RL Baselines3 Zoo if you want to train or benchmark standard Gymnasium environments with Stable Baselines3 and prefer tuned YAML hyperparameters over writing your own loop. Do not adopt it if you need a custom training loop, non-Gymnasium environments, or a quantitative benchmark: the README states the benchmark corresponds to only one run.
- Can I use it commercially?
- Yes. MIT is a permissive licence: you can use, modify and sell software built on it, as long as you keep its copyright and licence notices.
- Is it still maintained?
- Yes. The repository last received commits 6 days ago.
- What is it written in?
- Mainly Python, according to GitHub's language statistics.
Answers come from the project's GitHub data, last synced on September 15, 2026, and from our analysis. They are not legal advice.
DEEP OPEN-SOURCE ANALYSIS
What RL Baselines3 Zoo solves, and for whom
Stable Baselines3 gives you the algorithms. It does not give you the surrounding decisions: which learning rate, which batch size, how often to evaluate, how to store the resulting policy, how to plot the training curve afterwards. RL Baselines3 Zoo is that layer. The README describes it as a training framework for RL using Stable Baselines3, with scripts for training, evaluating agents, tuning hyperparameters, plotting results and recording videos.
The audience is narrow and fairly clear. It is for people who work on the environments already covered by the repository: Atari games such as Breakout and Pong, classic control tasks such as CartPole-v1 and Pendulum-v1, and the PyBullet robotics environments that appear in the topics list. For those, the repository ships tuned hyperparameters and, in many cases, an agent already trained with them. The README lists goals in order: a simple interface to train and enjoy RL agents, benchmarking the algorithms, providing tuned hyperparameters per environment and algorithm, and having fun with the trained agents.
That ordering matters. This is a benchmarking and reproducibility tool first. If your problem is a novel environment with a custom observation space, the framework will still run, but the tuned hyperparameters that justify using it do not exist for you yet, and you are back to tuning from scratch.
How the training pipeline is wired
The core mechanism is a lookup. Hyperparameters for each environment live in YAML files under hyperparams/, named after the algorithm, as the README states: "The hyperparameters for each environment are defined in hyperparameters/algo_name.yml." When you pass --algo and --env, the training script resolves that pair against the YAML file and applies the stored settings to the Stable Baselines3 algorithm it constructs. There is no separate configuration language to learn; the tuned values are the configuration.
Around that core sit the supporting scripts. train.py runs training and periodic evaluation. enjoy.py loads a saved policy and renders it. A benchmark entry point, run as python -m rl_zoo3.benchmark, recomputes the final performance table found in benchmark.md. The repository layout reflects this: train.py and enjoy.py at the top level, the package under rl_zoo3/, hyperparams/ for the YAML files, scripts/ for tooling, tests/ for the test suite, and a rl-trained-agents submodule holding the trained policies. The .gitmodules entry is why the README insists on git clone --recursive.
One design consequence is worth stating plainly. Because the algorithm settings come from a file keyed by environment id, the framework is only as good as that file for your environment. The README notes the collection is incomplete and that contributors are wanted, which tells you the coverage is uneven across the environment list.
Installing rl_zoo3 and training a first agent
The README gives two installation paths. The minimal one installs from source with pip install -e . or from PyPI as pip install rl_zoo3. The package name is rl_zoo3, and installing it also gives you a command-line entry point: the README notes that rl_zoo3 train is equivalent to python train.py, and that python -m rl_zoo3.train works from any folder.
If you need the extra environments and test dependencies, the full installation is heavier. It requires system packages first, then the extras:
apt-get install swig cmake ffmpeg
pip install -e .[plots,tests,extras]On macOS the README warns that pybullet may fail to build with errors from stdio.h, and gives a workaround that sets a compiler flag for the same extras install:
CFLAGS="-fno-define-target-os-macros" pip install -e .[plots,tests,extras]With the package in place, training a covered environment is one command. The README's example uses SAC on a PyBullet task with explicit evaluation settings:
python train.py --algo sac --env HalfCheetahBulletEnv-v0 --eval-freq 10000 --eval-episodes 10 --n-eval-envs 1That evaluates every 10000 steps over 10 episodes using a single evaluation environment. Expect a progress display driven by tqdm and rich, and a log directory under logs/ holding the run. To watch an agent instead of training one, the README shows enjoy.py with a timestep count:
python enjoy.py --algo a2c --env BreakoutNoFrameskip-v4 --folder rl-trained-agents/ -n 5000That plays A2C on Breakout for 5000 timesteps from the pre-trained agents folder. If the folder is empty, you cloned without --recursive.
Where the framework gets in your way
The sharpest limitation is stated by the project itself. The benchmark table in benchmark.md is not a quantitative benchmark, because it corresponds to only one run per configuration. The README is explicit about this and points to issue #38. So the numbers are useful for spotting a broken configuration or a badly tuned agent, and not useful for claiming that algorithm A beats algorithm B.
A second constraint is the environment contract. The hyperparameter lookup assumes your environment id exists in the YAML files. The README notes that the collection still needs contributors, and the Atari tables in the README show additional games marked as to be completed. For an environment outside the covered set, you supply your own hyperparameters, and the main value of the framework drops to convenience scripts.
There is also a packaging trap. The trained agents live in a git submodule, so a plain clone gives you the code without the policies, and the README says so directly. Anyone who skips the --recursive flag will find enjoy.py failing for reasons that have nothing to do with their setup. Finally, the dependency set is not small: optuna, shimmy, huggingface_sb3, pyyaml and pytablewriter are all required, and the extras pull in gym 0.26.2 alongside modern gymnasium. That mix is what makes the PyBullet and legacy Atari environments work, but it also means a fresh install can fail at the system library stage before Python is even involved. If your project already pins a different gymnasium range, expect to spend time on resolution.
RL Baselines3 Zoo compared with a plain Stable Baselines3 script
The obvious alternative is Stable Baselines3 on its own. The difference is not algorithmic: the zoo calls the same PPO, SAC, DQN and A2C implementations from the same library, and the README points to the Stable Baselines3 documentation for installation alternatives. The difference is everything around the algorithm.
With a plain script you write the training loop, choose the hyperparameters, add evaluation callbacks, name the output directory, and build your own plotting. With the zoo, that scaffolding exists, and the hyperparameters for covered environments are already tuned and stored in YAML. You trade control for a working baseline in one command. The zoo also adds Optuna-based hyperparameter search as a first-class feature rather than something you bolt on, and it integrates with Weights & Biases for tracking and Hugging Face for storing and sharing trained models, per the README's integrations section.
The trade-off is real in both directions. A plain script adapts to any environment and any logging stack without fighting the framework's assumptions. The zoo gives you reproducibility across the environments it covers, which is exactly what a benchmarking tool should do, and friction everywhere else. If your work is a paper on a custom simulator, the zoo's tuned files are irrelevant to you. If your work is comparing PPO and SAC on HalfCheetah, they are the whole point.
Maintenance, upgrades and the MIT licence
The repository is not archived, and its last push was on 2026-09-09, which is recent. Releases are frequent and small: v2.9.1 on 2026-06-15 relaxed the Gymnasium version range to allow any 1.x, v2.9.0 the same day updated dependencies, and v2.8.0 on 2026-04-01 added default hyperparameters for environments not listed in the YAML files plus other quality-of-life improvements. That last one matters for anyone with a custom environment, because it means the framework now has a fallback rather than failing outright on an unknown id.
The upgrade cost is mostly dependency churn. The package requires Python 3.10 or newer, and the classifiers list 3.10 through 3.13. It pins sb3_contrib and stable-baselines3 to the 2.x line, gymnasium to below 2.0, and optuna to 3.0 or above. The v2.9.1 release exists precisely because a Gymnasium range was too tight, which is a fair warning that minor releases can move your environment stack. The optional extras are the fragile part: the plots extra pins rliable and pandas ranges, and the pyproject comments note a compatibility constraint with pandas 3.x. If you only need training, the minimal install avoids most of this.
The licence is MIT, declared in both the README badge context and the project metadata, with the LICENSE file at the repository root. MIT is permissive: it allows commercial use and modification with attribution and no warranty. That is a statement about the licence text, not advice about your situation; if you redistribute the pre-trained agents, check the terms attached to the environments and datasets they were trained on separately.
Editorial conclusion
Adopt RL Baselines3 Zoo if you want to train or benchmark standard Gymnasium environments with Stable Baselines3 and prefer tuned YAML hyperparameters over writing your own loop. Do not adopt it if you need a custom training loop, non-Gymnasium environments, or a quantitative benchmark: the README states the benchmark corresponds to only one run. Verify first that your environment id appears in hyperparams/, that your Python is 3.10 or newer, and that you cloned with --recursive if you want the pre-trained agents.
Frequently asked questions
What is RL Baselines3 Zoo used for?
It is a training framework for reinforcement learning agents built on Stable Baselines3. It provides scripts for training, evaluating, tuning hyperparameters, plotting results and recording videos, plus tuned hyperparameters and pre-trained agents for common environments.
How do I install RL Baselines3 Zoo?
The minimal installation is pip install -e . from source, or pip install rl_zoo3 as a package. The full installation with extra environments and test dependencies requires apt-get install swig cmake ffmpeg followed by pip install -e .[plots,tests,extras].
How do I train an agent with RL Baselines3 Zoo?
If the environment exists in hyperparams/algo_name.yml, run python train.py --algo algo_name --env env_id. You can add evaluation options such as --eval-freq, --eval-episodes and --n-eval-envs to control how often and how thoroughly the agent is checked during training.
Why does enjoy.py fail to find a trained agent in RL Baselines3 Zoo?
The trained agents are stored in a git submodule, so a normal clone does not include them. The README says you must clone with git clone --recursive https://github.com/DLR-RM/rl-baselines3-zoo to get the submodule contents.
Is the RL Baselines3 Zoo benchmark a fair comparison between algorithms?
No. The README states the benchmark corresponds to only one run per configuration and is not a quantitative benchmark. It is meant to check maximal algorithm performance, find potential bugs and give users access to pre-trained agents.
What Python version does RL Baselines3 Zoo require?
The project metadata sets requires-python to >=3.10 and lists classifiers for Python 3.10 through 3.13. Older Python versions are not supported by the package as published.
Community notes