Stable Baselines3: A Stable PyTorch RL Baseline That Stopped Adding Algorithms
PyTorch version of Stable Baselines, reliable implementations of reinforcement learning algorithms.
At a glance
- What is it?
- Stable Baselines3 is a set of PyTorch reinforcement learning algorithm implementations with a shared interface, aimed at researchers and engineers who need reproducible baselines. Its maintainers describe it as stable and now focused on bug fixes and maintenance, which is the main thing to weigh before adopting it.
- Who is it for?
- Adopt Stable Baselines3 if you need a maintained, MIT-licensed PyTorch baseline for PPO, SAC, TD3, DQN, A2C or their neighbours, with custom environments and callbacks behind one interface. Do not adopt it expecting new algorithms in the core package; the README states there are no major changes planned and that newer algorithms go to SB3-Contrib, faster variants to SBX, and training scripts to the RL Zoo.
- Can I use it commercially?
- Yes. MIT is a permissive licence: you can use, modify and sell software built on it, as long as you keep its copyright and licence notices.
- Is it still maintained?
- Yes. The repository last received commits 6 days ago.
- What is it written in?
- Mainly Python, according to GitHub's language statistics.
Answers come from the project's GitHub data, last synced on September 15, 2026, and from our analysis. They are not legal advice.
DEEP OPEN-SOURCE ANALYSIS
The Problem: Reproducible RL Baselines, Not New Algorithms
Reinforcement learning research has a reproducibility problem that is partly a software problem. A published result often depends on an implementation that exists in one lab, with hyperparameters that were tuned for one environment and a training loop that is not the same as the one in the next paper. Stable Baselines3 exists to remove that variable. The README says the algorithms are meant to make it easier for the research community and industry to replicate, refine, and identify new ideas, and to create good baselines to build projects on top of. It also states the intent plainly: the tools should be used as a base around which new ideas can be added, and as a way to compare a new approach against existing ones.
The audience follows from that. This is for someone who already knows what a policy gradient is and wants a PPO or SAC implementation they do not have to debug before they can run an experiment. The README is explicit that the library assumes some knowledge of RL: despite its simplicity of use, you should not use it without some practice, and the documentation includes an RL guide for people who need it. So it is not a teaching library and not a no-code tool. It is infrastructure for people who can read a training curve and tell whether something is wrong.
A second audience is engineers building on top of an algorithm rather than studying it. Because the interface is shared across algorithms, swapping PPO for SAC is a small code change, which matters when the algorithm is not the interesting part of your project.
One Interface Across Algorithms, Policies and Callbacks
The mechanism that makes SB3 useful is not any single algorithm. It is the common interface listed in the feature table, alongside custom environments, custom policies, custom callbacks, Dict observation space support, Tensorboard support and type hints. Those features are the architecture. An algorithm in SB3 is a class that owns a policy object and a rollout or replay buffer, and the training loop lives in a learn() method on the algorithm. The policy object holds the neural network and the distribution over actions. Callbacks hook into the loop at defined points without modifying the algorithm.
That separation is what lets the same training script work across algorithms. You construct an environment, construct a model with a policy string and hyperparameters, and call learn. The policy can be swapped for a custom one, which is the extension point for a different network shape. Callbacks are the extension point for evaluation, checkpointing or logging, and the README lists custom callback support as a first-class feature rather than an afterthought.
The feature table also lists GSDE among the repository topics, which is the generalized State Dependent Exploration method used in the SB3 policy implementations. That detail matters for continuous control: exploration noise is part of the policy, not a separate wrapper, so it is configured through the algorithm rather than bolted on.
What the material does not give is a diagram of the data flow or a description of the replay buffer internals. The architecture above follows from the documented feature list and the public API shape, not from a code walkthrough in the README.
Installing SB3 and the Version Constraints That Come With It
Installation is a pip install, but the constraints are the part people trip over. The README states that Stable-Baselines3 supports PyTorch >= 2.8 and requires Python 3.10+. Those are not soft recommendations. The v2.9.0 release notes say the release updated dependencies to gymnasium 1.3.0 support and torch>=2.8, and made pandas optional. The v2.8.0 release notes say Python 3.9 was dropped and Python 3.13 support was added. If you are pinned to an older Python or an older torch, you are pinned to an older SB3, and the release notes are where you check which one.
On Windows, the README does not give a command. It points to the documentation instead: to install stable-baselines on Windows, please look at the documentation. That is a real gap in the README, and it is worth reading the docs page before assuming a plain pip install will work.
The README does not list the pip package name or a full install command in the excerpt available here, so the exact install line cannot be confirmed from this material. What can be confirmed is the dependency floor and the Python floor, and those are the two things to check against your environment before anything else.
Two adjacent packages are worth knowing about at install time. RL Baselines3 Zoo is a separate repository that provides scripts for training, evaluating agents, tuning hyperparameters, plotting results and recording videos, plus tuned hyperparameters for common environments and agents trained with those settings. SB3-Contrib is a second package that holds experimental algorithms. Neither is part of the core install.
The Core Is Frozen: Where New Algorithms Actually Live
The most important limitation is stated in the README rather than discovered later. Since most of the features from the original roadmap have been implemented, there are no major changes planned for SB3; it is now stable. Development is focused on bug fixes and maintenance, described as doc update and user experience work. The recent release history is consistent with that: v2.7.1 is bug fixes and documentation improvements, v2.8.0 is Python version changes and a MaskablePPO bug fix, v2.9.0 is dependency updates.
So if you arrive expecting the newest algorithm from a recent paper, SB3 is the wrong package. The README routes you elsewhere. Newer algorithms are regularly added to SB3-Contrib, which the README describes as the place for experimental features including Recurrent PPO (PPO LSTM), CrossQ, Truncated Quantile Critics (TQC), Quantile Regression DQN (QR-DQN) and PPO with invalid action masking (Maskable PPO). Faster variants are developed in SBX, the Jax version. Training scripts and tuned hyperparameters live in the RL Zoo.
That split is deliberate: it allows SB3 to maintain a stable and compact core while still providing the latest features. The trade-off is that an algorithm you want may require a second dependency with a different stability expectation. SB3-Contrib is described as experimental, and experimental means exactly that. A second limitation is speed. SBX is described as a proof of concept with a minimal number of features compared to SB3 but potentially much faster, with the README citing up to 20x. That number comes from the project's own comparison against its Jax port, not from an independent measurement, and it is the kind of claim you would want to reproduce on your own environment before making a decision on it.
What the Documentation Does and Does Not Tell You About Performance
SB3 makes a performance claim and then qualifies it in a way worth reading carefully. The README says the performance of each algorithm was tested and points to the Results section of each algorithm's page, and to two GitHub issues for details. It also says detailed logs and reports are on the OpenRL Benchmark platform. That is a more honest arrangement than a single headline number, because it puts the evidence next to the algorithm rather than in a marketing table.
The limitation is that tested does not mean tested on your problem. The Results sections are tied to specific environments. If your observation space, action space or reward structure differs, the published curve tells you the implementation is not broken, not that it will learn your task. The README's own framing supports this: the library is a baseline for comparison, which means the comparison is the work you do, not something the library does for you.
One more thing the material does not cover: there is no stated support policy, no long-term support commitment and no deprecation timeline in the README. The release notes show Python 3.9 being dropped in v2.8.0, which is the practical shape of the maintenance policy. Upgrades will occasionally require a Python or torch bump, and the release notes are the only advance notice you get.
Alternatives: SB3-Contrib, the RL Zoo and SBX Solve Different Problems
The honest comparison is not against another RL library. It is against the three repositories the README itself points to, because they divide the same problem space differently.
SB3-Contrib keeps the same interface and the same style of implementation but relaxes the stability bar to ship newer algorithms. If you need recurrent policies or invalid action masking, that is where you go, and the cost is that you are now depending on code the project labels experimental. The difference in approach is the acceptance criteria for merging an algorithm, not the API.
The RL Baselines3 Zoo is a training framework rather than an algorithm library. It provides scripts for training, evaluating, tuning, plotting and recording, plus a collection of tuned hyperparameters for common environments and trained agents. If your problem is close to a common benchmark, the Zoo saves you the hyperparameter search that SB3 itself does not do. If your environment is unusual, the tuned hyperparameters are less likely to transfer, and you are back to tuning by hand.
SBX is the Jax port. The README describes it as a proof of concept with a minimal number of features compared to SB3 but potentially much faster. The trade-off is feature coverage against wall-clock time. If your bottleneck is training throughput and you can live with fewer features, SBX is the intended answer. If you need the full feature set and the stable API, SB3 is. These are not competing products so much as a deliberate division of labour, and the README states that division explicitly.
Licence and the Cost of Staying on a Maintained Baseline
SB3 is MIT licensed. In practical terms that is a permissive licence: it allows use, modification and redistribution with few conditions, and it is compatible with commercial work. This is not legal advice, and the licence text is the authority, but the identifier is worth noting because it removes the licensing question from your adoption decision in a way that copyleft RL tooling would not. The adjacent repositories are separate projects with their own licences, so if you pull in SB3-Contrib or the RL Zoo, check those separately.
The maintenance cost of SB3 itself is low and predictable. The project is in bug-fix mode, so the core API is not moving. What does move is the dependency floor: v2.8.0 dropped Python 3.9 and added Python 3.13, and v2.9.0 moved to torch>=2.8 and gymnasium 1.3.0 while making pandas optional. If you are on a managed Python distribution or a locked torch version, those bumps are the upgrade events you will actually feel, and they arrive on the project's schedule rather than yours.
The upgrade cost is asymmetric depending on how you use the library. If you train a model and ship the weights, an SB3 upgrade is mostly a matter of checking that loading still works. If you subclass policies or write callbacks against internal attributes, a dependency bump is more likely to reach you, because internal structure is not the part the project promises to keep stable. The README's statement that SB3 is now stable is about the feature set, not a guarantee that every internal attribute survives a minor release.
Editorial conclusion
Adopt Stable Baselines3 if you need a maintained, MIT-licensed PyTorch baseline for PPO, SAC, TD3, DQN, A2C or their neighbours, with custom environments and callbacks behind one interface. Do not adopt it expecting new algorithms in the core package; the README states there are no major changes planned and that newer algorithms go to SB3-Contrib, faster variants to SBX, and training scripts to the RL Zoo. Before you commit, verify that torch>=2.8 and Python 3.10+ are available in your environment, and check the algorithm page for your chosen algorithm to see whether its documented performance was measured on the environment you care about.
Community notes