sb3-contrib: experimental RL algorithms bolted onto Stable-Baselines3
Contrib package for Stable-Baselines3 - Experimental reinforcement learning (RL) code
At a glance
- What is it?
- SB3-Contrib is a companion package that ships algorithms the main Stable-Baselines3 repository will not take, from MaskablePPO to CrossQ. It keeps the SB3 API and style, but the code is explicitly experimental and the version pinning is tight.
- Who is it for?
- Adopt sb3-contrib if you need invalid action masking, a recurrent PPO policy, or distributional value learning and you already run Stable-Baselines3, because the algorithms share the SB3 API. Do not adopt it as a general-purpose RL framework or as a source of production-stable training code, and do not expect the maintainers to treat a regression in CrossQ the way they would treat one in PPO.
- Can I use it commercially?
- Yes. MIT is a permissive licence: you can use, modify and sell software built on it, as long as you keep its copyright and licence notices.
- Is it still maintained?
- Yes. The repository last received commits 54 days ago.
- What is it written in?
- Mainly Python, according to GitHub's language statistics.
Answers come from the project's GitHub data, last synced on September 15, 2026, and from our analysis. They are not legal advice.
DEEP OPEN-SOURCE ANALYSIS
The gap sb3-contrib was created to fill
Stable-Baselines3 has a deliberately narrow scope. The README of sb3-contrib describes a recurring problem: community contributions such as better logging utilities, environment wrappers, extended action-space support and new learning algorithms were often judged too niche for the main repository, or too awkward to integrate without disturbing existing code. The stated goal of sb3-contrib is to accept that material anyway, without requiring a tidy integration and without a bar for what counts as too niche. The README puts it as almost everything remotely useful goes.
That makes the target audience fairly specific. It is for someone who already writes Stable-Baselines3 training loops and needs one algorithm that the main package does not provide. It is not aimed at someone choosing an RL framework from scratch, and it is not aimed at someone who wants a curated set of well-settled implementations. The package name carries the warning: the code is described as experimental, and the README frames the goal as keeping the simplicity, documentation and style of stable-baselines3 for less matured implementations. Style consistency is the promise. Maturity is explicitly not.
What ships in the package, algorithm by algorithm
The README lists seven algorithms and one wrapper. On the algorithm side: Augmented Random Search (ARS), Quantile Regression DQN (QR-DQN), PPO with invalid action masking (MaskablePPO), PPO with a recurrent policy (RecurrentPPO, also called PPO LSTM), Truncated Quantile Critics (TQC), Trust Region Policy Optimization (TRPO), and CrossQ, described as batch normalization in deep reinforcement learning. Each entry links to a paper.
These are not variations on one theme. ARS is a derivative-free method, so it does not train a policy network with backpropagation at all. QR-DQN and TQC are distributional value-based methods, with TQC aimed at continuous control. MaskablePPO and RecurrentPPO are both PPO, but they change different parts of it: one consumes an action mask supplied by the environment, the other replaces the feedforward policy with a recurrent one and therefore changes how rollouts are stored and replayed. TRPO is the trust-region predecessor that PPO approximates. CrossQ is the most recent addition and the one whose inclusion most clearly fits the experimental framing.
The wrapper list is short: a Time Feature Wrapper, which the README links to a paper on time as an input feature. One wrapper against seven algorithms is worth noting, because it tells you where the contribution pressure has been.
Why the algorithms live here and not upstream
The README gives the reason directly: some utilities proved too difficult to integrate well into the existing code without creating a mess. That is a design statement about the main repository as much as about this one. Stable-Baselines3 maintains a common interface across algorithms (the learn, predict and save/load surface, plus shared buffers, policies and callbacks). Anything that breaks an assumption in that shared machinery is expensive to merge.
RecurrentPPO is the clearest example of the pattern. A recurrent policy needs the rollout buffer to carry hidden states and needs sequence-aware minibatch sampling, which is a different data path from the one a feedforward PPO uses. MaskablePPO needs the action distribution to be renormalized over legal actions, and it needs the mask to reach the policy at both rollout and update time. Both changes touch shared code. Putting them in a separate package lets the shared code stay as it is.
The trade-off is real. Because sb3-contrib tracks Stable-Baselines3 rather than forking it, the two packages have to be installed together and kept in step. The README recommends using the master version of Stable-Baselines3, and gives a separate command for installing that master build. That recommendation is a hint about how closely coupled the two codebases are.
Installing it and keeping the two packages aligned
The README gives two installation paths. The released package:
pip install sb3-contrib
And the development version:
pip install git+https://github.com/Stable-Baselines-Team/stable-baselines3-contrib
The README also recommends using the master version of Stable-Baselines3, installed with:
pip install git+https://github.com/DLR-RM/stable-baselines3
That last step matters more than it looks. If you install the released sb3-contrib against a Stable-Baselines3 version it was not built for, you are outside the configuration the maintainers describe. The release history shows why: v2.9.0 is titled Updated dependencies, v2.8.0 carries a fix for MaskablePPO plus a documentation change, and v2.7.1 fixes a TensorBoard log name. A package whose patch releases are dependency bumps and logging-name fixes is one where the coupling to its host library is the main source of churn. Pin both packages together in whatever environment file you use, and re-check the pairing on every upgrade rather than letting a resolver decide.
The README does not document a configuration file, environment variables, or CLI entry points. Configuration happens in Python, through the constructor arguments of whichever algorithm class you import, which is consistent with how Stable-Baselines3 itself works.
The experimental label is not decoration
The README's own framing should be taken at face value. The package exists so that less matured implementations can be published under a consistent style, and the maintainers state that they do not set limits on what is too niche. That is a reasonable policy for a research-adjacent project, and it has a direct consequence for anyone depending on it: the support contract is thinner than the one you get from Stable-Baselines3. An algorithm here may be correct, documented, and still be the least-exercised code in your training stack.
There is a second, more practical failure mode. Because these algorithms are not in the main repository, they are also not covered by the main repository's assumptions about API stability. When Stable-Baselines3 changes a shared component, sb3-contrib has to follow, and the release titles suggest that following is most of what releases consist of. If your project pins Stable-Baselines3 for reproducibility, you may find yourself unable to move to an sb3-contrib version that fixes something you need, or unable to move Stable-Baselines3 without breaking sb3-contrib.
Where it is the wrong tool: if you need a single, stable, well-trodden PPO or SAC implementation and nothing more, sb3-contrib adds a dependency and a version-alignment chore for no benefit. Use Stable-Baselines3 directly in that case.
How this compares with implementing the algorithm yourself
The obvious alternative is not another package. It is taking the paper sb3-contrib links to and writing the algorithm on top of Stable-Baselines3 yourself, subclassing the existing policy and algorithm classes. That is what the contributors to this repository did, and the README's history of community contributions suggests it is a common path.
The difference in approach is where the maintenance sits. A local implementation is yours: you control the update rule, you can strip it to exactly the environment you care about, and nothing breaks when Stable-Baselines3 releases a new version, because you are the one who decides when to rebase. In exchange, you own the correctness question. For something like MaskablePPO, where the mask has to be applied consistently during both rollout collection and the update, a subtle mistake produces training that runs without error and learns badly, which is a costly kind of bug.
sb3-contrib makes the opposite trade. You inherit an implementation that follows the repository's style and documentation conventions, and that has been reviewed as a contribution, but you also inherit the version coupling and the experimental status. Neither option is strictly better. If the algorithm is central to your work and you expect to modify it, the local route is defensible. If you need it to work this week and you are willing to track releases, sb3-contrib is the shorter path.
Licence, maintenance and what to check before you commit
The repository is MIT licensed, which is permissive and imposes few conditions on reuse. The README adds a citation request that is separate from the licence: it asks that publications cite Stable-Baselines3 rather than this repository, and provides a BibTeX entry for the Journal of Machine Learning Research paper. That is a request, not a licence term, but it tells you how the maintainers view the relationship between the two projects. This is not legal advice; if licence terms matter to your organisation, read the LICENSE file and the SB3 licence yourself.
On maintenance, the observable signals are the release cadence and the release contents. The last push recorded is 2026-07-24 and the repository is not archived. Recent releases are small and mostly corrective: a dependency update, a MaskablePPO fix, a TensorBoard log-name fix. That pattern is consistent with a project that is maintained but not rapidly expanding, and it means a bug you hit in one of the less-used algorithms may sit for a while.
The first thing to verify is version alignment between sb3-contrib and Stable-Baselines3 in your environment, since the README's own recommendation is to run both from master. The second is the documentation page for the specific algorithm you plan to use, at sb3-contrib.readthedocs.io, because the README lists names and papers without describing per-algorithm constraints. The third is whether the algorithm you need is still only available here; if it has since landed upstream, the version-coupling cost disappears and you should take the upstream copy instead.
Editorial conclusion
Adopt sb3-contrib if you need invalid action masking, a recurrent PPO policy, or distributional value learning and you already run Stable-Baselines3, because the algorithms share the SB3 API. Do not adopt it as a general-purpose RL framework or as a source of production-stable training code, and do not expect the maintainers to treat a regression in CrossQ the way they would treat one in PPO. Before committing, check that the sb3-contrib release you install matches the Stable-Baselines3 version you already have, and read the algorithm page for the specific class you intend to use rather than the package README, which lists names without describing their constraints.
Community notes