MushroomRL: a modular Python RL library that treats the environment as a plug-in
Python library for Reinforcement Learning.
At a glance
- What is it?
- MushroomRL separates agents, policies, environments and replay buffers so that a Q-Learning run and a SAC run share the same experiment skeleton. The trade-off is that you assemble the pieces yourself, and the 2.0 line is still at release-candidate stage.
- Who is it for?
- Adopt MushroomRL if you want classical and deep RL algorithms behind one interface and you are willing to wire agents, policies, environments and replay buffers together in a script. Do not adopt it if you need a single end-to-end trainer that reads a config file, or if you cannot tolerate a release candidate in your dependency set.
- Can I use it commercially?
- Yes. MIT is a permissive licence: you can use, modify and sell software built on it, as long as you keep its copyright and licence notices.
- Is it still maintained?
- Yes. The repository last received commits 1 day ago.
- What is it written in?
- Mainly Python, according to GitHub's language statistics.
Answers come from the project's GitHub data, last synced on September 15, 2026, and from our analysis. They are not legal advice.
DEEP OPEN-SOURCE ANALYSIS
The problem MushroomRL addresses: one experiment skeleton, many algorithms
Reinforcement learning code tends to be written twice. A tabular Q-Learning script and a SAC script often share nothing beyond a loop, even though both need an environment, a policy, a replay buffer and a training schedule. MushroomRL's stated aim is to remove that duplication. The README describes it as a library "whose modularity allows to easily use well-known Python libraries for tensor computation (e.g. PyTorch, Tensorflow) and RL benchmarks (e.g. Gymnasium, PyBullet, Deepmind Control Suite)". The audience is research-oriented: the project publishes a JMLR citation, the examples live under a papers folder, and the maintainers ask for an email or a Twitter mention rather than a Discord invite. If you are comparing algorithms on the same task and want to change one component at a time, that is the intended workflow. If you want a training script that produces a checkpoint with no further thought, this is not the shape of the library.
How the pieces fit together: agents, policies, environments, replay buffers
The architecture is component-based. An agent holds a policy and consumes environment transitions; the environment can be a Gymnasium task, an Atari 2600 game through the Arcade Learning Environment, or a physics simulator such as PyBullet or MuJoCo. Because those integrations are optional packages rather than hard dependencies, the environment layer is genuinely swappable, and the same agent code can be pointed at a different simulator. The algorithm list spans two generations: classical methods (Q-Learning, SARSA, FQI) and deep methods (DQN, DDPG, SAC, TD3, TRPO, PPO). The repository topics confirm the same set, adding pytorch, qlearning and mujoco. What the README does not describe is the internal data flow: it does not say how a replay buffer is attached to an agent, whether policies are registered by name, or how an experiment is serialized. You have to read the examples folder and the Read the Docs pages to learn that, because the README points there instead of explaining it. That is a documentation gap worth knowing about before you start.
Installing MushroomRL and its optional environment stack
The minimal install is a single command: pip install mushroom-rl. That gets you the core library without Gymnasium, Atari or the physics simulators, because the README states that "support for these classes is not enabled by default". To pull in the optional components, run pip install mushroom-rl[all]. The README is explicit that this still excludes Box2D and PyBullet, so [all] is not literally everything. Live monitor support is a separate extra: pip install mushroom-rl[monitors]. On Ubuntu above 20.04 the README lists system packages needed for pygame and gym, including libsdl-image1.2-dev, libsdl-mixer1.2-dev, libsdl-ttf2.0-dev, libsdl1.2-dev, libsmpeg-dev, libportmidi-dev, ffmpeg, libswscale-dev, libavformat-dev, libavcodec-dev and swig. It notes that some of these are needed on other operating systems too, naming swig for macOS. A local editable install uses pip install -e . or pip install -e .[all]. To check that the pieces work together, the README gives one command: python3 examples/papers/car_on_hill_fqi.py. That example is a classical FQI run, so it exercises the core library without the optional simulator stack.
Version 2.0 is a release candidate, and that shapes your dependency pins
The recent release history is the most concrete constraint in the material. 2.0.0-rc2 was published on 2026-07-24 and 2.0.0-rc1 on 2026-06-18. The last stable release listed is 1.10.2 from 2025-04-14. The default branch is dev, not main or master, and the README's documentation and CI badges both point at that dev branch. Taken together, this says the project is in an active transition between the 1.x and 2.x lines, and that the branch you read about is the one under development. For a research repository that is normal. For a pipeline that pins dependencies and expects stable APIs, it means you should decide deliberately between the 1.10.2 line and the 2.0 release candidates, and re-read the release notes before moving. The README does not include a migration guide for 1.x to 2.x, so if you are upgrading an existing experiment you should expect to consult the documentation site rather than the README.
Where MushroomRL is the wrong tool
The library gives you components, not a pipeline. There is no command that takes a config file and trains an agent; the README says experiments require "a script file that provides the necessary information for the experiment". You write that script, and you choose which agent, policy, environment and buffer to combine. That is a real cost if your goal is a reproducible training run rather than a comparison between algorithms. The optional-dependency story is the second limitation. Installing [all] does not install Box2D or PyBullet, and the Linux system packages for pygame and gym sit outside pip entirely, which means a container image needs both a Python layer and an apt layer to cover the same ground. Third, the README does not state supported Python versions, so you cannot confirm compatibility from it alone. Fourth, the entry point for help is an email address and a Twitter account, not an issue tracker or a chat server, which is a slower loop if you hit a bug mid-experiment.
How this differs from Stable-Baselines3
Stable-Baselines3 is the obvious comparison point and the difference is in where the abstraction sits. Stable-Baselines3 wraps algorithms as trainer objects with a small, stable set of methods, so a PPO run and a SAC run look nearly identical and the environment interface is the main thing you adapt. MushroomRL keeps the agent, the policy, the environment and the replay buffer as separate objects you compose yourself, and it covers the classical algorithms (Q-Learning, SARSA, FQI) alongside the deep ones. That combination is unusual: Stable-Baselines3 is a deep RL library, so a tabular experiment and a deep experiment do not share a codebase there. The cost of MushroomRL's approach is that more of the wiring is yours, and the benefit is that swapping a PyTorch policy for a TensorFlow one, or a Gymnasium environment for a MuJoCo one, is a substitution rather than a rewrite. If your work is one algorithm on one benchmark, the extra composition buys you little.
Licence, maintenance and upgrade cost
MushroomRL is MIT licensed, which permits commercial and closed-source use and requires only that the licence notice and copyright line travel with the code. The project asks for a citation when you use it in scientific publications, and provides the BibTeX entry for the 2021 JMLR paper by D'Eramo, Tateo, Bonarini, Restelli and Peters. That citation request is a norm of the community, not a licence condition, so it does not restrict what you can build. The maintenance signal in the material is the CI and documentation badges pointing at the dev branch, plus the qlty.sh maintainability and coverage badges, which indicate that automated checks run on the repository. What the material does not tell you is how many maintainers there are, how often releases ship, or when 2.0.0 will lose the rc suffix. The upgrade cost is concentrated in the 1.x to 2.x transition: two release candidates in roughly five weeks, a dev default branch, and no migration notes in the README. Budget time to read the release notes and the documentation site rather than assuming the API you learned on 1.10.2 carries over.
Editorial conclusion
Adopt MushroomRL if you want classical and deep RL algorithms behind one interface and you are willing to wire agents, policies, environments and replay buffers together in a script. Do not adopt it if you need a single end-to-end trainer that reads a config file, or if you cannot tolerate a release candidate in your dependency set. Before committing, run python3 examples/papers/car_on_hill_fqi.py on your machine, confirm which optional extras your environments need, and check whether 2.0.0-rc2 or the 1.10.2 line matches your dependency pins.
Community notes