# MushroomRL: a modular Python RL library that treats the environment as a plug-in

> MushroomRL separates agents, policies, environments and replay buffers so that a Q-Learning run and a SAC run share the same experiment skeleton. The trade-off is that you assemble the pieces yourself, and the 2.0 line is still at release-candidate stage.

**MushroomRL/mushroom-rl** — Python library for Reinforcement Learning.

- Repository: https://github.com/MushroomRL/mushroom-rl
- Stars: 945 · Forks: 164
- Language: Python
- License: MIT
- Published: 2026-09-10 · Updated: 2026-09-10 · Language: en
- Canonical page: https://hysenlabs.com/projects/mushroomrl-mushroom-rl

## The problem MushroomRL addresses: one experiment skeleton, many algorithms

Reinforcement learning code tends to be written twice. A tabular Q-Learning script and a SAC script often share nothing beyond a loop, even though both need an environment, a policy, a replay buffer and a training schedule. MushroomRL's stated aim is to remove that duplication. The README describes it as a library "whose modularity allows to easily use well-known Python libraries for tensor computation (e.g. PyTorch, Tensorflow) and RL benchmarks (e.g. Gymnasium, PyBullet, Deepmind Control Suite)". The audience is research-oriented: the project publishes a JMLR citation, the examples live under a papers folder, and the maintainers ask for an email or a Twitter mention rather than a Discord invite. If you are comparing algorithms on the same task and want to change one component at a time, that is the intended workflow. If you want a training script that produces a checkpoint with no further thought, this is not the shape of the library.

## How the pieces fit together: agents, policies, environments, replay buffers

The architecture is component-based. An agent holds a policy and consumes environment transitions; the environment can be a Gymnasium task, an Atari 2600 game through the Arcade Learning Environment, or a physics simulator such as PyBullet or MuJoCo. Because those integrations are optional packages rather than hard dependencies, the environment layer is genuinely swappable, and the same agent code can be pointed at a different simulator. The algorithm list spans two generations: classical methods (Q-Learning, SARSA, FQI) and deep methods (DQN, DDPG, SAC, TD3, TRPO, PPO). The repository topics confirm the same set, adding pytorch, qlearning and mujoco. What the README does not describe is the internal data flow: it does not say how a replay buffer is attached to an agent, whether policies are registered by name, or how an experiment is serialized. You have to read the examples folder and the Read the Docs pages to learn that, because the README points there instead of explaining it. That is a documentation gap worth knowing about before you start.

## Installing MushroomRL and its optional environment stack

The minimal install is a single command: pip install mushroom-rl. That gets you the core library without Gymnasium, Atari or the physics simulators, because the README states that "support for these classes is not enabled by default". To pull in the optional components, run pip install mushroom-rl[all]. The README is explicit that this still excludes Box2D and PyBullet, so [all] is not literally everything. Live monitor support is a separate extra: pip install mushroom-rl[monitors]. On Ubuntu above 20.04 the README lists system packages needed for pygame and gym, including libsdl-image1.2-dev, libsdl-mixer1.2-dev, libsdl-ttf2.0-dev, libsdl1.2-dev, libsmpeg-dev, libportmidi-dev, ffmpeg, libswscale-dev, libavformat-dev, libavcodec-dev and swig. It notes that some of these are needed on other operating systems too, naming swig for macOS. A local editable install uses pip install -e . or pip install -e .[all]. To check that the pieces work together, the README gives one command: python3 examples/papers/car_on_hill_fqi.py. That example is a classical FQI run, so it exercises the core library without the optional simulator stack.

## Version 2.0 is a release candidate, and that shapes your dependency pins

The recent release history is the most concrete constraint in the material. 2.0.0-rc2 was published on 2026-07-24 and 2.0.0-rc1 on 2026-06-18. The last stable release listed is 1.10.2 from 2025-04-14. The default branch is dev, not main or master, and the README's documentation and CI badges both point at that dev branch. Taken together, this says the project is in an active transition between the 1.x and 2.x lines, and that the branch you read about is the one under development. For a research repository that is normal. For a pipeline that pins dependencies and expects stable APIs, it means you should decide deliberately between the 1.10.2 line and the 2.0 release candidates, and re-read the release notes before moving. The README does not include a migration guide for 1.x to 2.x, so if you are upgrading an existing experiment you should expect to consult the documentation site rather than the README.

## Where MushroomRL is the wrong tool

The library gives you components, not a pipeline. There is no command that takes a config file and trains an agent; the README says experiments require "a script file that provides the necessary information for the experiment". You write that script, and you choose which agent, policy, environment and buffer to combine. That is a real cost if your goal is a reproducible training run rather than a comparison between algorithms. The optional-dependency story is the second limitation. Installing [all] does not install Box2D or PyBullet, and the Linux system packages for pygame and gym sit outside pip entirely, which means a container image needs both a Python layer and an apt layer to cover the same ground. Third, the README does not state supported Python versions, so you cannot confirm compatibility from it alone. Fourth, the entry point for help is an email address and a Twitter account, not an issue tracker or a chat server, which is a slower loop if you hit a bug mid-experiment.

## How this differs from Stable-Baselines3

Stable-Baselines3 is the obvious comparison point and the difference is in where the abstraction sits. Stable-Baselines3 wraps algorithms as trainer objects with a small, stable set of methods, so a PPO run and a SAC run look nearly identical and the environment interface is the main thing you adapt. MushroomRL keeps the agent, the policy, the environment and the replay buffer as separate objects you compose yourself, and it covers the classical algorithms (Q-Learning, SARSA, FQI) alongside the deep ones. That combination is unusual: Stable-Baselines3 is a deep RL library, so a tabular experiment and a deep experiment do not share a codebase there. The cost of MushroomRL's approach is that more of the wiring is yours, and the benefit is that swapping a PyTorch policy for a TensorFlow one, or a Gymnasium environment for a MuJoCo one, is a substitution rather than a rewrite. If your work is one algorithm on one benchmark, the extra composition buys you little.

## Licence, maintenance and upgrade cost

MushroomRL is MIT licensed, which permits commercial and closed-source use and requires only that the licence notice and copyright line travel with the code. The project asks for a citation when you use it in scientific publications, and provides the BibTeX entry for the 2021 JMLR paper by D'Eramo, Tateo, Bonarini, Restelli and Peters. That citation request is a norm of the community, not a licence condition, so it does not restrict what you can build. The maintenance signal in the material is the CI and documentation badges pointing at the dev branch, plus the qlty.sh maintainability and coverage badges, which indicate that automated checks run on the repository. What the material does not tell you is how many maintainers there are, how often releases ship, or when 2.0.0 will lose the rc suffix. The upgrade cost is concentrated in the 1.x to 2.x transition: two release candidates in roughly five weeks, a dev default branch, and no migration notes in the README. Budget time to read the release notes and the documentation site rather than assuming the API you learned on 1.10.2 carries over.

## Conclusion

Adopt MushroomRL if you want classical and deep RL algorithms behind one interface and you are willing to wire agents, policies, environments and replay buffers together in a script. Do not adopt it if you need a single end-to-end trainer that reads a config file, or if you cannot tolerate a release candidate in your dependency set. Before committing, run python3 examples/papers/car_on_hill_fqi.py on your machine, confirm which optional extras your environments need, and check whether 2.0.0-rc2 or the 1.10.2 line matches your dependency pins.

## FAQ

### Does a minimal MushroomRL install pull in PyTorch?

Yes. The unconditional dependency list in pyproject.toml includes torch, opencv-python>=4.7, av, pygame-ce, scikit-learn, scipy, matplotlib, joblib, and tqdm, so the command presented as a minimal installation still installs a deep learning stack. Python 3.11 or newer is required.

### What does pip install mushroom-rl[all] leave out?

Box2D and PyBullet. setup.py builds the all group by walking the extras dictionary and skipping the box2d and bullet names, and the page repeats that those two are not installed even by the full install, without naming a separate command for either.

### Which optional dependency groups does MushroomRL define?

Ten: gymnasium, atari (ale-py, Pillow, opencv-python), minigrid, box2d (box2d-py), bullet (pybullet), mujoco (mujoco>=3.11), mujoco_warp, dm_control (dm_control>=1.0.44), monitors (pyqtgraph, PySide6), and wandb. Only the monitors group gets its own install block on the page.

### How does MushroomRL determine its own version?

It is declared dynamic in pyproject.toml and comes from mushroom_rl.__version__, which setup.py imports at build time. That import also means the package is loaded during a source build, and the script prints the resolved all dependency group to standard output at the same time.

### How do you run a MushroomRL experiment?

By writing a script file that carries the experiment parameters. The page points at the examples folder, which is split into algorithms, data, environments, papers, and tools, and gives one command, python3 examples/papers/car_on_hill_fqi.py, with no flags or environment variables involved.

### What is the state of the MushroomRL 2.0 release?

Two candidates exist, 2.0.0-rc1 from 2026-06-18 and 2.0.0-rc2 from 2026-07-24, following 1.10.2 from 2025-04-14. No 2.0.0 final has been tagged, the default branch is dev, and the last push on record is 2026-09-30.

## Sources

- [Issues](https://github.com/MushroomRL/mushroom-rl/issues)
- [License: MIT](https://github.com/MushroomRL/mushroom-rl/blob/dev/LICENSE)
- [MushroomRL/mushroom-rl on GitHub](https://github.com/MushroomRL/mushroom-rl)
- [README](https://github.com/MushroomRL/mushroom-rl/blob/dev/README.md)
- [Releases](https://github.com/MushroomRL/mushroom-rl/releases)

---

Hysen Labs editorial analysis, written from the project's own repository and release notes. Cite the canonical page: https://hysenlabs.com/projects/mushroomrl-mushroom-rl