MORL-Baselines: multi-objective reinforcement learning algorithms in PyTorch

What multi-objective RL means here, and who this is for

MORL-Baselines is a library of Multi-Objective Reinforcement Learning algorithms, and the repository states that it aims to contain reliable MORL algorithms implementations in PyTorch. The word reliable is doing work, and the rest of the README explains what reliability means to the project.

The features list claims that performances are automatically reported in Weights and Biases dashboards, that linting and formatting are enforced by pre-commit hooks, that all algorithms are automatically tested, that utility functions are provided such as pareto pruning and experience buffers, and that performances have been tested and reported in a reproducible manner. Hyperparameter optimization is listed as available. The project status badge says Active.

The subject matter is the difference from ordinary RL. In a single-objective setting the agent maximises one reward, so there is one right policy. In a multi-objective setting the agent maximises a vector of rewards, and there is generally no single policy that is best on every component, so what you compute is a set of trade-off policies, usually called a Pareto front. The README points to a practical guide to multi-objective reinforcement learning and planning for the definitions, and to a taxonomy paper on decomposition for the techniques the algorithms use.

The target reader is a researcher in reinforcement learning, or a practitioner in robotics, resource allocation or any setting with genuinely conflicting objectives, who needs a method they can trust to match the published algorithm and to reproduce the published numbers. The package metadata classifies the audience as Science/Research and the topic as Artificial Intelligence.

There is also a Colab tutorial for MO-Gymnasium and MORL-Baselines, which is the fastest way to see an algorithm working without setting up a local environment.

One API difference from Gymnasium, and it is the reward type

MORL-Baselines strictly follows the MO-Gymnasium API, and the README is precise about how that differs from standard Gymnasium. It differs in exactly one respect: the environment returns a numpy array as the reward, instead of a scalar.

That single change is worth dwelling on, because it propagates. Every algorithm in the library has to handle a vector of rewards rather than a number, which means the value estimates are vectors, the return computation is a vector operation, and the policy update has to decide how to trade components off. The clean RL code you already know will not transfer, and the library exists because getting that right in each algorithm is the bulk of the work.

The dependency list shows the stack the library sits on. `mo-gymnasium >=1.3.0` and `gymnasium>=1.0.0` provide the environments and the standard API, `torch >=1.12.0` is the deep learning framework, and `pymoo >=0.6.0` handles the multi-objective optimisation that the decomposition-based methods need. `scipy >=1.7.3` covers the numerical work, `wandb >=0.14.0` is the experiment tracker, and `imageio` and `moviepy` are there to record and render the environments.

Two dependencies deserve a note. `cvxpy` is present, and `pycddlib==2.1.6` is pinned exactly. The comment on the optional dependency group says that OLS requires pycddlib and libgmp to be installed, which does not work on macOS for now. So the library has at least two algorithms that will not install on a Mac, which is a real constraint for a research group with mixed laptops.

The algorithm table: single-policy, multi-policy, SER and ESR

The implemented algorithms are presented in a table, and the columns are the ones that matter for understanding the field.

The first column is single-policy versus multi-policy. A single-policy method produces one policy for a given preference or weight vector over the objectives, while a multi-policy method trains a set of policies that together cover the trade-off space, so you can query the set afterwards for the policy that suits your priorities. The second column is the criterion under which the method is optimal: SER, the expected scalarised return criterion, and ESR, the expected scalarised return with a chance constraint on the worst case.

Looking at what is in the table, the multi-policy column dominates. GPI-LS plus GPI-PD is multi-policy under SER, with continuous observations and discrete or continuous actions. The Jax implementation of GPI-LS is also there, multi-policy under SER. MORL/D is multi-policy under both SER and ESR, and it handles discrete or continuous observations and actions. Envelope Q-Learning, CAPQL and PGMORL are all multi-policy under SER, and Pareto Conditioned Networks appear under both SER and ESR.

The papers are linked for each entry, with supplementary documents where they exist, so the mapping from implementation to publication is direct. That is the core value of the repository. Reimplementing MORL/D from its paper, with a decomposition method and a conditional expectation network, is a research project in itself, and having a maintained PyTorch version with tests attached changes what a paper reproduction costs.

The images and the animation in the README show a cheetah running with multiple policies, which is the visual shorthand for this whole family of methods: one behaviour, several trade-offs.

Hyperparameter sweeps, W&B dashboards and the experiment workflow

The features list mentions hyperparameter optimization, and the repository layout shows how that is wired.

There is an `experiments/` directory and an `examples/` directory, and the example files show the granularity of the work. There are examples for Envelope Q-Learning on the Minecart environment, for MORL/D on the Cheetah, Hopper and Lunar Lander, for GPI-PD on the Hopper and Minecart, for PGMORL on HalfCheetah, for Pareto Conditioned Networks on Minecart, for Lorenz Conditioned Networks on Fruit Tree, and for the discrete and multi-policy Q-Learning variants on the Deep Sea Treasure environment. Two files are notable for a different reason: `morld_lunar_lander_restore.py` and `ols_dst.py`, which suggest that saving and restoring, and the OLS method on Deep Sea Treasure, each got their own example.

Each of those environments is a MO-Gymnasium environment, and the naming convention is informative on its own. Deep Sea Treasure is the classic small multi-objective benchmark, Minecart is a discrete multi-objective navigation problem, HalfCheetah and Hopper are the continuous control tasks, and Fruit Tree is the sparse multi-objective case. Seeing which algorithm ships with which environment is how you decide whether the library supports the setting you are working in.

The Weights and Biases integration is what makes the results comparable across runs. With wandb in the dependency list and the claim that performances are automatically reported in dashboards, a multi-objective study involves many runs, one per weight vector or per policy, and the ability to plot a Pareto front across them is the practical point. A study that cannot be re-plotted is a study that cannot be revised.

The other piece of tooling is the `docs/` directory, with a static images folder holding the cheetah animation, and the homepage at lucasalegre.github.io/morl-baselines.

Packaging, the extra that decides whether your algorithm runs

The packaging is conventional setuptools with a `pyproject.toml`, and there is a `setup.py` left in the tree that still reads the version out of the source.

The `setup.py` does something worth noting: `get_version()` opens `morl_baselines/__init__.py`, scans for the line starting with `__version__`, and takes the last token after splitting on whitespace and stripping quotes. If no such line exists it raises a RuntimeError with the message bad version data in __init__.py. The `pyproject.toml` also declares a dynamic version. The practical consequence is that the version lives in exactly one place in the source tree, and the packaging reads it from there.

The optional dependency groups are where the practical decisions are. As declared in the packaging metadata, they look like this:

[project.optional-dependencies]
ols = ["pycddlib==2.1.6"]
gpi = ["pycddlib==2.1.6"]
jax = ["jax>=0.8.2", "flax>=0.12.2", "optax", "orbax-checkpoint"]

There is an `ols` extra and a `gpi` extra, each holding `pycddlib==2.1.6`, which reflects the comment that OLS requires pycddlib and libgmp and that this does not work on macOS for now. The `jax` extra is what the Jax implementation of GPI-LS needs. There is an `all` extra that the comment says must be updated if any dependency is added or removed, and it collects the OLS and GPI requirements plus the Jax ones. And there is a `testing` extra pinning `pytest ==7.1.3` exactly.

That pin on pytest is a compatibility constraint, not a style choice, and it is a hint about the state of the project: a library that has to pin its test runner to a specific old version is a library whose dependencies have not been re-evaluated recently.

The project requires Python 3.8 or later, and the classifiers list 3.8 through 3.12.

Where MORL-Baselines is the wrong choice

The first wrong-tool case is a single-objective problem. If your reward is a scalar, all the machinery here, the vector returns, the Pareto fronts, the decomposition, the conditional expectation networks, is overhead with no payoff. Standard single-objective RL libraries do that job.

The second case is an algorithm the repository does not implement, or one you need a variant of. The table is long but it is a list of specific published methods, and a research contribution usually means modifying one of them. Once you start changing the algorithm, you are maintaining your own fork, and the value of the library drops to the parts you did not touch. The repository describes itself as a set of baselines, and baselines are meant to be compared against, not edited.

The third is the platform constraint. The OLS and GPI extras depend on pycddlib and libgmp, and the project states this does not work on macOS for now. A Mac-only research group cannot use those methods, and the Jax extra adds a second large dependency stack with its own version floor.

The fourth is production use. The development status is Beta, with a comment in the packaging metadata saying to change it to Production/Stable when ready. The library is written for research reproducibility, and the pre-commit, black and isort configuration, the pinned pytest, and the Weights and Biases defaults all point at an academic workflow rather than a service.

The alternative worth naming is the algorithm's own reference implementation, when the authors published one, since most MORL papers of this kind ship code. The difference in approach is that an official implementation is the ground truth for the paper, including the details the paper does not mention, and it is the right thing to compare against when you are checking a reimplementation. MORL-Baselines gives you a consistent PyTorch codebase where every method has been brought to the same API, tested and instrumented, which is what you want when you are comparing methods against each other rather than reproducing a single paper.

Release cadence and what the version history says

The release history has three entries in the visible list, and the spacing between them is informative.

Version 1.1.0 was released on 2024-10-28, and its release note spells out its contents: Gymnasium 1.0, MORL/D, HPO, bug fixes and improvements. Version 1.2.0 came on 2025-08-06, and version 1.3.0 on 2026-06-26, named MORL Baselines Release v1.3.0. The last push to the default branch was on 2026-09-21, and the repository is not archived.

So the project moved from a beta release that introduced Gymnasium 1.0 compatibility, a new algorithm and hyperparameter optimization, through a year of work, to a 1.3 release, and has been receiving commits since. That is a project on a roughly annual release cadence with continuous work between releases, and the last push being within a month of today's date indicates it is still being worked on.

The v1.1.0 release note is the one to read if you are upgrading from an older environment, because the Gymnasium 1.0 change is a breaking change in the ecosystem this library sits in, and a MORL environment that returns a numpy array of rewards has to be checked against the current Gymnasium behaviour.

The licence is MIT, with a `LICENSE` file in the repository and the metadata recording MIT License. For a research library that academic and industrial groups both build on, that permissive choice matters, since a copyleft algorithm collection would be awkward to vendor into a commercial codebase. This is a description of the published terms and not legal advice.

The tree also carries a `CITATION.bib` file, which is the piece to copy if you publish results built on these implementations, and a Discord badge for support alongside the pre-commit and code style badges.

Editorial conclusion

Adopt MORL-Baselines when you need a reference implementation of a published multi-objective reinforcement learning method, when you are working inside the MO-Gymnasium ecosystem, and when you want a hyperparameter sweep and a Weights and Biases dashboard rather than a bare algorithm file. Do not adopt it for a single-objective control problem, where the added machinery of a Pareto front buys nothing, and do not treat it as a production framework, since the package classifies itself as Beta. Before you build on it, check the optional dependencies your chosen algorithm needs, because the OLS and GPI extras require `pycddlib` and libgmp, which the project notes does not work on macOS, and confirm that the `jax` extra at `jax>=0.8.2` is compatible with your environment if you plan to use the Jax implementation of GPI-LS.

Frequently asked questions

Official sources

1. License: MIT
2. LucasAlegre/morl-baselines on GitHub
3. Project website
4. README
5. Releases