MO-Gymnasium: a Gymnasium API where the reward is a vector

One change to the Gymnasium API, and everything it implies

MO-Gymnasium is an open source Python library for developing and comparing multi-objective reinforcement learning algorithms. It provides a standard API to communicate between learning algorithms and environments, as well as a standard set of environments compliant with that API.

The mechanism is described precisely: the environments follow the standard Gymnasium API, but return vectorized rewards as numpy arrays. That is the entire delta, and it is smaller than it sounds.

The importance is on the research side rather than the engineering side. In single-objective reinforcement learning, the environment returns a scalar reward and the algorithm is written against that. A multi-objective setting has several objectives per step, and there is no agreed convention for how to return them. Some libraries return a dict, some return a list, some return the objectives concatenated with the scalar reward. Without a shared convention, every algorithm implementation has to be adapted to every environment, and a benchmark comparison across papers becomes a comparison across data formats as much as across algorithms.

MO-Gymnasium fixes the convention, and the payoff is that an algorithm written against the Gymnasium API can be dropped into any environment in the set without modification. The project describes its purpose as being about comparing algorithms, and that is the feature that makes the comparison fair.

The environment set itself draws on two sources. It includes environments taken from the MORL literature, so the classic benchmark problems are there, as well as multi-objective versions of classical environments such as MuJoCo, so that a multi-objective result can be compared against the single-objective literature on the same simulator.

The API in one snippet

The usage example in the documentation uses the `minecart-v0` environment and is short enough to hold in your head:

import gymnasium as gym
import mo_gymnasium as mo_gym
import numpy as np

# It follows the original Gymnasium API ...
env = mo_gym.make('minecart-v0')

obs, info = env.reset()
# but vector_reward is a numpy array!
next_obs, vector_reward, terminated, truncated, info = env.step(your_agent.act(obs))

The `make` call is the Gymnasium constructor with a MO-Gymnasium environment id, `reset` returns the usual observation and info pair, and `step` returns the usual five element tuple. The only thing that changes is the second element of that tuple, which is a numpy array of per-objective rewards instead of a float.

That means the standard Gymnasium wrapper and vectorization machinery applies, and the `gymnasium` import in the example is there because the types come from there.

The second half of the example shows a wrapper that addresses the obvious follow-up question. If you have a single-objective algorithm and you want to run it in a multi-objective environment, you need a way to collapse the vector into a scalar:

# Optionally, you can scalarize the reward function with the LinearReward wrapper
env = mo_gym.wrappers.LinearReward(env, weight=np.array([0.8, 0.2, 0.2]))

A weight vector of one entry per objective, and the wrapper does the dot product. This is the simplest possible scalarization, and it is also the most contestable one, since a fixed weight vector decides the trade-off between objectives before the algorithm has any say. It is a baseline and a debugging tool rather than a solution, and the wrapper being available in the standard package is a good sign for a library that wants to be usable rather than ideological.

For the theory behind the vector rewards, the documentation points to a practical guide to multi-objective reinforcement learning and planning, and there is a colab notebook demo in the repository for people who want to run the example without installing anything.

Installing one family at a time, because some are awkward

The base install is one command:

pip install mo-gymnasium

What that does not do is install the dependencies for every family of environments, and the documentation is explicit that this is deliberate. Some of those dependencies can be problematic to install on certain systems.

Instead you install the extras for the family you need:

pip install "mo-gymnasium[mujoco]"

or all of them at once with `pip install "mo-gymnasium[all]"`.

The optional dependency list in the project manifest shows what those families cost. The `mario` family needs `nes-py` and `gym-super-mario-bros`, and pins numpy below 2.0, which is a good signal of where the awkwardness lies. The `minecart` family needs `scipy >= 1.7.3`. The `mujoco` family needs `mujoco >= 2.2.0` and `imageio >= 2.14.1`. The `highway` family needs `highway-env >= 1.8`. The `box2d` family needs `box2d-py == 2.3.5` and `swig == 4.*`, and the pinned swig version is a hint that this is the one that compiles against system libraries. There is also a `testing` extra pinning `pytest == 7.1.3`.

The base dependencies are modest: `gymnasium >=1.1.0`, `numpy >=1.21` and `pygame-ce >=2.1.3`, with `requires-python` set to 3.11 or newer. The manifest classifies the project as Beta, and the comment in the source says to change it to Production and Stable when ready, which tells you where the maintainers think it sits.

The pin of `gymnasium >=1.1.0` matters in practice, because it sets a floor on the Gymnasium version installed alongside the library. Depending on a floor rather than an exact version is a deliberate choice that keeps the library compatible with a range, at the cost that you have to keep an eye on your Gymnasium version when you upgrade.

Version suffixes exist so old results stay reproducible

MO-Gymnasium keeps strict versioning for reproducibility reasons, and the rule is mechanical. All environments end in a suffix like `-v0`, and when changes are made to environments that might impact learning results, the number is increased by one to prevent potential confusion.

This is a small convention with a large effect on the research literature, and it is worth spelling out why. In reinforcement learning, a result is a number produced by a particular environment version. If the environment changes silently, a paper's reported performance no longer corresponds to the code, and a reader who clones the repository and reruns the experiment gets a different number with no obvious reason. Nobody can tell whether the algorithm got worse or the benchmark moved.

The version suffix makes that impossible to miss. `minecart-v0` and `minecart-v1` are different environments as far as a result is concerned, and the suffix in the environment id tells you which one a number came from.

Compare that with the convention in some other corners of reinforcement learning, where the environment id stays the same and the behaviour drifts, and a benchmark table becomes a set of numbers nobody can reproduce. The version suffix costs one line of change management and removes a whole category of confusion.

The same instinct shows up in the citation block, which the documentation asks you to use if you use the repository in your research. The BibTeX entry points at a NeurIPS 2023 paper on reliable benchmarking in multi-objective reinforcement learning, with the author list given. Asking for a citation of a benchmarking toolkit is a claim that the environments are meant to produce comparable results, and the version suffix is what makes that claim hold.

What the project is not, and how it relates to algorithm implementations

MO-Gymnasium is an environment library, not an algorithm library, and the distinction matters for how you use it.

It does not contain reinforcement learning algorithms. It contains the problems. A multi-objective algorithm you find elsewhere implements the MORL API and runs against these environments. The repository does name a related project, MORL-Baselines, described as a repository containing various implementations of MORL algorithms by the same authors as MO-Gymnasium, which relies on the MO-Gymnasium API and shows various examples of the usage of wrappers and environments.

The relationship is worth reading carefully because it tells you how the projects divide responsibility. MO-Gymnasium owns the environment contract, the wrappers and the versioning policy. MORL-Baselines owns the algorithms and the examples. Neither depends on the other in a way that would lock you in, because the API is documented and the environments are plain Python classes.

What MO-Gymnasium does not do is tell you how to solve a multi-objective problem. There is a whole family of approaches with different answers, some of which produce a set of policies covering the trade-off surface and some of which produce a single policy chosen by scalarization. The `LinearReward` wrapper supports the second approach and the API supports the first, but the choice is the researcher's, and the library's job is to make sure both can be evaluated fairly.

The second limitation is the environment coverage itself. The full list is published on the documentation site rather than in the README, and the environments come from the MORL literature plus multi-objective versions of classical benchmarks. If your research needs a specific domain, and that domain is not a MORL literature classic, it is not here, and writing a conforming environment is the alternative.

The third limitation is the Python version floor. Python 3.11 or newer is required, and the classifiers list 3.11 through 3.14. On a team with an older standard library image, that is a blocker that a virtual environment does not solve.

The project structure and who maintains it

The repository layout is the standard Farama pattern. There is a `mo_gymnasium/` package directory, a `tests/` directory, a `docs/` directory, a `pyproject.toml` and a `setup.py`, a `CITATION.bib` file, a `CODE_OF_CONDUCT.rst` and a `mo_gymnasium_demo.ipynb` notebook at the top level.

The `setup.py` file is worth a look because it explains something about the version. The `get_version` function reads the `mo_gymnasium/__init__.py` file, scans for the line starting with `__version__`, splits it and returns the value. The `pyproject.toml` marks the version as dynamic for the same reason. There is no separate version file to forget to update, which is a small thing that prevents a real class of release mistake.

The rest of the setup file is a set of commented publishing commands, the standard setuptools sdist and bdist wheel pair followed by twine uploads to PyPI, with a note about the different repository URLs for the live and test indexes.

On people, the project lists a project manager, Lucas Alegre, and says maintenance is also contributed by the broader Farama team, with the team page linked. The package is published under the MIT licence with the authors field naming the Farama Foundation.

The development infrastructure is the usual set of badges: a Python versions badge and a PyPI version badge, a tests workflow badge, a pre-commit badge and a black code style badge. Pre-commit with black means the formatting is fixed by tooling rather than by review comment, which matters for a repository where most changes are mechanical.

The last push was on 2026-10-16 and the repository is not archived. The release history shows 1.3.0 and 1.3.1 on the same day in October 2024, and 1.3.2 in February 2026 with a release note mentioning Numpy 2.0 support and a MOAsyncVectorEnv, so the gap between releases is long but the changes when they come are not small.

Editorial conclusion

MO-Gymnasium fits a research team that needs to compare multi-objective reinforcement learning algorithms on the same set of environments, because the environment set spans the MORL literature and multi-objective versions of classical benchmarks, and because the environment versioning scheme means a change that would affect learning results gets a new suffix rather than silently altering results. Skip it for single-objective work, where plain Gymnasium is the simpler choice, and expect to install extra dependencies per environment family since the base install excludes them. Before you start, confirm the Python version on your machine is 3.11 or newer and that Gymnasium 1.1 or later is available, since the manifest requires both, and note the release picture, with 1.3.1 and 1.3.0 in October 2024 and 1.3.2 in February 2026, and the last push on 2026-10-16.

Frequently asked questions

Official sources

1. Farama-Foundation/MO-Gymnasium on GitHub
2. License: MIT
3. Project website
4. README
5. Releases