MADRL: Four Multi-Agent Environments and the Code That Trains on Them

Four environments, and a note that two of them have moved on

MADRL provides implementations of four multi-agent reinforcement learning environments. Pursuit Evasion and Waterworld sit under the pursuit directory, Multi-Agent Walker under walker, and Multi-Ant under the mujoco/ant path. The most useful line in the project documentation is the note at the top, which points out that a maintained version of two of these environments with various fixes is included with PettingZoo, the Farama Foundation project, with a link to the SISL environments page there. That note changes how you should approach the repository. If you want to run one of these benchmarks today, the PettingZoo version is the one with an active community behind it, and MADRL is the place to look for the original code and the training setup rather than the first choice for a new experiment.

The dependency chain, and why the clone must be recursive

The package requires both OpenAI Gym and a forked version of rllab, specifically the multiagent branch, and the documentation links a fork maintained under the rejuvyesh account. There are further requirements listed in the rllab/environment.yml file if you are using the anaconda distribution. The setup instruction follows from that structure, because the easiest way to install MADRL and its dependencies is a recursive clone of the repository.

git clone --recursive [email protected]:sisl/MADRL.git

The recursion is not a convenience. The repository has a .gitmodules file, and the rllab and rltools directories are checked in as submodules rather than as ordinary directories, so a plain clone would leave you without the training framework the runners import. That is the shape of a research codebase: the framework is pinned as a fork of a specific branch, because the multi-agent extensions to it are not upstream.

Putting the repository on the Python path

After the clone, the directories have to be added to PYTHONPATH, and the setup section gives the exact form.

export PYTHONPATH=$(pwd):$(pwd)/rltools:$(pwd)/rllab:$PYTHONPATH

Three paths, in order: the repository root, then rltools, then rllab. The root is where the environments and the runners live, rltools is a local tools directory and rllab is the framework fork. Because the paths are relative to the current directory, the command only works from the repository root, which is worth remembering when a runner fails to import something and the error looks like a missing dependency rather than a path problem. The setup notes also suggest looking into the rllab/environment.yml file to install the remaining requirements, which is the shortest way to get a working environment for this stack.

Running a training job with a curriculum

The usage example is a runner invocation with a curriculum file, and the flags in it are the ones you would change for your own experiment.

python3 runners/run_multiwalker.py rllab \ # Use rllab for training
    --control decentralized \ # Decentralized training protocol
    --policy_hidden 100,50,25 \ # Set MLP policy hidden layer sizes
    --n_iter 200 \ # Number of iterations
    --n_walkers 2 \ # Starting number of walkers
    --batch_size 24000 \ # Number of rollout waypoints
    --curriculum lessons/multiwalker/env.yaml

The positional argument chooses the training framework, with rllab as the option shown, and the control flag picks the training protocol, with decentralized as the example. The policy is described as an MLP with the hidden layer sizes given as 100,50,25, and the curriculum parameter points at a YAML file under the lessons directory, which is how the environment is configured rather than through a Python default. The batch size of 24000 rollout waypoints is large, and it reflects the sample efficiency problem in continuous control, where the walkers have to move a long way before a policy signal appears.

What the repository directory layout reveals

The top level shows a research project with the usual moving parts. The madrl_environments/ directory holds the four environments, runners/ holds the training scripts including the multiwalker runner, and rllabwrapper/ is the bridge between the rllab framework and these environments. There is a lessons/ directory with the YAML curriculum files, a maps/ directory for the pursuit environments, pipelines/ for processing, heuristics/ for scripted behaviour, a vis/ directory for visualisation, and pursuit_policy.py at the root for a hand-written policy. There is also a fabfile.py, which is a fabric deployment file, and a sample_spec.yaml. The runners directory matters most for a user, because that is where the training protocol options live, and the lessons directory matters second, because that is where the environment difficulty and layout are configured.

Where the policy definitions live, and the citation

The documentation points out that policy definitions exist in rllab/sandbox/rocky/tf/policies, which is a detail worth having before you start looking for a network definition. The path names the framework convention, the rocky style, the TensorFlow backend, and the policies directory, so a policy is a file in the framework rather than something in the MADRL tree itself, and the MADRL side is the environment and the training loop. The paper to cite is Cooperative multi-agent control using deep reinforcement learning, by Gupta, Egorov and Kochenderfer, in the International Conference on Autonomous Agents and Multiagent Systems, pages 66 to 83, published by Springer in 2017. The multi-agent walker and multi-ant environments in particular are the ones the paper's experiments are built on.

Maintenance, licences and the practical state of the stack

The repository is not archived and the last push was on 2026-07-07, with no GitHub releases, so the code is in the repository rather than in a tagged distribution. The licence is MIT, and the third-party components carry their own terms, which matters more than usual here because the stack is old. OpenAI Gym has been superseded by the Gymnasium line, and rllab itself is a research framework from the RLPy family rather than something under active maintenance. A forked branch of a research framework is the right thing to use if you are reproducing a specific paper and the wrong thing to build a new project on. The PettingZoo note at the top of the documentation is the signal to take seriously: the community has already moved part of this work to a maintained home, and the sensible workflow is to check what exists there before committing to this stack.

Editorial conclusion

MADRL fits a researcher who wants the original cooperative control experiments runnable, with the environment code and the training loops intact rather than reimplemented from a paper. It is the wrong choice for new work, because the maintained versions of two of these environments now live in PettingZoo and are the better-supported option, and it is the wrong choice if you need a modern reinforcement learning stack, since it depends on OpenAI Gym and a forked rllab. Before you start, decide whether you need the rllab path at all, and check the PettingZoo versions first.

Frequently asked questions

Official sources

1. Issues
2. License: MIT
3. README
4. sisl/MADRL on GitHub