ML-Agents: Training Agents Inside Unity Scenes With PPO, SAC and Imitation Learning
The Unity Machine Learning Agents Toolkit (ML-Agents) is an open-source project that enables games and simulations to serve as environments for training intelligent agents using deep reinforcement learning and imitation learning.
At a glance
- What is it?
- Unity's ML-Agents Toolkit connects a Unity scene to a Python training loop over a gRPC channel, so the same build that ships as a game can act as a reinforcement learning environment. It is a strong fit when the environment already exists in Unity and a poor fit when it does not.
- Who is it for?
- Adopt ML-Agents if your environment is already a Unity scene and you need PPO, SAC, MA-POCA, self-play, BC or GAIL without writing the training loop yourself. Do not adopt it if your environment lives in Python, because you would be adding a Unity build step and a gRPC hop for no gain; use Gymnasium or PettingZoo directly instead.
- Can I use it commercially?
- Check first. The repository uses a licence we do not classify automatically, so read its LICENSE file before any commercial use.
- Is it still maintained?
- Yes. The repository last received commits 13 days ago.
- What is it written in?
- Mainly C#, according to GitHub's language statistics.
Answers come from the project's GitHub data, last synced on September 15, 2026, and from our analysis. They are not legal advice.
DEEP OPEN-SOURCE ANALYSIS
The problem ML-Agents solves is the environment, not the algorithm
Reinforcement learning code is not the scarce resource. Environments are. A researcher who wants to study NPC behaviour in a 3D scene with physics, occlusion and animation has to either build that scene in a simulator or accept a 2D proxy. ML-Agents takes the opposite position: the scene already exists, because someone built it in Unity, and the toolkit's job is to expose that scene to a trainer. The README frames this as a two-way benefit, describing a central platform where AI advances can be evaluated on Unity environments and then made accessible to game developers. That framing is honest about the audience. Game developers get PPO, SAC, MA-POCA and self-play without implementing them. Researchers get a scene authoring tool and a physics engine they did not have to write. Anyone whose environment is a Python function or a MuJoCo model gets nothing from this project, and the README does not pretend otherwise.
Two processes, one gRPC channel, and a decision request per step
The architecture splits cleanly. On the Unity side, the com.unity.ml-agents package provides an SDK that you attach to a scene. Agents observe, request decisions and act. On the Python side, the mlagents package runs the trainer. The two talk over a gRPC channel, which is why the README lists control of a Unity environment from Python as a feature and why the same scene can be driven by a trainer, by a Gym wrapper or by a PettingZoo wrapper. The step loop is not a free-running simulation. Agents request decisions, and the README lists On Demand Decision Making as a feature, which means an agent can be configured to ask for a policy action only when it needs one rather than every frame. That matters for turn-based games and for scenes where most frames are visually identical. The trainer side is PyTorch. The README states that the provided algorithm implementations are based on PyTorch, and it also documents a custom trainer plugin path, so a lab that wants to swap the optimizer does not have to fork the C# SDK.
What ships in the box: PPO, SAC, MA-POCA, self-play, BC and GAIL
The README names the algorithm set directly: PPO, SAC, MA-POCA and self-play for reinforcement learning, plus BC and GAIL for imitation learning. It also lists neuroevolution as something the Python API supports. The multi-agent story is the part that distinguishes this toolkit from a generic RL library. MA-POCA is a cooperative multi-agent algorithm, and self-play covers the competitive case. If your problem is a single agent in a single scene, you are using a large surface area to solve a small problem, and a plain Gymnasium loop with a PPO implementation would be less machinery. If your problem is a team of agents that must coordinate, or an adversary that must co-adapt, the built-in support is the reason to be here. Curriculum learning and environment randomization are also listed as features, both aimed at the same difficulty: tasks that a policy cannot solve from a cold start.
Getting it running: the package, the CLI, and the config file
The README points at two artifacts per release. Release 23 pairs Python package 1.1.0 on PyPI with Unity package 4.0.0, and the README states that com.unity.ml-agents is verified for Unity 2020.1 and later. The installation path is therefore pip install mlagents on the Python side and adding the com.unity.ml-agents package through the Unity Package Manager on the editor side. Training is driven by a YAML configuration file passed to the mlagents-learn command, and the README's own links show where the details live: the Learning-Environment-Examples page for the 17+ sample scenes, the Python-Custom-Trainer-Plugin page for replacing the trainer, and the Python-LLAPI, Python-Gym-API and Python-PettingZoo-API pages for driving a build from Python. I have not run any of this, so treat the exact flag names as something to read off the package documentation rather than from this article. The one configuration detail worth flagging is the version pinning. The README warns that the develop branch is under active development and may be unstable, and it separates the develop row from the Release 23 row in its own table. Pin to a release tag unless you need a fix that only exists on develop.
The docs moved, and the old site is now a dead end
The README carries a migration notice stating that Unity Package documentation is now the primary developer documentation and that maintenance of the web docs at unity-technologies.github.io/ml-agents has been deprecated. This is a real operational constraint, not a cosmetic one. Search results, blog posts and Stack Overflow answers from the Release 20 and Release 21 era point at the deprecated site, and the API has moved since then. Anyone following a 2023 tutorial will hit renamed or removed configuration keys and conclude the toolkit is broken. The mitigation is to start from the package documentation URL in the README's table, which is versioned per release, and to use the Migration page when moving between releases. The README also links a Versioning page that explains how releases are managed across components, which is worth reading before you file an issue about a version mismatch between the Python package and the Unity package.
Where ML-Agents is the wrong tool
The toolkit assumes Unity is in the loop. Every training run requires a Unity build or an editor instance, a gRPC connection, and a scene that has been instrumented with the SDK. That is a heavy dependency for problems that do not need rendering or physics. If your environment is a matrix, a graph or a text corpus, the Unity layer is pure overhead. There is a second limitation that the README implies rather than states. The feature list is oriented toward continuous control and game-like tasks: 2D, 3D and VR/AR games, NPC behaviour, automated testing of builds, pre-release evaluation of design decisions. There is no claim of support for offline RL from a fixed dataset, and the imitation learning support is BC and GAIL, which need demonstrations rather than a logged dataset of arbitrary transitions. A team whose data is a warehouse of past gameplay logs, with no simulator to step, should look elsewhere. The third constraint is the release cadence. Release 21 landed in October 2023, Release 22 in October 2024, and Release 23 in August 2025. That is roughly annual. If your project depends on a fix that is not in a tagged release, you are either on develop or you are waiting a year.
The alternative is Gymnasium, and the difference is who owns the environment
The honest comparison is Gymnasium, the standard Python interface for single-agent environments, and PettingZoo for multi-agent. ML-Agents can wrap a Unity scene as both, according to the README, which is the tell: the wrappers exist so that Unity scenes can be consumed by the wider Python RL ecosystem. If your environment is already a Python object, Gymnasium gives you the interface and you pick your own trainer. ML-Agents gives you the trainer and the algorithms, but only for environments it can talk to. The difference in approach is where the boundary sits. Gymnasium puts the boundary inside Python, so the environment and the policy share a process and a debugger. ML-Agents puts the boundary at gRPC between a C# runtime and a Python runtime, which buys you a real physics engine and a real renderer at the cost of a build step and a serialization hop. Neither is better in the abstract. The choice follows from whether the environment already exists as a Unity scene.
Licence, upgrades and what to check before you commit
The README displays an Apache-2.0 licence badge and links to LICENSE.md on the release/4.0.0 branch, but the repository metadata reported here lists the licence as NOASSERTION. Those two signals disagree, and the badge is a rendering of a file rather than a machine-readable declaration. Read LICENSE.md on the exact tag you intend to ship, and if your organisation has a policy gate, route it through whoever normally handles that rather than treating the badge as sufficient. This is not legal advice. On upgrade cost, the README provides a Migration page specifically for moving between releases, which tells you the project expects breaking changes across major versions. The package versioning is also split: the Unity package is at 4.0.0 while the Python package is at 1.1.0, so a single release number does not describe both halves. Track them separately. If you are evaluating this for a project that ships, start by building one of the 17+ example environments from the package documentation, confirm the editor version you have is one the package supports, and only then instrument your own scene.
Editorial conclusion
Adopt ML-Agents if your environment is already a Unity scene and you need PPO, SAC, MA-POCA, self-play, BC or GAIL without writing the training loop yourself. Do not adopt it if your environment lives in Python, because you would be adding a Unity build step and a gRPC hop for no gain; use Gymnasium or PettingZoo directly instead. Before committing, verify the Unity editor version against the package documentation, confirm whether your project needs the develop branch or Release 23, and check the LICENSE.md file on the release branch rather than trusting the Apache-2.0 badge in the README, since the repository metadata reports the licence as NOASSERTION.
Community notes