Acme: DeepMind's RL framework, four years ahead of its own releases

Who Acme is written for

Acme is a library of reinforcement learning building blocks and agents from Google DeepMind. The README is unusually direct about its audience. It states that Acme is first and foremost a framework for RL research written by researchers, for researchers, that the team uses it daily, and that while they will try to keep everything in working order, things may break occasionally.

That framing should be read as a contract. This is not a library that promises API stability or a support window. It is a set of reference implementations that double as strong baselines, and a starting block for novel research.

The README names two roles for the included agents. They serve as reference implementations and performance baselines, and they are meant to be simple and flexible enough that you can take one apart and build on it. Readability is stated as an explicit goal, which is a different priority from most agent libraries that optimise for throughput first.

The practical reader for this is someone doing RL research who wants to start from a known-good implementation of an algorithm and change it. Someone who wants to train an agent on a problem and move on has easier options.

Components instead of one agent class

The second structural idea in the README is scale. It says the building blocks are designed so agents can run at multiple scales, giving the example of single-stream versus distributed agents. That is a different commitment from a library that hands you a single Agent class with a train() method.

The dependency list in setup.py shows what makes that possible. Alongside the core requirements of absl-py, dm-env, dm-tree, numpy, pillow and typing-extensions, the TensorFlow and JAX extras pull in dm-reverb and dm-launchpad. Reverb is DeepMind's experience replay system, and Launchpad is its library for launching distributed programs, so the distributed path is a real dependency rather than an aspiration.

The core install is deliberately thin. The README warns that the set of dependencies included for a bare dm-acme install is minimal, and that to run any of the included agents you also need either JAX or TensorFlow depending on the agent. So pip install dm-acme alone gives you the components and no runnable agent.

The repository layout is small for what it claims: acme/, docs/, examples/ and setup.py at the root. The README points at the examples subdirectory as the fastest way in, and names a quickstart notebook, a tutorial notebook that goes into agent internals, and a technical report on arXiv for the design background.

Installing Acme and running an example

The README strongly recommends a virtual environment first, to avoid version conflicts:

python3 -m venv acme
source acme/bin/activate
pip install --upgrade pip setuptools wheel

Then install the library with the backend you intend to use. The README recommends installing both, since which one you need depends on the agent:

pip install dm-acme[jax,tf]

Environments are a separate extra, and the README lists gym, dm_control and bsuite as what it brings in:

pip install dm-acme[envs]

For the bleeding edge, the README gives the install from a clone, run from the directory containing setup.py:

pip install .[jax,tf,testing,envs]

After that, the README's suggested path is the examples directory for runnable code, the quickstart notebook for a single agent, and the tutorial notebook for how an agent is constructed internally. It does not include a copy-paste training script in the README itself, so expect to read the examples rather than paste from the page.

The pinned versions are from 2022 and 2023

The setup.py pins its heavy dependencies to exact versions, and those versions are old. The TensorFlow extra pins tensorflow==2.8.0, tensorflow_probability==0.15.0, tensorflow_datasets==4.6.0, dm-reverb==0.7.2 and dm-launchpad==0.5.2. The JAX extra pins jax==0.4.3 and jaxlib==0.4.3, and it includes the whole TensorFlow list as well.

The comment above those pins explains why: any particular version of reverb needs to be pinned against a particular version of TensorFlow because of how it is built. That is a legitimate reason to pin tightly, and it is also the reason this library is awkward to install on a current Python.

TensorFlow 2.8.0 dates from early 2022 and jax 0.4.3 from early 2023. Both predate recent Python releases, so on a current interpreter you should expect unavailable wheels, dependency resolution that runs for a long time, or a resolution failure. The practical answer is to build the environment on an older Python version rather than fight the resolver, and to install into the virtual environment the README asks for rather than system-wide.

Note also that the JAX extra is not a small alternative to the TensorFlow extra. It installs the TensorFlow stack too, so choosing both is the normal case rather than an economy.

Releases stopped in 2022 while commits did not

There is a gap between how alive the repository looks and what you can install from PyPI. The last push to the repository was on 2026-09-11. The newest tag is 0.4.0, published on 2022-02-10, with 0.3.0 on 2022-01-08 and 0.2.4 on 2021-12-09.

So the code has moved on considerably since the last release, and none of that work is in a tagged version. Anyone who installs dm-acme from PyPI gets something reflecting the repository as it stood in early 2022, with the dependency pins to match. Anyone who wants current behaviour has to install from a clone and accept the higher risk that comes with a development checkout.

That is consistent with the README's own warning that things may break occasionally. It does mean the version number on your install tells you very little about which commit you are running.

For maintenance planning the honest statement is this: the project is not abandoned, and it is not releasing. Upgrades are pulls from master, and there is no changelog to read between them.

Stable Baselines3 as the alternative

The obvious alternative for someone who wants to train an agent rather than study one is Stable Baselines3, and the difference in approach is philosophical as much as technical.

Stable Baselines3 implements algorithms in PyTorch and is built around a single-process training loop with a uniform API, extensive documentation and a set of tuning and logging utilities aimed at getting reliable results on a specific environment. Its goal is that a practitioner gets a working agent.

Acme does the opposite. It exposes the components an agent is made of, expects you to assemble or modify them, and supports running them distributed through Launchpad with replay handled by Reverb. Readability and flexibility are stated goals; a polished practitioner experience is not.

If your problem is that you need a trained policy, Stable Baselines3 is the shorter path. If your problem is that you want to change how an algorithm works and compare against a reference implementation of it, Acme is built for exactly that, and the arXiv technical report gives you the reasoning behind the component boundaries.

Licence and citation

Acme is Apache-2.0, which is permissive and allows commercial use with attribution and the usual patent grant. There is no copyleft obligation on code that uses it, and the repository carries a LICENSE file at the root.

The README asks that academic work citing Acme cite the technical report, arXiv 2006.00979, rather than the software alone, and gives the full BibTeX entry with the 2020 author list. That is worth following if you publish, because the report is where the design decisions are written down.

The repository also carries a .pylintrc and a readthedocs configuration, and the tests run through a test.sh script at the root, which is how the CI badge in the README is produced.

Editorial conclusion

Adopt Acme if you are doing reinforcement learning research and want to start from readable reference implementations you can take apart, especially if you plan to run agents distributed through Launchpad with Reverb handling replay. Do not adopt it if you want a supported, versioned dependency, because the newest tag is 0.4.0 from 2022-02-10 and current work is only available from a clone, or if you need current Python, since the pins are TensorFlow 2.8.0 and jax 0.4.3. If you only need a trained policy, Stable Baselines3 will get you there faster. Before committing, create the virtual environment the README asks for, install with pip install .[jax,tf,testing,envs] from a clone, and confirm the examples directory runs on your Python version before you write anything against the API.

Frequently asked questions

Official sources

1. google-deepmind/acme on GitHub
2. Issues
3. License: Apache-2.0
4. README
5. Releases