Reverb: a data storage and transport system for reinforcement learning research

What Reverb is and what the warning in the README means

Reverb is an efficient and easy-to-use data storage and transport system designed for machine learning research. The README places it primarily as an experience replay system for distributed reinforcement learning algorithms, and notes that the system also supports multiple data structure representations such as FIFO, LIFO and priority queues.

The install section opens with a caveat that deserves to be read before anything else. Reverb is not hardened for production use, and while the maintainers do their best to keep things in working order, things may break or segfault. There is also a warning that Reverb currently only supports Linux based operating systems.

That is an unusual thing for a project README to lead with, and it is a sign of honesty rather than of weakness. The system is built for research use, where the alternative is often a replay buffer written in the training loop itself, and the trade being offered is a scalable store with a well-defined interface in exchange for the usual production guarantees. A segfault in a research cluster job is a rerun; a segfault in a service is an incident.

The licence field on the repository says Apache-2.0, the primary language is C++, and the last push was on 2026-10-06, so the code is still being worked on even though the repository has no GitHub releases.

Installing Reverb and the TensorFlow version coupling

The recommended install is pip.

$ pip install dm-reverb[tensorflow]

# Without Tensorflow install and version dependency check.
$ pip install dm-reverb

The extras exist because of a version coupling, and the README explains it in the releases section. Due to some underlying libraries such as `protoc` and `absl`, Reverb has to be paired with a specific version of TensorFlow. If you install Reverb as `pip install dm-reverb[tensorflow]`, the correct version of TensorFlow is installed for you.

There is a nightly package as well, for the development line.

$ pip install dm-reverb-nightly[tensorflow]

# Without Tensorflow install and version dependency check.
$ pip install dm-reverb-nightly

The version table is the part worth memorising, because it is the compatibility contract. The Nightly branch tracks tf-nightly. Release 0.14.0 pairs with TensorFlow 2.14.0, 0.13.0 with 2.14.0, 0.12.0 with 2.13.0, 0.11.0 with 2.12.0, 0.10.0 with 2.11.0, 0.9.0 with 2.10.0, 0.8.0 with 2.9.0 and the 0.7.x line with 2.8.0.

The same section records the Python version milestones. 0.13.0 dropped Python 3.8 support, 0.11.0 was the first version to support Python 3.11, and 0.10.0 was the last to support Python 3.7.

Building from source is documented separately, and the README notes that the guide uses the same docker images the releases are built with, which is a sensible way to guarantee the toolchain matches.

Starting a server, creating a client and inserting items

The quick start is four Python snippets, and they show the whole model of the system.

A server is created with a list of tables.

import reverb

server = reverb.Server(tables=[
    reverb.Table(
        name='my_table',
        sampler=reverb.selectors.Uniform(),
        remover=reverb.selectors.Fifo(),
        max_size=100,
        rate_limiter=reverb.rate_limiters.MinSize(1)),
    ],
)

Every table has a sampler and a remover, which are the two ends of the queue, plus a maximum size and a rate limiter. Here the sampler is Uniform, so items come out at random, and the remover is Fifo, so the oldest item goes when the table is full.

A client then connects to the server's port.

client = reverb.Client(f'localhost:{server.port}')
print(client.server_info())

Writing a single item is one call, with a priority.

# Creates a single item and data element [0, 1].
client.insert([0, 1], priorities={'my_table': 1.0})

And a trajectory writer appends several data elements and then creates a single item referencing all of them, which is how a sequence of transitions is stored as one replay unit.

with client.trajectory_writer(num_keep_alive_refs=3) as writer:
  writer.append({'a': 2, 'b': 12})
  writer.append({'a': 3, 'b': 13})
  writer.append({'a': 4, 'b': 14})

  writer.create_item(
      table='my_table',
      priority=1.0,
      trajectory={
          'a': writer.history['a'][:],
          'b': writer.history['b'][:],
      })

  writer.flush()

The flush is what blocks until the item has been inserted and confirmed by the server, and the num_keep_alive_refs argument is what keeps the intermediate elements alive until the item that references them exists.

Tables, items and data elements, and why references matter

The data model is the part that takes a moment to understand, and the README explains it precisely.

A Reverb server consists of one or more tables. A table holds items, and each item references one or more data elements. Tables also define sample and removal selection strategies, a maximum item capacity, and a rate limiter.

The key property is that multiple items can reference the same data element, even if those items exist in different tables. This is because items only contain references to data elements rather than a copy of the data itself. And this means a data element is only removed when there exists no item that contains a reference to it.

That is reference counting, and it is what makes a table more than a queue. The README's example is a set of two tables over the same underlying data, one configured as a Prioritized Experience Replay for transitions of length two and another as a FIFO queue of sequences of length three. The same elements are in the store once, and the two tables offer different views of them.

The alternative, copying the data into each item, would make a prioritised and a FIFO view of the same trajectory cost twice the memory, and would make it impossible to have one table's removal policy free up data that another table still needs.

The rate limiter completes the picture. The example uses MinSize(1), which is a natural fit for replay, since a table should not hand out an item until there is at least one available, but the same mechanism can throttle producers.

The alternative uses the README describes

The README explains the motivation in a way that is worth quoting in substance, because it frames the whole project.

Experience replay has become an important tool for training off-policy reinforcement learning policies, used by algorithms such as Deep Q-Networks, Soft Actor-Critic, Deep Deterministic Policy Gradients and Hindsight Experience Replay. Building an efficient, easy to use and scalable replay system is challenging, and for good performance Reverb is implemented in C++, and to enable distributed usage it provides a gRPC service for adding, sampling and updating the contents of the tables. Python clients expose the full functionality of the service in an easy to use fashion, and native TensorFlow ops are available for performant integration with TensorFlow and tf.data.

So there are three layers: a C++ server, a gRPC service, and two client surfaces, one in Python and one in native TensorFlow ops for the tf.data path.

The README then makes the point that this flexibility extends past the original use case. Although originally designed for off-policy reinforcement learning, Reverb's flexibility makes it just as useful for on-policy reinforcement, or even unsupervised learning. And it says creative users have used Reverb to store and distribute frequently updated data such as model weights, acting as an in-memory lightweight alternative to a distributed file system where each table represents a file.

That last use is a good illustration of why the table abstraction is general. A table with a FIFO remover and a very large size limit is a file. A table with a prioritised sampler is a replay buffer. The system does not care which, which is the point of the design.

Where Reverb is the wrong tool

The first limitation is stated in the README and it is not a small one. Reverb is not hardened for production use, and things may break or segfault. For a research system that is an accepted trade; for anything serving traffic, it is disqualifying.

The second is the platform. Reverb currently only supports Linux based operating systems, so a Windows or macOS research workstation cannot run the server at all, and a mixed-environment lab has to keep a Linux machine for the server even if the training code runs elsewhere.

The third is the version coupling. Because of protoc and absl, Reverb has to be paired with a specific TensorFlow version, and the coupling is not merely a recommendation. A team with an existing TensorFlow setup, or one that needs to move TensorFlow forward for unrelated reasons, is constrained in when they can move Reverb. The nightly package helps with one direction of that problem and creates another.

The fourth is the scope of the system. Reverb is a data structure server with a few selection strategies, not a general storage engine. It has no durability story that the README claims, no replication protocol to speak of, and the checkpointing feature is one item in a table of contents that the README does not elaborate on in the excerpt available here.

The fifth is the maturity of the release process. There are no GitHub releases, so the version table in the README is the only record of which Reverb goes with which TensorFlow, and it stops at 0.14.0 while the last commit is from October 2026. The table is therefore the thing to trust, not the tags.

How Reverb compares with a local replay buffer and with Ray and with a plain gRPC store

The comparison with a replay buffer written in the training loop is the one the project is built against, and the difference in approach is distribution and data structure. A local buffer is a list in the same process, which is fast and simple and has no network in the path, but it caps the replay size at the memory of one machine, cannot be shared between parallel workers without copying, and is usually specialised to one algorithm's data layout.

Reverb pays for that in a gRPC round trip on every insert and sample, and buys a store that several workers can share, that can hold far more than one machine's memory, and that can change its selection strategy without the training loop changing.

The comparison with Ray is the closest one in spirit, since Ray is also a distributed system for machine learning research with an actor model and a placement group abstraction. The difference is that Ray is a general framework, in which Reverb would be one component, while Reverb is one service with one job. A team already running Ray might add Reverb for replay specifically, but the overlap in machinery is real and the README does not address it.

The third comparison is with a plain gRPC service with a queue behind it. That is what Reverb is, in the sense that the transport is gRPC and the server is C++, and the difference is that Reverb gives you a fixed set of table semantics, a reference-counted data model, rate limiters and native TensorFlow ops rather than an interface you design yourself. You give up the flexibility of a custom service and gain the parts that are easy to get wrong.

Licence, project structure and what to check before you install

The licence field on the repository says Apache-2.0, with the LICENSE file at the root of the repository. A permissive licence with a patent grant is a good fit for a research infrastructure project that other labs will build on.

The project is not archived and the last push was on 2026-10-06, so the code is current even though there are no GitHub releases. The repository layout shows the build system and the packaging in one glance: a WORKSPACE and a MODULE.bazel alongside a BUILD file and a .bazelversion, which is the Bazel workspace layout. There is a configure.py, an oss_build.sh, a docker/ directory, a third_party/ directory, a reverb/ directory for the server and the Python package, docs/ and examples/ directories, and a run_python_tests.sh script.

The examples are two notebooks, demo.ipynb and frame_stacking.ipynb, and the README points at the demo notebook as an interactive tutorial, which is the fastest way to see the trajectory writer and the sampler working together.

Four things to check before you install.

The Reverb and TensorFlow version pair from the table, and whether the pair you need is one the table lists, because the version check in the pip install is there to catch a mismatch.

Whether your Python version is compatible, given the milestones the README records, in particular that 0.10.0 was the last to support Python 3.7 and 0.13.0 dropped Python 3.8.

Whether you have a Linux host for the server, since the platform support is Linux only.

And whether the research use justifies the C++ and gRPC dependency at all, because for a single-machine off-policy run with a modest buffer, the in-process list is faster and has no version coupling to negotiate.

Editorial conclusion

Reverb suits a research team running off-policy reinforcement learning that needs a replay store that scales across machines and that can be swapped for a different data structure without rewriting the training loop. It is the wrong tool for a production service, because the README says outright that it is not hardened for production use and that things may break or segfault, and the wrong choice on anything but Linux. Before deploying it, pin the Reverb and TensorFlow versions using the table in the README, since the two are coupled through protoc and absl and a mismatched pair will not build.

Frequently asked questions

Official sources

1. google-deepmind/reverb on GitHub
2. Issues
3. License: Apache-2.0
4. README