# Behaviour Suite for Reinforcement Learning

> bsuite is a Python library for running shared reinforcement learning experiments with automatic result logging. It is aimed at researchers who need comparable evaluations across small, configurable environments.

**google-deepmind/bsuite** — bsuite is a collection of carefully-designed experiments that investigate core capabilities of a reinforcement learning (RL) agent

- Repository: https://github.com/google-deepmind/bsuite
- Stars: 1,566 · Forks: 192
- Language: Python
- License: Apache-2.0
- Published: 2026-10-16 · Updated: 2026-10-16 · Language: en
- Canonical page: https://hysenlabs.com/projects/google-deepmind-bsuite

## bsuite's shared RL evaluation problem

bsuite addresses the problem of comparing reinforcement learning agents without writing a separate evaluation harness for every capability. The README describes it as a collection of carefully designed experiments whose two purposes are to provide scalable problems that expose issues in efficient and general learning algorithms, and to study agent behavior through performance on shared benchmarks. The intended users are reinforcement learning researchers, students, and developers who need a reproducible evaluation baseline.

The library is not a general-purpose agent library. Its center of gravity is measurement. Each experiment is designed to investigate a particular capability or issue, and the resulting environments are deliberately small. The README says their small observation sizes allow reasonable performance with a small network on a CPU. That makes bsuite useful for research iterations and controlled comparisons, but it also means the suite should not be confused with a full simulation platform for robotics, industrial control, or other high-dimensional continuous-control problems.

The project emphasizes shared benchmarks and analysis rather than one optimized training workload. Users can run the same environment definitions and settings across agents, then use the included notebook and analysis files to inspect results. This is the main practical distinction for an engineering team choosing a tool: bsuite is primarily an evaluation suite, not a replacement for the training framework that contains the agent.

## How bsuite experiments and analysis are organized

The repository layout defines the architecture. Experiments live under the `experiments` subdirectory, and each experiment has its own directory. That directory contains an environment definition, a `sweep.py` file with a `SETTINGS` sequence of keyword arguments, and an `analysis.py` file defining plots for a Jupyter notebook. This structure makes an experiment's environment, settings, and analysis relatively easy to locate.

Environments are selected through a `bsuite_id` string, such as `deep_sea/7` or `catch/0`. The string identifies both the experiment and the indexed environment settings. `bsuite.load_from_id()` loads an environment. The module also exposes experiment-specific ID sequences through constants such as `sweep.DEEP_SEA` and `sweep.DISCOUNTING_CHAIN`, while `sweep.SWEEP` contains the sequence needed to run all experiments. `sweep.TAGS` groups IDs by tags such as `basic` and `scale`.

The logging design is important. `bsuite.load_and_record_to_csv()` logs results from within the environment and writes one CSV file for each `bsuite_id`. According to the README, this format suits experiments split across multiple machines. A terminal logger is also available as `bsuite.load_and_record_to_terminal()`. Because recording happens when the environment is loaded, the result format is produced without imposing a particular internal structure on the agent or learning algorithm.

## Installing bsuite and running the first catch experiment

The README documents Python 3.6 and Python 3.7 as the tested versions. It recommends a Python virtual environment so the dependencies do not affect the system installation. The following commands create and activate that environment, upgrade `pip` and `setuptools`, and install `bsuite` from PyPI. The reader should see the active virtual environment in the shell after running `source bsuite/bin/activate`.

```bash
python3 -m venv bsuite
source bsuite/bin/activate
pip install --upgrade pip setuptools
```

After the environment is active, install the package itself:

```bash
pip install bsuite
```

The first direct use can load the `catch/0` environment and inspect its number of actions. This code uses the public loading function and the `dm_env` action specification exposed by the environment.

```python
import bsuite

env = bsuite.load_from_id('catch/0')
num_actions = env.action_spec().num_values
```

For a complete interaction loop, the README shows an agent with a `step()` method. `bsuite_num_episodes` supplies the episode count required by bsuite analysis, which can vary by environment. The final `agent.step(timestep)` call after the loop passes the terminal timestep to the agent.

```python
for _ in range(env.bsuite_num_episodes):
  timestep = env.reset()
  while not timestep.last():
    action = agent.step(timestep)
    timestep = env.step(action)
  agent.step(timestep)
```

The README also documents an optional `bsuite[baselines]` installation for baseline dependencies, excluding OpenAI and Dopamine examples.

```bash
pip install bsuite[baselines]
```

## dm_env compatibility and OpenAI Gym conversion

bsuite environments implement the Python interface defined by `dm_env`. Observations are returned as NumPy arrays, and actions are discrete, zero-based integers. The README explains that a scalar NumPy array with shape `()` is equivalent. This gives an agent a small, explicit interaction contract, but it also means code written around a different action representation will need adaptation.

For codebases that use the OpenAI Gym interface, bsuite provides `GymFromDMEnv` in `utils/gym_wrapper.py`. The wrapper is useful when an existing training or evaluation loop expects Gym-style calls. However, the README explicitly says that Gym is not included in bsuite's default dependencies, so installing the base package does not guarantee that a Gym-based workflow will work immediately.

```python
import bsuite
from bsuite.utils import gym_wrapper

env = bsuite.load_and_record_to_csv('catch/0', results_dir='/path/to/results')
gym_env = gym_wrapper.GymFromDMEnv(env)
```

The wrapper does not change the underlying result logging arrangement. The example still loads a logged bsuite environment and then adapts its interface. That distinction matters when integrating bsuite with an existing codebase: the compatibility layer addresses the environment API, while bsuite continues to handle experiment settings and recording.

The README does not provide a complete list of all supported agent architectures or a universal adapter for every reinforcement learning framework. It does state that logging does not constrain the structure of agents or algorithms, so the interaction loop is the main integration point.

## CSV logging, notebook analysis, and distributed evaluation

The CSV logger is the clearest supported path for collecting bsuite results across machines. `load_and_record_to_csv()` creates one file per `bsuite_id`, rather than one file for an entire sweep. That arrangement can make experiment-level collection and later analysis easier to distribute, although the README does not document aggregation, retry behavior, file naming details, or a rollback procedure.

The project includes a terminal logger for smaller interactive runs and points users to `logging/csv_logging.py` as the simplest reference for writing a custom logging mechanism. That is useful when CSV is not the right storage format, but it means storage integration is partly an implementation task for the user. The README mentions that older versions included an SQLite logger and says to contact the maintainers to update and reinstate it. It should therefore not be treated as a currently supported SQLite backend based on the README.

Analysis is organized around a pre-made Jupyter notebook at the linked Colab address and `analysis/results.ipynb`. Each experiment's `analysis.py` file defines plots used by the notebook. The suite is consequently oriented toward producing comparable result files first and plotting or studying them afterward. The README does not document a command-line report generator, a database schema, or a documented rollback workflow.

For local or multi-machine execution, `run_on_gcp.sh` appears in the repository's top-level entries, but the README does not explain its required settings or commands. It would therefore be inappropriate to present it as a ready-to-run deployment recipe without inspecting that file directly.

## bsuite's limits, maintenance, and license

The clearest limitation is scope. bsuite focuses on discrete-action environments with small observations and a shared evaluation structure. It is not the right tool when the immediate requirement is a production RL service, a continuous-control simulator, or a training pipeline whose primary value is distributed infrastructure rather than benchmark comparison. The README does not document support for arbitrary environment types beyond the interface and examples it describes.

A second limitation is result infrastructure. CSV and terminal logging are documented, and the CSV layout is compatible with experiments split across machines. The README does not document built-in database-backed storage, monitoring, automatic retries, or rollback. A team that requires those features would need to supply them or use a different logging layer based on the CSV reference.

Maintenance status is favorable by the available facts. The repository is not archived, and its last push was on 2026-10-05. Recent releases include `0.3.6` on 2026-05-08, following older releases from 2021 and 2020. These dates show recent repository activity and a release in 2026, but the README still describes Python 3.6 and 3.7 as tested versions, so compatibility should be checked against the reader's runtime rather than inferred from the release number.

The repository identifies its license as Apache-2.0. That is useful for understanding the project's licensing terms, but this review does not provide legal advice or an interpretation of how a particular distribution, dependency, or internal modification should be handled.

## Conclusion

bsuite fits researchers who want repeatable RL experiments and shared evaluation settings, especially with CPU-sized networks. It is less suitable for applications needing continuous-control environments, built-in Gym dependencies, or a database-backed logging system. Verify first that the required environments and `dm_env` interaction model match your agent, then confirm whether CSV logging satisfies your evaluation storage needs.

## FAQ

### How do I install bsuite?

The README recommends creating a Python virtual environment, activating it, upgrading `pip` and `setuptools`, and then running `pip install bsuite`. It says Python 3.6 and 3.7 were tested.

### How do I load and log a bsuite environment?

Use `bsuite.load_from_id()` to load an environment such as `catch/0`. For automatic recording, the README documents `bsuite.load_and_record_to_csv()` and `bsuite.load_and_record_to_terminal()`.

### Can bsuite work with an OpenAI Gym codebase?

Yes, through `bsuite.utils.gym_wrapper.GymFromDMEnv`. The README says Gym is not a default dependency, so a separate installation may be required.

## Sources

- [google-deepmind/bsuite on GitHub](https://github.com/google-deepmind/bsuite)
- [Issues](https://github.com/google-deepmind/bsuite/issues)
- [License: Apache-2.0](https://github.com/google-deepmind/bsuite/blob/main/LICENSE)
- [README](https://github.com/google-deepmind/bsuite/blob/main/README.md)
- [Releases](https://github.com/google-deepmind/bsuite/releases)

---

Hysen Labs editorial analysis, written from the project's own repository and release notes. Cite the canonical page: https://hysenlabs.com/projects/google-deepmind-bsuite