# TMRL and the distributed reinforcement learning framework for real-time control

> tmrl is a Python framework for training deep reinforcement learning policies in real-time settings, built on a single-server, many-client architecture so that samples can be collected on robots and training run on a cluster. Its demonstration is a self-driving pipeline for TrackMania 2020, and the library half of the project is aimed squarely at industrial control problems.

**trackmania-rl/tmrl** — Reinforcement Learning for real-time applications

- Repository: https://github.com/trackmania-rl/tmrl
- Website: https://pypi.org/project/tmrl
- Stars: 748 · Forks: 93
- Language: Python
- License: MIT
- Published: 2026-10-21 · Updated: 2026-10-21 · Language: en
- Canonical page: https://hysenlabs.com/projects/trackmania-rl-tmrl

## A racing game as the front door to real-time RL

tmrl is presented as a fully-fledged distributed reinforcement learning framework for robotics, built to help you train deep reinforcement learning agents in real-time applications, with robots, video games and high-frequency control named as the target domains. The flagship example is a self-driving pipeline for the TrackMania 2020 video game, and the project calls it a fun and safe robot proxy for vision-based autonomous driving.

That framing is deliberate, because the example removes the risk that normally stops an industrial team from experimenting. The TrackMania pipeline trains a policy, in the RL sense of the word, that maps observations such as images and speed to actions such as gas, brake and steering angle, using state-of-the-art algorithms including Soft Actor-Critic and Randomized Ensembled Double Q-Learning. The policy learns the physics of the game from histories or observations that are equally spaced in time.

The README also gives the vocabulary early, noting that in the context of reinforcement learning an AI is called a policy, which is the sort of definition that makes a library easier to enter.

Four audiences are addressed separately. AI and TrackMania enthusiasts get a tutorial and a pre-trained agent to watch, machine learning developers and roboticists get the library and its real-time foundations, developers who only want the game get a ready-made Gymnasium environment, and everyone is pointed at the TrackMania Roborace League, a vision-based competition for real-time self-racing agents.

That last audience split is a useful signal about what the project actually is: a research library with a friendly demo in front of it, not a productised training platform.

## One server, many workers, training on the cluster

The architecture is the part that matters for real applications. tmrl is based on a single-server, multiple-clients architecture, which lets you collect samples locally from one worker or from arbitrarily many, while training happens remotely on a High Performance Computing cluster.

For a robotics team this is the whole point. A robot's control loop runs at a rate no cluster can help with, and the samples it produces are the scarce resource. If the same policy is being improved by a large batch of replay data, the work has to be split: collect locally, train centrally, ship the weights back out. The framework describes itself as a complete framework for implementing ad-hoc reinforcement learning pipelines for real-world applications, with secure remote training and fine-grained customizability as the named properties, and it is fully compatible with real-time environments such as robots.

The README credits this design to the rtgym project, which is the real-time Gym framework underneath, and the TrackMania Gymnasium environment itself is built on rtgym as well. rtgym was later packaged as a standalone library, one of several sub-projects that the tmrl project gave birth to.

A complete tutorial for building such a pipeline for a specific application lives in the repository under readme/tuto_library.md, and there is a minimal drone example at tmrl/tuto/tuto_minimal_drone.py, which is the fastest way to see what a pipeline looks like end to end.

## SAC and REDQ, a replay memory, and the reward that decides everything

The TrackMania example pipeline is described as readily implemented, and the two named algorithms are Soft Actor-Critic and Randomized Ensembled Double Q-Learning, with the documentation also carrying sections on reinforcement learning basics, on SAC, on REDQ, and on what the project calls a clever reward.

The mechanics are the standard off-policy loop. Samples are stored in a large dataset called a replay memory, and in parallel those samples are used to train the neural network that maps observations to actions. The parallel part is what makes the loop efficient: collection and training overlap rather than alternate.

The reward function is the part the project singles out with its own section, and that is not vanity. In a racing game the reward decides whether the agent learns a racing line or learns to press the accelerator, and a badly shaped reward produces a policy that looks competent and drives badly. The README's own framing of the section title, a clever reward, says that reward shaping is treated as a first-class problem here rather than an afterthought left to the user.

Available action spaces and available observation spaces each have their own documentation section, which matters when you port the pipeline to a robot, since a real vehicle rarely offers the same control authority as a virtual gamepad.

## Screenshots or LIDAR: two very different learning problems

The example pipeline offers two ways to observe the world, and the choice changes the problem more than the model architecture does.

The headline path trains policies that drive from raw screenshots captured in real time, using a Convolutional Neural Network to process the camera images. The alternative, offered for beginners, is a simpler rangefinder observation, described as less potent but easier to learn from, and processed with a Multi-Layer Perceptron. The environment is documented in two parts in the README, a LIDAR environment and a full environment, so you can start with the cheaper signal and move up.

The second detail is the control path. The example pipeline controls the game through a virtual gamepad, and that is what enables analog actions. Without a virtual input layer, a policy could only emit the discrete choices a keyboard allows, and a car controlled by three binary keys is a different and much harder control problem.

For a robotics team, both pieces map onto reality in a direct way. A virtual gamepad corresponds to a real actuator accepting continuous commands, and the screenshot path corresponds to a camera stream, while the LIDAR path corresponds to the depth sensor that many vehicles already carry. The project is careful to note that these are proxies, not the real thing, which is the reason the library half of tmrl exists.

## What the installation actually creates on your machine

The installation itself is documented in separate pages for Windows and Linux under readme/, and the package is published on PyPI, so the mechanics are the usual pip workflow with a twist: the package also fetches a resources archive at install time. The build metadata is minimal, and the packaging entry point records the version it ships:

```python
TMRL_VERSION = '0.7.1'
```

The resources come from a fixed release URL:

```python
RESOURCES_URL = "https://github.com/trackmania-rl/tmrl/releases/download/v0.6.0/resources.zip"
```

That line is worth a second look. The package version is 0.7.1, while the archive it pulls is still tagged v0.6.0, so the shipped resources have not moved with the package for at least one release. It also means the install needs network access to GitHub, and that a failure to download raises a ConnectionError rather than leaving a partial install.

The download lands in a dedicated folder in your home directory:

```python
HOME_FOLDER = Path.home()
TMRL_FOLDER = HOME_FOLDER / "TmrlData"
```

and that folder is subdivided into checkpoints, dataset, reward, weights and config directories, each created if it does not already exist. The layout tells you how the framework is meant to be used in practice: checkpoints for saved policies, dataset for collected samples, reward for the reward configuration, weights for trained model weights, and config for everything else. The install script also refuses to proceed on Python older than 3.7, exiting with a message that Python below 3.7 is not supported, and a comment in the source warns that the same code is duplicated under tmrl.tools.init_package.init_tmrl, so changes to the resources URL have to be made in both places.

## Gymnasium environment, sub-projects and a competition

The Gymnasium environment is the fastest entry point for someone who already has a training framework. Once the library is installed, the environment is easy to use in your own code, and it is based on rtgym, which is on PyPI as its own project. The README explains that tmrl gave birth to sub-projects of more general interest that were cut out and packaged as standalone Python libraries, and rtgym is the one it names.

The TrackMania Roborace League is the other half of the public face of the project, a vision-based AI competition where participants design real-time self-racing agents in TrackMania 2020. Its own documentation lives in readme/competition.md. Competitions of this kind are a good forcing function for a real-time framework, because the latency budget is the same one a robot has.

The documentation set is broad and worth knowing before you plan an evaluation. There is a getting started page, a quick reference guide, the library tutorial, the competition page, and the API reference on readthedocs, alongside the Windows and Linux installation pages. The project also carries a DOI badge for Zenodo, which is how research software of this kind usually gets archived and cited.

One section of the README is worth a mention because it is unusual to see: a security section marked as important in the table of contents. A framework that accepts remote training connections and runs code from workers is exactly the kind of software where the security notes deserve reading before the first deployment.

## Release cadence, licence and the shape of the project

The release history is steady but not fast. The tagged versions are v0.7.1 on 2025-05-05, v0.7.0 on 2025-02-06 and v0.6.6 on 2024-05-15, and the last push to the repository was on 2026-06-09. The project is not archived, so the code is still being worked on, but the most recent published release is over a year older than the newest commit, and the version string in setup.py tracks 0.7.1.

The licence is MIT, with a LICENSE file at the root of the repository and a PyPI licence badge. MIT is permissive, allowing commercial and academic use alike, with the obligation to carry the copyright notice, and reading the file rather than relying on a summary is the right approach for a library you intend to depend on.

The repository layout is small and readable. There is a tmrl/ package, a readme/ directory holding the installation pages, tutorials, reference guide and competition notes, a docs/ directory for the readthedocs build, a pyproject.toml alongside a setup.py, and a MANIFEST.in. The project is packaged with setuptools, and the pyproject build system section is correspondingly short:

```toml
[build-system]
requires = [
    "setuptools",
]
build-backend = "setuptools.build_meta"
```

For a framework whose selling point is real-time robotics work, the takeaway is that the library half is young and lightly versioned, while the research half, the TrackMania pipeline and the competition, is where the accumulated effort and the documentation sit.

## Conclusion

Adopt tmrl if you have robots or a real-time control loop that cannot generate samples fast enough on the training machine, because the single-server, many-client split is the reason to reach for this framework rather than a plain Gym setup. Skip it if your problem is a standard offline reinforcement learning benchmark, where a Gymnasium environment and a training script are enough, and treat the version numbers carefully, since the package declares 0.7.1 while the resources it downloads are still pulled from the v0.6.0 release. Verify first that the samples your workers return can be collected at the rate your policy needs, and read the readme/Install.md page for your platform, since the Windows and Linux instructions are separate and the install is not a plain pip install.

## FAQ

### What is tmrl used for?

tmrl is a Python framework for training deep reinforcement learning policies in real-time applications such as robots, video games and high-frequency control. It ships an example self-driving pipeline for TrackMania 2020 as a safe proxy for vision-based autonomous driving.

### How does tmrl handle real-time training?

It uses a single-server, multiple-clients architecture, so samples can be collected locally from one or many workers while training runs remotely on an HPC cluster, and it is built on the rtgym real-time Gym framework.

### Which reinforcement learning algorithms does the TrackMania pipeline implement?

The README names Soft Actor-Critic and Randomized Ensembled Double Q-Learning, with samples stored in a replay memory and used in parallel to train the policy.

### What does installing tmrl put in my home directory?

The install creates a TmrlData folder in your home directory with checkpoints, dataset, reward, weights and config subdirectories, and downloads a resources archive from the v0.6.0 GitHub release during setup.

### What Python version does tmrl support?

The setup script exits with a message that Python below 3.7 is not supported, and the package version it declares is 0.7.1.

## Sources

- [License: MIT](https://github.com/trackmania-rl/tmrl/blob/master/LICENSE)
- [Project website](https://pypi.org/project/tmrl)
- [README](https://github.com/trackmania-rl/tmrl/blob/master/README.md)
- [Releases](https://github.com/trackmania-rl/tmrl/releases)
- [trackmania-rl/tmrl on GitHub](https://github.com/trackmania-rl/tmrl)

---

Hysen Labs editorial analysis, written from the project's own repository and release notes. Cite the canonical page: https://hysenlabs.com/projects/trackmania-rl-tmrl