RLtools: header-only C++ deep RL that compiles to a single binary
The Fastest Deep Reinforcement Learning Library
At a glance
- What is it?
- RLtools is a C++17 reinforcement learning library distributed as headers, built so that training and inference run on the same code path from a laptop down to a microcontroller. It fits engineers who need to ship a policy, not researchers who need a framework.
- Who is it for?
- Adopt RLtools if you are writing C++ control code for robotics or embedded targets and want the trained policy to be the same code you deploy, with TD3, SAC, PPO and multi-agent PPO already implemented. Do not adopt it if your work is environment research, offline datasets or a Python-first stack: there is no Python training API in the material, and the algorithm set stops at four.
- Can I use it commercially?
- Yes. MIT is a permissive licence: you can use, modify and sell software built on it, as long as you keep its copyright and licence notices.
- Is it still maintained?
- Yes. The repository last received commits 5 days ago.
- What is it written in?
- Mainly C++, according to GitHub's language statistics.
Answers come from the project's GitHub data, last synced on September 15, 2026, and from our analysis. They are not legal advice.
DEEP OPEN-SOURCE ANALYSIS
The deployment gap RLtools is built to close
Most deep RL is written in Python against a simulator, then the trained network is exported and reimplemented somewhere else for the real system. That reimplementation is where numerical drift, missing normalisation and shape mismatches appear. RLtools attacks that by keeping training and inference in one C++17 codebase. The README frames the payoff twice: a quadrotor policy trained in 18s on a 2020 MacBook Pro (M1) using TD3, and inference frequency benchmarks for a two-layer [64, 64] fully-connected network across different microcontrollers, listed under the topic tinyrl. The intended user is an engineer who already writes C++ for a robot or a device and wants the policy to be a compile target, not an artifact. It is not aimed at someone who wants to swap environments in a Python notebook.
Headers, a backend macro, and no build system
The quick start is a single g++ invocation: g++ -std=c++17 -O3 -ffast-math -I include src/rl/zoo/l2f/sac.cpp. There is no CMake step in the README, no package manager entry, no install target. The include directory is the distribution. That is the whole integration story, and it is the reason the library can be dropped into firmware trees that have their own build. The trade-off is explicit in the flags: -ffast-math is on by default in the documented command, which changes floating point semantics. For a control policy that is usually acceptable, but if your environment or your safety envelope depends on strict IEEE behaviour you have to decide that yourself, because the README does not discuss it. Linear algebra comes from a backend selected at compile time. On macOS you append -framework Accelerate -DRL_TOOLS_BACKEND_ENABLE_ACCELERATE, which the README says gives roughly 4s of training on an M3. On Ubuntu you apt install libopenblas-dev and append -lopenblas -DRL_TOOLS_BACKEND_ENABLE_OPENBLAS, roughly 6s on Zen 5. Those two numbers are the only backend performance claims in the material, and both are attributed to the README rather than to independent measurement.
What the algorithm table actually ships
Four algorithm families are listed with runnable examples. TD3 has Pendulum, Racing Car, MuJoCo Ant-v4 and Acrobot examples. PPO has Pendulum, Racing Car, MuJoCo Ant-v4 on CPU and a separate CUDA variant under ppo/cuda. SAC has Pendulum on CPU, Pendulum on CUDA and Acrobot. Multi-Agent PPO has a single example, Bottleneck, at src/rl/zoo/bottleneck-v0/ppo.h. The layout matters more than the list: CPU and CUDA paths live in sibling directories (ppo/cpu/training.h against ppo/cuda/training_ppo.cu, sac/cpu/training.cpp against sac/cuda/sac.cu), so the device is a directory choice, not a runtime flag. That keeps the embedded path free of CUDA headers, which is the point. It also means CUDA and CPU variants are maintained as separate translation units, and the material gives no indication of how much of a training recipe is shared between them. If you need an algorithm outside these four, the library gives you nothing to start from.
Running a Zoo example and seeing the result
The documented loop is short. Compile the Zoo SAC example, then run ./a.out 1337, where the argument is the seed. Then run ./tools/serve.sh and open http://localhost:8000, navigating to the ExTrack UI to watch the quadrotor flying. Two things are worth noting. First, the seed is a positional command line argument, so runs are reproducible by construction rather than by a config file. Second, the visualisation is a separate local server, not a library dependency, which is consistent with a library that has no runtime dependencies to speak of. The README also points at a browser demo at rl.tools, C++ notebooks in the documentation at docs.rl.tools, a Binder link for container tutorials and a Colab notebook for the Python interface. That Colab link is the only mention of Python in the README, and it is a notebook, not a training API. Treat the Python surface as unverified from this material.
Where the header-only approach costs you
Compile time is the first bill. Every translation unit that includes the training headers recompiles the whole template stack, and with -O3 that is not cheap. The README does not report build times, so you should measure your own before deciding how many files include the training path. The second cost is the environment interface. To add your own environment the README sends you to rl-tools/example and the Getting Started page in the documentation, which means the contract is defined outside this repository. That is a real dependency on an external repo staying in sync with the headers. The third is scope. There is no mention of offline RL, no replay dataset format, no distributed training, and no Python training entry point. If your problem is imitation learning from logged data, or you need to run sweeps from a Python orchestration layer, RLtools is the wrong tool and you will spend more time on glue than on training.
How this differs from Stable-Baselines3 or Gymnasium
The obvious alternative is Stable-Baselines3 on top of Gymnasium. The difference is not the algorithm list, which overlaps heavily, but where the artifact lives. Stable-Baselines3 trains a PyTorch policy in Python and you export it for deployment; the deployment step is yours to write. RLtools compiles the policy into your binary, and the same headers that train it also run it, which is what makes the microcontroller inference benchmarks in the README meaningful. The cost of that choice is ecosystem: Gymnasium environments, wrappers and vectorised runners do not apply, and you write your environment against RLtools' own interface. RLtools also stays narrow by design. Four algorithms and a Zoo of examples is a small surface compared to the SB3 algorithm catalogue. If you need breadth, take the Python stack and accept the export step. If you need the policy to be a C++ object with no interpreter in the loop, the Python stack is the wrong starting point.
Maintenance, versioning and the MIT licence
The release history is sparse and readable: v2.0.0 in November 2024, v2.1.0 and v2.2.0 in October 2025. That is a major version roughly a year apart with a minor bump two weeks later, so expect the interface to move in steps rather than continuously. The repository is not archived and the last push is recent, but the material gives no support commitments, no deprecation policy and no compatibility statement across major versions. Pin a tag if you vendor the headers. The licence is MIT, which is permissive and compatible with shipping inside a closed product; the headers carry no copyleft obligation in the usual reading. That is not legal advice, and if you redistribute modified headers you should read the licence text yourself. For a library you compile into a robot, MIT is the least complicated option, which is worth something on its own.
Editorial conclusion
Adopt RLtools if you are writing C++ control code for robotics or embedded targets and want the trained policy to be the same code you deploy, with TD3, SAC, PPO and multi-agent PPO already implemented. Do not adopt it if your work is environment research, offline datasets or a Python-first stack: there is no Python training API in the material, and the algorithm set stops at four. Before committing, verify the OpenBLAS or Accelerate backend actually links on your toolchain, and confirm that the environment interface in rl-tools/example can express your observation and action shapes.
Community notes