TorchRL: a TensorDict-first toolkit for building RL systems in PyTorch
A modular, primitive-first, python-first PyTorch library for Reinforcement Learning.
At a glance
- What is it?
- TorchRL is not an algorithm you run, it is a set of composable pieces for environments, policies, collectors, replay buffers and losses, all passing one TensorDict through the loop. It fits engineers who want to assemble RL pipelines in PyTorch rather than adopt someone else's trainer.
- Who is it for?
- Adopt TorchRL if your pipeline already lives in PyTorch and you want named, device-aware data flowing from environment to loss without hand-written glue. Do not adopt it if you want a single command that trains a published baseline on a fixed benchmark, or if your team cannot absorb TensorDict as a dependency.
- Can I use it commercially?
- Yes. MIT is a permissive licence: you can use, modify and sell software built on it, as long as you keep its copyright and licence notices.
- Is it still maintained?
- Yes. The repository last received commits 1 day ago.
- What is it written in?
- Mainly Python, according to GitHub's language statistics.
Answers come from the project's GitHub data, last synced on September 15, 2026, and from our analysis. They are not legal advice.
DEEP OPEN-SOURCE ANALYSIS
The glue code problem TorchRL is aimed at
Reinforcement learning codebases tend to rot in the same place. One environment returns a tuple, another returns a dict, recurrent states live in a separate list, done masks sit beside the data instead of inside it, and the loss silently assumes a batch layout that the collector does not guarantee. The README names exactly this: 'RL code tends to accumulate special cases.' TorchRL's answer is to make those assumptions explicit by putting every field into a single container with names.
The target user is an engineer who already writes PyTorch and wants to build a training loop, not an engineer who wants to download a finished agent. The README is explicit that TorchRL 'is not a single algorithm implementation or a narrow benchmark suite.' The repository does ship a sota-implementations directory, but the library's centre of gravity is the components. If your job is to reproduce a paper number on a standard benchmark, you are not the primary audience.
TensorDict as the single object moving through the loop
The mechanism is a data flow, not an abstraction. A TensorDict enters a policy module, which writes actions and log-probs under named keys. The environment reads those actions and writes next observations, rewards and done flags back into the same structure. A collector batches trajectories from one or many workers, a replay buffer stores and samples them, a loss module reads named keys and writes differentiable losses, and an ordinary PyTorch optimizer updates ordinary parameters.
TensorDict itself comes from a separate repository, pytorch/tensordict, and the README describes it as a dictionary-like tensor container with PyTorch operations, device transfers, shared-memory support, memmaps, lazy views and nn.Module wrappers. That last point matters: because it wraps as a module, the container can pass through nn.Sequential-style code without special handling. The README's example shows stack, reshape, .to("cuda") and slicing all preserving structure and operating on every compatible value.
The design constraint this imposes is real. Every component in the stack has to agree on key names. Change "observation" to "obs" in your environment and the policy, the transform chain and the loss all need to match. TorchRL trades implicit coupling for explicit coupling. That is a better failure mode, but it is still coupling.
Getting a rollout running from the documented example
The README gives a complete local rollout. It imports TensorDictModule from tensordict.nn and PendulumEnv, StepCounter and TransformedEnv from torchrl.envs. The environment is constructed as TransformedEnv(PendulumEnv(), StepCounter(max_steps=200)), so the step limit is a transform stacked on the base environment rather than a constructor argument.
The policy is a plain nn.Sequential wrapped in TensorDictModule with in_keys=["observation"] and out_keys=["action"]. The rollout call is env.rollout(max_steps=32, policy=policy), and the README asserts that rollout.batch_size equals torch.Size([32]) and that rollout["next", "reward"].shape[:1] equals torch.Size([32]).
Two details are worth noting for anyone adapting this. The example uses nn.LazyLinear, so the first forward pass determines the input width and the module must see data before its parameters are complete. And the "next" key in rollout["next", "reward"] is a nested access path, not a string concatenation. The README states that the same keys-and-TensorDict interface is used by batched environments, multi-agent tasks, collectors, replay buffers, recurrent modules, transforms and losses. Installation itself is not shown in the supplied material; the README links to a getting-started page and the PyPI badges indicate packages named torchrl and torchrl-nightly, but the exact pip command is not quoted here.
Where the abstraction costs you
The clearest limitation is the dependency shape. TensorDict is a separate project with its own release cadence. TorchRL's data model is only as stable as that external package, and the README's own framing ties the library's three core ideas to it. Teams that cannot add a second non-trivial dependency to their stack should treat this as a real cost, not a footnote.
A second limitation is version churn. Three patch releases landed between June and July 2026, and the README describes a 0.13 cycle that changed recurrent paths, added MuJoCo environments, expanded multi-agent coverage and reworked collector and replay-buffer ergonomics. That is a fast-moving surface. Code written against 0.13.1 may need attention at 0.13.3, and the nightly package exists for users who want to track main.
A third case where TorchRL is the wrong tool: single-file research scripts. If your entire training loop is 200 lines and you never intend to vectorize, distribute or swap the buffer, the TensorDict indirection adds a layer between you and the tensors without buying anything. The README's own claim is that research code should scale 'without changing the data model.' If you have no scaling ambition, you are paying the tax without the benefit.
How this differs from Stable-Baselines3
The natural comparison is Stable-Baselines3. SB3 ships finished algorithms behind a small API: you pick PPO or SAC, hand it a Gymnasium environment, call learn(), and you get a trained model. The algorithm is the unit of composition. TorchRL inverts this. The unit of composition is the component, and the algorithm is something you assemble from a policy module, a collector, a loss and an optimizer.
That difference shows up in what each library makes easy. SB3 makes it easy to get a baseline number on a standard environment with almost no code, and hard to change the internals of the loss without forking. TorchRL makes it easy to rewire the loss, swap the replay buffer for a prioritized one, or move from a single environment to a vectorized multiprocess collector, and hard to get a result without writing the loop yourself. Neither is better in the abstract. The choice depends on whether your bottleneck is setup time or the ability to change the internals.
Recent work and what it signals about maintenance
The 0.13 highlights are specific enough to read as a direction of travel. Recurrent RL got faster paths including scan and Triton GRU/LSTM reset handling. Custom MuJoCo environments and macro-control policies were added. Multi-agent coverage grew through MAPPO, IPPO, MultiAgentGAE, value-normalization utilities and mixer configs. Collector and replay-buffer ergonomics improved with async prioritized writes, ordered storage access, compact observations, HER, and optional CUDA wheels for CUDA-based prioritized replay-buffer kernels.
That last item is the one to watch if you plan to use prioritized replay on GPU. The README describes the CUDA kernels as optional wheels, which means the default install may not include them. Whether your environment needs the extra install step is something you would have to confirm against the installation docs, which are not reproduced in the supplied material.
The repository also publishes CI timing, flaky-test and benchmark dashboards. Those are process signals rather than quality signals, but they indicate the project tracks its own test stability rather than treating it as invisible.
Licence, upgrade cost and what to check before adopting
TorchRL is MIT licensed, which is permissive and places few obligations on downstream use beyond retaining the licence notice. TensorDict, the separate dependency the data model rests on, is a different repository; check its licence independently rather than assuming it matches. Nothing here is legal advice.
The upgrade cost is the part that is easy to underestimate. Three patch releases in roughly five weeks, plus a nightly channel, plus a data model owned by an external package, means a pinned version is the sane default for anything in production. The README does not describe a deprecation policy or a compatibility guarantee between minor versions, so treat version bumps as something to test rather than something to assume.
Before adopting, run the README's Pendulum example as written, then replace PendulumEnv with your own environment and check whether the observation and reward keys survive the TransformedEnv wrapper without renaming. That single test tells you more about fit than any feature list, because it exercises the exact coupling TorchRL is built around.
Editorial conclusion
Adopt TorchRL if your pipeline already lives in PyTorch and you want named, device-aware data flowing from environment to loss without hand-written glue. Do not adopt it if you want a single command that trains a published baseline on a fixed benchmark, or if your team cannot absorb TensorDict as a dependency. Before committing, install torchrl from PyPI, run the Pendulum rollout from the README, and check that your own environment's observation and reward shapes survive a TransformedEnv wrapper unchanged.
Community notes