RLMatrix: Deep Reinforcement Learning for C# Developers on TorchSharp

The gap RLMatrix is trying to fill

Reinforcement learning tooling is concentrated in Python. If your product is a .NET service, a Unity title or a Godot game, that concentration forces an awkward split: train in Python, export weights, then reimplement the inference path in C# and keep the two in sync. RLMatrix targets that split. It is a C# library built on TorchSharp, the .NET binding to LibTorch, and it ships PPO, DQN with the modifications the README says go up to C51 and DQN Rainbow, and GAIL. The intended audience is a C# developer who wants the training loop and the agent in the same language and the same type system, rather than a researcher chasing the newest published method. The README states the framework is battle-tested in Unity and Godot, which is the strongest concrete positioning in the document: game engines are where the C#-only constraint bites hardest.

Source generation instead of a hand-written training loop

The mechanism the README emphasizes most is C# source generation. Rather than wiring an agent by hand, you describe your environment and the toolkit generates the surrounding code, so the domain problem stays in view. The README describes this as focusing on your domain problem rather than wrestling with complex API requirements. What the README does not show is a template or a generated file, so the exact boundary between what you write and what is emitted cannot be confirmed from the material here. The same caution applies to the multi-head claim: the README says a single agent can handle continuous, discrete and mixed action spaces simultaneously, which matters because most libraries force you to pick one head type per agent. RNN support is presented as an option toggle for sequential or partial observability problems, again without a code sample. Treat these as the design intent and check the generated output yourself.

Training across parallel and networked environments

The scaling story in the README has two layers. The first is multi-environment training, described as parallel and optionally networked. The second is what the README calls industrial-grade distributed training: a fault-tolerant networked architecture. Both point at the same idea, that rollouts are collected from more than one environment process and fed back to a learner, but the README gives no protocol, no configuration keys and no throughput figures. There is also a built-in dashboard for visualizing training metrics, which is the one piece of operational tooling named in the document. For a team used to TensorBoard, a built-in dashboard is convenient but it is also a smaller ecosystem: you get what the project ships. The absence of any published benchmark means the claim that RLMatrix is faster and more stable than stable-baselines, ml-agents and Godot RL agents should be read as a project claim, not a measured result.

Installing and getting a first agent running

The README does not contain a quickstart snippet. It directs readers to the official documentation at rlmatrix.net for getting-started guides, and it carries a warning that the examples in the repository are being updated, pointing to a separate testing repository for more current examples. The installable artifact is the RLMatrix package on NuGet, referenced by the badge at the top of the README, so the first step is adding that package to a .NET project and letting NuGet pull TorchSharp transitively. Beyond the package name, every other concrete detail (the environment interface you implement, the configuration keys for algorithm choice, the toggle for RNN, the dashboard endpoint) lives in the external documentation rather than in the repository. That is the single biggest practical friction for an evaluator: you cannot judge the API from the README alone.

Where the project is thin

The roadmap is candid about unfinished work, and it reads as a list of the current weak points: improve documentation and examples, expand the algorithm library, further enhance Godot and Unity integration, add more imitation learning and GAIL tooling, test and optimize non-Windows non-CUDA platforms, and simplify inference and deployment. The platform item deserves attention. If the supported path is Windows with CUDA, then Linux CI runners and CPU-only machines are explicitly not yet optimized, which affects both training throughput and where you can host inference. The examples warning compounds this: a reader arriving at the repository today is told the in-repo examples are stale and redirected elsewhere. None of this is unusual for a young library, but it means the README is closer to a feature list than to a validated guide.

RLMatrix against Stable-Baselines3

The obvious alternative is Stable-Baselines3, the Python library the README names as a performance comparison. The difference in approach is structural rather than algorithmic. Stable-Baselines3 lives in the Python ecosystem, so training, evaluation and logging sit next to NumPy, Gymnasium environments and the wider research code that surrounds them. RLMatrix lives inside .NET, so the agent shares a runtime, a type system and a build with the application that will use it. That eliminates serialization and reimplementation between training and deployment, which is a real cost in a Unity or Godot project. It also means you inherit TorchSharp's platform matrix and the library's smaller algorithm set. If your environment is already a Python simulation and your deployment target is a service, Stable-Baselines3 is the lower-friction choice. If your environment is a C# game loop, the reverse holds.

Licence and the cost of staying current

RLMatrix is MIT licensed, which permits commercial use, modification and redistribution provided the copyright notice and permission notice are retained. That is the permissive end of the spectrum and it removes the licensing negotiation that some RL frameworks require. It does not remove the maintenance cost. There are no retrieved releases in the material supplied, so versioning and upgrade cadence cannot be assessed from the repository metadata alone; the last push date of 2026-04-23 indicates active development rather than a frozen codebase. Because the library sits on TorchSharp, its upgrade path is partly tied to LibTorch and the .NET target framework you build against, and the roadmap's unresolved platform work means a bump in either dependency can land before the non-Windows path is settled. Pin the RLMatrix and TorchSharp versions together and test the upgrade on your own environment before moving a training run onto it.

Editorial conclusion

RLMatrix is worth evaluating if your team already ships .NET code, your agents live in Unity or Godot, and you need PPO, DQN variants or GAIL without a Python training loop in the middle. It is the wrong tool if you need the breadth of algorithm coverage and published reference results that the Python ecosystem provides, since the README lists no benchmark numbers and points readers to a separate testing repository for current examples. Before adopting, verify three things: that the RLMatrix NuGet package resolves against your target framework and TorchSharp version, that the source-generation templates cover the mixed action space you actually need, and that the multi-environment training path works on your platform, since the roadmap still lists non-Windows non-CUDA support as unfinished work.

Official sources

1. asieradzk/RL_Matrix on GitHub
2. Issues
3. License: MIT
4. README