ReinforcementLearning.jl: A Component-Based RL Stack for Julia
A reinforcement learning package for Julia
At a glance
- What is it?
- ReinforcementLearning.jl is a Julia package that splits reinforcement learning into four swappable pieces: policy, environment, stop condition, and hook. It is aimed at researchers who want to write new algorithms rather than call someone else's trainer, and the trade-off is that you assemble the loop yourself.
- Who is it for?
- Adopt it if you are writing a new RL algorithm in Julia and want to reuse environments and run-loop plumbing instead of rebuilding them. Do not adopt it if you need a finished trainer with published benchmark numbers, since the README's speed and feature sections are still marked TODO.
- Can I use it commercially?
- Check first. The repository uses a licence we do not classify automatically, so read its LICENSE file before any commercial use.
- Is it still maintained?
- Yes. The repository last received commits 57 days ago.
- What is it written in?
- Mainly Julia, according to GitHub's language statistics.
Answers come from the project's GitHub data, last synced on September 15, 2026, and from our analysis. They are not legal advice.
DEEP OPEN-SOURCE ANALYSIS
The four-slot experiment loop, and who it is built for
The README's quick-start example is the whole design in miniature: run(RandomPolicy(), CartPoleEnv(), StopAfterNSteps(1_000), TotalRewardPerEpisode()). Four arguments, four roles. A policy that picks actions, an environment that responds to them, a condition that decides when to stop, and a hook that records something while the loop runs. The package treats those as the extension points, and everything else is arrangement.
That tells you the intended user. Someone who wants to try a new exploration rule, a new replay scheme, or a new logging signal can write one object and drop it into an existing loop. The README states the design principles directly: reusability and extensibility, easy experimentation, and reproducibility "from traditional tabular methods to modern deep reinforcement learning algorithms". The topics list backs the deep end of that claim with deep-q-network and deep-reinforcement-learning tags.
What it is not aimed at is the person who wants a single call that trains an agent and prints a score. The example above never learns anything. RandomPolicy is described as generating a random action at each step, and TotalRewardPerEpisode only collects per-episode reward. It is a smoke test of the interface, not a baseline. If you want a trained agent you supply the learning policy, and the package supplies the frame it plugs into.
Policies, environments, hooks: what the abstractions actually buy
Each slot has a named abstract type in ReinforcementLearningBase.jl: AbstractPolicy, AbstractEnv, and AbstractHook. That base package is the bottom of the dependency tree, and both ReinforcementLearningEnvironments.jl and ReinforcementLearningCore.jl sit on top of it. The practical consequence is that an environment author never needs to depend on the policy code, and a policy author never needs to depend on the environment code. They only share the interface definitions.
The hook is the piece most people skip past. TotalRewardPerEpisode is described as one of the most common AbstractHooks, used to collect the total reward of each episode. The plural in the name matters: hooks are how you observe a run without editing the algorithm. If you want to log action distributions or count environment calls, you add a hook rather than fork a policy. Stop conditions work the same way, with StopAfterNSteps(1_000) as the README's example, so horizon and termination policy stay outside the agent.
This is the standard decomposition in RL libraries, and the interesting part here is that it is enforced by package boundaries rather than by convention inside one module. The cost is indirection: to understand a full run you follow a call from the core loop out into the base interface and back into whatever concrete type the user supplied. The benefit is that replacing any one of the four does not touch the other three.
The wrapper package and the subpackages under it
ReinforcementLearning.jl is not a monolith. The README says plainly that it "is just a wrapper around several other subpackages", and the ASCII diagram in the repository shows the shape: ReinforcementLearningBase.jl at the bottom, with ReinforcementLearningEnvironments.jl and ReinforcementLearningCore.jl branching off it, then ReinforcementLearningZoo.jl under Core, and DistributedReinforcementLearning.jl under Zoo.
The release history confirms these are versioned separately. ReinforcementLearningCore-v0.15.5 shipped on 2025-01-13, following v0.15.4 in December 2024, while ReinforcementLearningFarm-v0.0.3 landed on 2024-12-18. Core is at 0.15.x and Farm is at 0.0.x, which is a meaningful signal about where churn is concentrated. A 0.15 series that moves in month-sized increments is a package still finding its edges; a 0.0.3 is earlier than that.
For a user this split cuts both ways. You can depend on just ReinforcementLearningBase.jl if you only need the interface types, which keeps your dependency footprint small. On the other hand, "using ReinforcementLearning" pulls the wrapper, and the wrapper's own version number tells you very little about which Core you will resolve. Pin the subpackage you actually call into.
Getting a run going: install, then the four arguments
Installation is the standard Julia package flow. The README gives it as:
julia> ] add ReinforcementLearning
julia> using ReinforcementLearning
Then the example run:
julia> run( RandomPolicy(), CartPoleEnv(), StopAfterNSteps(1_000), TotalRewardPerEpisode() )
There is no configuration file and no key-value config to set. Everything is a constructor argument. StopAfterNSteps(1_000) takes the step count inline; the hook is constructed empty and fills as the loop runs. That is the entire documented surface for getting started, and the README points to the tutorial page for how the four components are assembled for other problems.
Because there are no config keys, there is also no config schema to validate. If you are wiring this into a larger pipeline, the integration point is the Julia call itself, not a settings file. That is convenient for interactive work and awkward if you want to record experiment parameters declaratively. Nothing in the supplied material describes a config-loading path, so treat any such expectation as unverified.
Where the documentation stops short
The README has a section titled "Why ReinforcementLearning.jl?" with two subsections, "Fast Speed" and "Feature Rich". Both contain the literal text [TODO:]. That is the most important limitation in the material, and it is not a small one. Two of the four reasons a reader would pick this package over an alternative are placeholders.
The package's own tagline is a link to "Make It Work Make It Right Make It Fast", which reads as an honest ordering rather than a performance claim. Combined with the empty speed section, the reasonable assumption is that performance has not been characterised in the README, and no throughput or sample-efficiency figure appears anywhere in the supplied text. Anyone choosing this for a workload where wall-clock time matters should measure it themselves rather than trust a summary.
The same applies to algorithm coverage. The "Feature Rich" heading is empty, so the README does not enumerate which algorithms ship in ReinforcementLearningZoo.jl. The topics list names deep-q-network, but that is a repository tag, not a supported-algorithm list. If your choice hinges on a specific algorithm being available, confirm it in the Zoo source before you commit, because the top-level README will not tell you.
The licence metadata disagrees with the badge
The repository metadata reports the licence as NOASSERTION, which means GitHub's detector could not map the LICENSE.md file to a known identifier. The README, meanwhile, displays an MIT badge linking to LICENSE.md. These two signals point the same direction but are not the same claim, and the discrepancy is worth resolving before you depend on the package.
The pragmatic step is to open LICENSE.md directly and read the text rather than the badge. If it is standard MIT, the obligations are minimal and the usual attribution requirement applies. If the file has been modified, or if different subpackages carry different terms, the wrapper's single badge will not reflect that. Because the subpackages are distributed separately, a per-subpackage licence check is the safer habit. None of this is legal advice; it is a note that the machine-readable field and the human-readable badge do not agree here.
Maintenance cost and what upgrading between 0.15.x releases involves
The release cadence visible in the material is Core at 0.15.4 in December 2024 and 0.15.5 in January 2025, with the repository last pushed in July 2026. Patch releases inside a 0.15 series are usually internal fixes, but the 0.x major version means the maintainers have not promised API stability, and the README's own design emphasis on extensibility implies the interfaces are expected to keep moving as new algorithms land.
In practice that means upgrade cost concentrates in whatever concrete types you wrote. If you implemented an AbstractPolicy or an AbstractHook, a change to the base interface can break your code even when you did not bump the wrapper. Pinning ReinforcementLearningCore.jl to a specific version, rather than letting the resolver float, is the cheap insurance. The Farm subpackage at 0.0.3 is a stronger warning: pre-0.1 packages are the ones most likely to reorganise.
The other maintenance consideration is the dependency graph itself. Depending on the wrapper means depending on Base, Environments, Core, and Zoo together, even if you only use one. For a research script that is fine. For a library you intend to publish, depending on ReinforcementLearningBase.jl alone gives you the interface types without dragging the rest along, and that is a decision worth making deliberately rather than by default.
How it differs from Stable-Baselines3 and the trainer-first model
The obvious comparison is Stable-Baselines3, which is built around ready-made algorithms you instantiate and train: you pick PPO or DQN, hand it an environment that follows the Gymnasium interface, call learn, and get a saved model. The algorithm is the unit of reuse. Here the unit of reuse is the component, and the algorithm is something you assemble from a policy, a stop condition and hooks. Neither approach is wrong; they answer different questions.
If your goal is to train a known algorithm on a known environment and compare against published numbers, the trainer-first model gets you there with less code, and ReinforcementLearning.jl's empty speed and feature sections give you no counterargument. If your goal is to modify the internals of an algorithm, the component model is the better fit, because swapping an exploration policy or adding a hook does not require touching the training loop. The Julia environment interface is also its own thing: CartPoleEnv is an AbstractEnv from ReinforcementLearningEnvironments.jl, not a Gymnasium wrapper, so porting an existing Python environment is real work rather than a one-line adapter.
The reproducibility principle in the README is the other differentiator worth naming. Keeping the stop condition and the hooks as explicit objects means the run configuration is visible in the call itself, which is a reasonable foundation for reproducing an experiment. It is a foundation, not a guarantee, since nothing in the material describes seeding or result storage.
Editorial conclusion
Adopt it if you are writing a new RL algorithm in Julia and want to reuse environments and run-loop plumbing instead of rebuilding them. Do not adopt it if you need a finished trainer with published benchmark numbers, since the README's speed and feature sections are still marked TODO. Before committing, verify the licence text in LICENSE.md against the NOASSERTION metadata, and check the version of ReinforcementLearningCore.jl you will actually resolve against the API in the current docs.
Community notes