ReinforcementLearning.jl: A Composable RL Research Package for Julia

What ReinforcementLearning.jl Is and Who It Is For

ReinforcementLearning.jl is a package for reinforcement learning research in Julia. The README describes its three design principles as reusability and extensibility, easy experimentation for running benchmark experiments and comparing algorithms, and reproducibility from tabular to deep RL methods.

The package targets researchers who already use Julia and want to implement new RL algorithms or run standard benchmark experiments. It is not a production deployment framework; the emphasis is on the experiment loop, algorithm comparison, and reproducibility. The README states that the project is written in pure Julia and encourages users to consider supporting the JuliaLang organization if they find it useful.

The package installs through Julia's standard package manager and is registered in the JuliaHub package index.

The Four-Component Experiment API

The README presents a single four-line example that demonstrates the core API:

julia> ] add ReinforcementLearning

julia> using ReinforcementLearning

julia> run(
           RandomPolicy(),
           CartPoleEnv(),
           StopAfterNSteps(1_000),
           TotalRewardPerEpisode()
       )

This example runs a RandomPolicy on the CartPoleEnv environment for 1,000 steps and collects the total reward per episode. The four arguments to run() represent the four core components of every experiment in the package.

A Policy (implementing AbstractPolicy) defines how actions are chosen. RandomPolicy is the simplest case and generates a random action at each step. An Environment (implementing AbstractEnv) defines the state space, action space, and transition dynamics. CartPoleEnv is a standard benchmark for RL algorithm testing. A Stop Condition tells the experiment when to terminate. StopAfterNSteps(1_000) halts after 1,000 environment steps. A Hook (implementing AbstractHook) collects data during the run. TotalRewardPerEpisode accumulates the reward sum for each episode. The README points to a tutorial page on juliareinforcementlearning.org for a deeper explanation of how these four components combine.

Package Structure and the Subpackage Architecture

ReinforcementLearning.jl is a thin wrapper around a collection of subpackages. The README describes the relationship between them with a tree diagram showing that the top-level package imports and re-exports the subpackages, which cover different responsibilities.

The repository structure includes a src/ directory, a docs/ directory, a test/ directory, and a Project.toml that lists the actual package dependencies. The subpackages are developed in the JuliaReinforcementLearning GitHub organization. This architecture means you can depend on individual subpackages if you only need part of the stack, rather than importing the entire framework.

The NEWS.md file tracks changes between versions. The CITATION.bib file provides BibTeX entries for citing the package in academic publications. The README notes an introduction blog post, 'An Introduction to ReinforcementLearning.jl,' as the most recommended reading for understanding the design philosophy behind the package.

Getting Help and Community Resources

The README lists four paths for getting help. First, the online documentation at juliareinforcementlearning.org includes examples and API references with a search feature. Second, the Julia Slack workspace has a #reinforcement-learnin channel (the README's spelling). Third, the Julia Discourse forum accepts questions under the Machine Learning category with a reinforcement-learning tag. Fourth, bugs with a minimal working example and reproduction steps should be filed as GitHub issues.

This is a community-maintained project. The README is explicit that ongoing development happens in contributors' spare time, without a dedicated funding source or company backing. Deep RL research requires large compute resources, and the README notes that these are not available to individual contributors. If you or your organization can contribute computing resources, the README invites contact.

Limitations and Honest Assessment of the README

The README has two placeholder sections labeled [TODO:]. The section on fast speed and the section on feature richness both contain the text [TODO:] with no content. This is a significant gap in the documentation. Readers cannot determine from the README alone which specific algorithms are implemented, what benchmark performance looks like, or how the speed compares to other Julia or Python RL frameworks.

This does not mean the package lacks these things, but it does mean the documentation is incomplete. Engineers evaluating the package for a specific algorithm will need to look at the subpackage repositories in the JuliaReinforcementLearning organization or the API documentation at juliareinforcementlearning.org rather than relying on the README.

The latest release is ReinforcementLearningCore-v0.15.5, published on 2025-01-13. The last push to the repository was on 2026-07-21, suggesting ongoing maintenance, but the release version of the core component trails that date by over a year and a half.

The license is listed as NOASSERTION in the repository metadata, which means the standard automated license detection tools did not identify a clear license. The README links to a LICENSE.md file and the repository contains that file, but users who need a confirmed license identifier should inspect LICENSE.md directly before depending on the package.

Comparison with Python RL Frameworks

The direct alternative for researchers outside the Julia ecosystem is Stable-Baselines3, a Python library that provides implementations of common deep RL algorithms built on PyTorch. Stable-Baselines3 uses a similar component model but targets Python users and provides out-of-the-box implementations of algorithms like PPO, SAC, and TD3 with documented hyperparameters.

CleanRL is another Python alternative that takes a different design approach: single-file implementations of RL algorithms that are fully self-contained and readable, prioritizing clarity over extensibility. CleanRL is useful for researchers who want to understand exactly what every line of the algorithm does.

ReinforcementLearning.jl's distinguishing design is that it lives in the Julia ecosystem, benefiting from Julia's multiple dispatch and just-in-time compilation. The README frames reusability and extensibility as primary goals, which matches the four-component composable API. Teams already invested in Julia for scientific computing will find it a natural fit. Researchers whose primary language is Python will incur a language-switching cost that neither framework requires.

Editorial conclusion

ReinforcementLearning.jl is the right starting point for Julia users who want to run and compare RL algorithms in a reproducible environment without rewriting the experiment scaffolding each time. The four-component API (Policy, Environment, Stop Condition, Hook) is concise and covers the standard experiment loop. Researchers coming from Python-based frameworks like Stable-Baselines3 or CleanRL will find the composability model similar in intent but will need to learn Julia's type system and multiple dispatch. The README notes that modern deep RL requires substantial computing resources, which individual contributors cannot always provide. The last push was on 2026-07-21. Before adopting it, verify that the specific algorithm you need is implemented in one of the subpackages, since the README's feature section contains TODO placeholders that indicate incomplete documentation.

Frequently asked questions

Official sources

1. Issues
2. JuliaReinforcementLearning/ReinforcementLearning.jl on GitHub
3. Project website
4. README
5. Releases