Pearl: Meta's Modular Reinforcement Learning Agent Library, Read Before You Install

The gap Pearl is trying to fill between research code and a running agent

Most reinforcement learning repositories are a training script with a paper attached. You get a loop, a network, and a set of hyperparameters tuned for one environment. Moving that code to a different problem means rewriting the loop. Pearl takes the opposite position: the agent is assembled from named parts, and the loop stays fixed. The README describes the goal as enabling researchers and practitioners to develop agents that prioritise cumulative long-term feedback over immediate feedback and can adapt to environments with limited observability, sparse feedback, and high stochasticity. That is a statement about the class of problems, and it tells you the intended user is someone whose environment does not hand over a clean reward signal at every step.

The library is aimed at two groups. The first is industry practitioners who have a decision problem (what to recommend, which action to take, how to allocate) and want to try sequential decision making without building the plumbing. The second is academic researchers who want a common substrate to compare algorithms against. The tutorial list reflects both: a single item recommender system built on a contrived environment derived from the MIND dataset, contextual bandits using UCI datasets, Frozen Lake, DQN and Double DQN on Cart-Pole, and actor-critic methods with safety constraints. Note the range. Bandits and recommender systems sit alongside classic control, which suggests the authors expect most real usage to be closer to the bandit end than the Cart-Pole end.

How a PearlAgent is assembled: learners, buffers, exploration and action representations

The architecture visible in the README is a composition, not an inheritance hierarchy. A PearlAgent is constructed from a policy_learner, a replay_buffer, and, as the serialization note mentions, optional subcomponents such as ExplorationModule and PolicyLearner. The quick start example instantiates DeepQLearning with state_dim, action_space, hidden_dims, training_rounds, and an action_representation_module, then pairs it with BasicReplayBuffer(10_000).

The action representation module is the piece worth pausing on. In the example, OneHotActionTensorRepresentationModule takes max_number_actions and converts discrete actions into a tensor form the learner can consume. This exists because the library is designed for problems where actions are not naturally a small integer set. A recommender system has a catalogue; a bidding system has a continuous range. Separating action representation from the policy learner means you can change how actions are encoded without touching the learning algorithm.

The data flow in the example is explicit and synchronous. env.reset() returns an observation and an action space, which are passed to agent.reset(). Then the loop calls agent.act(exploit=False), passes the returned action to env.step(), feeds the result to agent.observe(), and calls agent.learn(). Nothing is hidden behind a trainer abstraction. If you want to change when learning happens, or batch observations differently, you edit the loop. That is a deliberate trade: more control, more responsibility.

The exploit flag on act() is the exploration switch. Passing exploit=False means the agent should explore; the ExplorationModule subcomponent is what governs how. The README does not spell out the default exploration behaviour for DeepQLearning in the snippet shown, so if exploration policy matters to your setup, read the source rather than assuming.

Installation and the minimum toolchain the README actually specifies

Installation is a source install, not a package install. The README gives two commands and two version constraints:

git clone https://github.com/facebookresearch/Pearl.git cd Pearl pip install -e .

The constraints are pip version 21.3 or later and setuptools version 64 or later. Those are not decorative. Editable installs of a project with a modern build backend will fail on older pip, and the failure message is usually unhelpful. Check both before you clone.

There is no PyPI package mentioned and no release artefacts retrieved. That matters for how you consume the library. With pip install -e . you are installing a working copy, which means your environment is tied to whatever commit you cloned. If you need reproducibility across machines, record the commit hash yourself, because there is no version number to pin to beyond the v0.1 beta label in the README.

The quick start requires a Gym-style environment. GymEnvironment("CartPole-v1") is the example, and the README states that users can replace the environment with any real-world problems. That sentence carries more weight than it looks. Your production environment needs to expose reset(), step(), an action_space with an n attribute, and an observation_space with a shape. If your system cannot be shaped into that interface, Pearl will not wrap it for you.

State dict serialization and the compare method you are now required to write

The January 22, 2025 note describes a serialization mechanism that mirrors PyTorch. A PearlAgent can produce a state dict, which you save with torch.save and reload with torch.load into a structurally identical agent. The README's example then asserts that agent2.compare(agent) returns an empty string.

That assertion is the interesting part. compare is a newly introduced method, and the README states that when defining your own components you must define it, returning a string listing differences between two components. It is described as a general comparison method for testing purposes. In practice this is a contract the library imposes on extension authors: if you write a custom policy learner or exploration module, you owe Pearl a diff function. That is unusual and it is a real cost. It also gives you a cheap correctness check after a save/load cycle, which is exactly the kind of bug that silently degrades an agent in production.

The other constraint is the extra state problem. Attributes that are not parameters, buffers, or sub-modules are not captured automatically. The README points to ActorCriticBase.get_extra_state as an example of the pattern, and says you define get_extra_state and set_extra_state just as you would in PyTorch. So a custom component with a plain Python counter or a lookup table needs both methods, or that state vanishes on reload. The README is explicit about this rather than burying it, which is a point in its favour.

What Pearl does not give you: releases, upgrade paths and production guarantees

The README calls Pearl production-ready. The material available does not support that claim in the way an operations team would want it supported. There are no retrieved releases, so there is no changelog to read before upgrading and no version to pin. The library is labelled v0.1 beta. Beta and production-ready are in tension, and the README does not resolve it.

More concretely, the extension contract changed. The serialization note says components must now define compare, and that non-standard attributes now need get_extra_state and set_extra_state. Anyone who wrote custom components against an earlier commit has work to do, and without releases there is no deprecation window. You find out by reading the commit history or by your code breaking.

There is also a scope limitation that the README's framing obscures. Pearl gives you algorithms and interfaces. It does not give you a training orchestration layer, a distributed rollout system, or an evaluation harness. The quick start is a single-threaded while loop with one environment. Scaling that to a fleet of environments, handling checkpoint promotion, or running offline evaluation is your problem. The library is a component kit, and the README's own description of the design supports that reading: it says Pearl was built with a modular design so that practitioners or researchers can select any subset and combine features to construct an agent. Selecting a subset is the point, and it is also the work.

Pearl against Stable-Baselines3: composability versus a managed trainer

The natural comparison is Stable-Baselines3, which also provides RL algorithms behind a consistent API. The difference is where the abstraction line sits. Stable-Baselines3 hands you a model object with a learn() method that owns the training loop, the rollout collection, and the logging. You configure it and call learn(total_timesteps=N). Pearl hands you the loop itself, as the quick start shows, and expects you to drive act, observe, and learn in whatever order your problem requires.

That difference decides which one fits. If your problem is a standard Gym environment and you want results with minimal integration, Stable-Baselines3's managed loop is less code and fewer decisions. If your problem is a recommender system where the action space is a catalogue and the feedback arrives asynchronously, the managed loop is an obstacle, and Pearl's separation of action_representation_module from policy_learner is the thing you actually need. Pearl's contextual bandit tutorial, which the README says tests neural implementations of SquareCB, LinUCB, and LinTS against UCI datasets, sits in territory Stable-Baselines3 does not cover as directly.

The honest summary is that these are not competing on the same axis. One optimises for time-to-first-result on standard benchmarks. The other optimises for the ability to swap one part of the agent without disturbing the rest.

Licence, maintenance and what the MIT terms mean for your fork

Pearl is MIT licensed, per the badge and the licence identifier in the repository metadata. MIT is permissive: you can use it commercially, modify it, and redistribute it, provided the copyright notice and permission notice are retained. It does not grant patent rights and it comes with no warranty. If you fork Pearl and ship it inside a product, the obligation is the notice, not the source. That is a lighter burden than a copyleft licence would impose, and it is the main reason a library like this is usable inside a company at all. This is a description of the licence text, not legal advice; your counsel should confirm how it interacts with your distribution model.

Maintenance cost is harder to estimate from the material. The repository is not archived and the last push is recent, so there is active work. But the absence of releases means every upgrade is a commit-range review. Budget for reading diffs rather than reading changelogs. The serialization change is the precedent: a required new method on every custom component, delivered without a version boundary. If you extend Pearl heavily, expect to spend time on each pull, and consider vendoring the components you depend on so an upstream change cannot break your build without warning.

Who should clone Pearl and who should close the tab

Clone it if your problem is sequential decision making over a non-trivial action space and you are willing to own the training loop. The contextual bandit and recommender tutorials are the strongest signal about intended use, and they are the two notebooks to read first, before the Cart-Pole example, because they show the parts of the library that differ from a standard RL framework. Clone it also if you want to compare bandit algorithms under one interface; the README states the contextual bandit tutorial tests SquareCB, LinUCB, and LinTS, which is a comparison harness you would otherwise write yourself.

Do not clone it if you need a versioned dependency, a documented upgrade path, or a framework that manages rollouts and checkpointing for you. Do not clone it if your environment cannot be expressed as reset/step with an action_space and observation_space, because nothing in the material suggests Pearl will bridge that gap. And do not clone it expecting the production-ready label to mean operational tooling; the README's own description of the design points to a component kit, and a component kit is what you get.

Editorial conclusion

Adopt Pearl if you are already comfortable with PyTorch RL internals and want a component system you can rearrange, particularly for contextual bandits or recommender-style problems where the tutorials give you a starting point. Do not adopt it if you need a pinned release, a documented upgrade path, or a library that hides its abstractions from you. Before writing any code, verify three things: that pip and setuptools meet the stated minimums, that your environment can be wrapped to satisfy the GymEnvironment interface, and that the component you intend to customise implements both get_extra_state/set_extra_state and compare, because without compare your serialization round-trip has no test.

Official sources

1. facebookresearch/Pearl on GitHub
2. Issues
3. License: MIT
4. README