PARL turns a Python class into a remote actor with one decorator, and xparl runs submitted code on every node

Three classes build an agent: Model, Algorithm and Agent

PARL builds an agent recursively out of three abstractions. Model constructs the forward network that defines a policy network or critic network, taking state as input. Algorithm describes the mechanism that updates parameters in a Model and often contains at least one Model. Agent is the data bridge between the environment and the algorithm: it handles data I/O with the outside environment and describes preprocessing before data is fed into training.

The framework lists four goals. Reproducible algorithms that stably reproduce the results of many influential reinforcement learning papers. Large scale, with parallelization over thousands of CPUs and multiple GPUs. Reusable, in that an algorithm can be adapted to a new task by defining a forward network, after which the training mechanism is built automatically. Extensible, by inheriting the abstract classes.

The split has a practical effect. Swapping the network touches only Model, and changing the update rule touches only Algorithm, so an experiment that changes one does not rewrite the other.

parl.remote_class moves an object onto another machine

Parallelization is the headline feature. PARL describes a compact API where code becomes a parallelized version by adding a decorator. The Hello World example in the repository shows it:

#============Agent.py=================
@parl.remote_class
class Agent(object):

    def say_hello(self):
        print("Hello World!")

    def sum(self, a, b):
        return a+b

parl.connect('localhost:8037')
agent = Agent()
agent.say_hello()
ans = agent.sum(1,5) # it runs remotely, without consuming any local computation resources

Two steps are needed. Decorate a class with parl.remote_class, which turns it into a class that can run on other CPUs or machines, and call parl.connect before creating an object. Calling any function on the object does not consume local computation resources, because execution happens elsewhere.

In the described architecture, real actors run on a CPU cluster while the learner runs on the local GPU with several remote actors attached. Users write code as if it were multi-thread code, with actors consuming remote resources. The repository points to IMPALA and A2C as parallelized examples.

Installation is pip install parl, with paddlepaddle optional

Installation is a single command:

pip install parl

The dependency list names Python 3.6 or newer, with Python 3.8 or newer preferable for distributed training. The paddlepaddle package, version 2.3.1 or higher, is marked optional if you only want the APIs related to parallelization alone.

That optional flag has a direct use: a team that wants only the remote_class decorator for a non-Paddle workload can skip the deep learning framework install. A team that wants the bundled algorithms needs paddlepaddle, since those algorithms define their networks on it.

A separate Detailed Installation Guide at docs/installation_guide.md is described as continuously updated, so platform-specific problems are answered there and not in the main page. The root of the repository also holds a CMakeLists.txt, a setup.py and a MANIFEST.in.

On licensing, the project is Apache-2.0, and setup.py carries a PaddlePaddle Authors copyright header dated 2018.

xparl runs Python code, so the cluster secret is the whole security model

xparl is the cluster tool, and its security section opens with the key fact: xparl runs Python code on trusted cluster machines. Before startup, every master, worker and client needs a shared, randomly generated XPARL_AUTH_TOKEN of at least 32 bytes.

All ZeroMQ connections require CURVE authentication and encryption. Control metadata uses validated JSON. The master and HTTP services bind to loopback by default, and HTTP monitoring and log routes require credentials.

The stated considerations are blunt. Only trusted users and machines may hold the cluster secret or submit code. Keep all cluster and heartbeat ports on a private network behind a firewall. Whoever holds the token can run arbitrary Python on every node, so treat it like an SSH key for the whole cluster.

The security and upgrade guide at docs/xparl_security.md covers configuration, monitoring access and migration steps.

Upgrading is all-or-nothing and released packages need the fix

Two of the security notes affect operations. Upgrade all nodes and clients together, because older processes cannot use the new protocol. A rolling upgrade where half the cluster still runs the old code leaves those nodes unable to talk to upgraded ones.

The second note says existing published packages require an upgrade to patched source or a release containing the fix. This matters because the tagged releases are old. The listed ones are v2.2 on 2023-03-10, v2.1 on 2022-11-24 and v2.0.0 on 2021-06-03. A pip install parl that resolves to one of those older tags is, by that statement, not the patched version.

The repository itself is not archived and its default branch is develop. The last push was on 2026-09-30, well after the last tag. Anyone who needs the secured xparl has to install from source on develop or wait for a release containing the fix; the documentation shown names no release number for it.

Examples cover value-based, policy-gradient and multi-agent algorithms

The examples directory is the library's real inventory. The README lists QuickStart, DQN, ES, DDPG, A2C, TD3, SAC, QMIX, MADDPG, PPO, CQL and IMPALA. The folder also holds AlphaZero, Baselines, CARLA_SAC, DQN_variant, OAC, Waymax-RL, tutorials and others.

Three entries are competition solutions: the NIPS2018 AI for Prosthetics Challenge, the NIPS2019 Learn to Move Challenge and the NIPS2020 Learning to Run a Power Network Challenge. They are labeled winning solutions, which makes them useful as long worked examples and less useful as small starting points.

Beginners without a reinforcement learning background are pointed to an introductory course with a Bilibili video and code in examples/tutorials. The Getting Started links cover how to solve the cartpole problem, how to set up a cluster with xparl, and how to create customized algorithms.

Waymax-RL adds GPU reinforcement learning for driving simulation

A section marked as the 2025 update covers Waymax-RL: end-to-end GPU reinforcement learning for Waymax autonomous driving simulation. Full documentation lives in waymax_rl/README.md, and the repository root contains a waymax_rl directory, so it is a separate subproject rather than a single example file.

The repository also contains evo_kit, papers, benchmark and test_tipc directories. Documentation is hosted on parl.readthedocs.io in English and Chinese, with a Simplified Chinese README alongside.

A mismatch to watch: the Waymax-RL section describes a 2025 feature, while the newest tag is 2023. Code reached through pip install parl may not contain what the README describes for master.

The documentation hub splits into a tutorial that solves cartpole, an xparl usage page for setting up a cluster and computing in parallel, an advanced tutorial on creating customized algorithms, and an API reference starting at the model page. Read the xparl usage page together with the security guide, because cluster setup and the token requirement are two halves of the same task.

Editorial conclusion

Use PARL when you want reference implementations of DQN, PPO, SAC, IMPALA and similar algorithms, or a decorator-based way to push actors onto a CPU cluster. Do not point xparl at an untrusted network, since it runs submitted Python by design. Before adopting, check that your nodes can be upgraded together, because the security change breaks older processes, and note the last tagged release is v2.2 from 2023-03-10.

Frequently asked questions

Official sources

1. License: Apache-2.0
2. PaddlePaddle/PARL on GitHub
3. Project website
4. README
5. Releases