Model or dataset
areal-project/AReaL avatar
areal-project/AReaL

AReaL: An Asynchronous RL System for Agentic LLM Training

The RL Bridge for LLM-based Agent Applications. Made Simple & Flexible.

5,912 stars660 forksPythonApache-2.0

At a glance

What is it?
AReaL is a Python-based, Apache-2.0 licensed reinforcement learning infrastructure for training LLM-based agents. It emphasizes a fully asynchronous paradigm and a microservice architecture, but its complexity and hardware demands mean it is not for every team.
Who is it for?
Adopt AReaL if you are a research group or engineering team with substantial GPU clusters and a need to train or fine-tune large reasoning or agentic models at scale, especially if you value a fully asynchronous paradigm and microservice flexibility. Do not adopt it if you are prototyping on a single machine or need a quick, lightweight RL loop; consider AReaL-lite for that.
Can I use it commercially?
Yes. Apache-2.0 is a permissive licence: you can use, modify and sell software built on it, as long as you keep its copyright and licence notices.
Is it still maintained?
Yes. The repository received new commits within the last day.
What is it written in?
Mainly Python, according to GitHub's language statistics.

Answers come from the project's GitHub data, last synced on November 15, 2026, and from our analysis. They are not legal advice.

From daily snapshots

Star history

  • areal-project/AReaL5,912 ★ · +163
Star history
Repository2026-09-102026-11-15
areal-project/AReaL5.7k5.9k

163 Stars in the last 66 days, measured from the daily GitHub snapshot. Compare with other projects

Editorial analysis

The Problem: RL Training for Agentic Models Is Infrastructure-Heavy

Training a large language model to act as an agent, one that reasons, searches, or uses tools, requires more than a standard supervised fine-tuning loop. The model must generate trajectories, receive rewards based on outcomes, and update its weights iteratively. Doing this at scale with many parallel actors is a distributed systems problem. AReaL directly targets this gap. It is an RL infrastructure that bridges foundation model training with agent-based applications. The README states it was developed by researchers and engineers from Tsinghua IIIS and Ant Group. The intended users are developers and researchers who need to train reasoning and agentic models, not hobbyists running a single GPU. The project claims to be efficient and scalable, but the real value is in providing a structured way to run asynchronous RL where different components do not block each other.

The Asynchronous Paradigm and Microservice Refactor

AReaL's core design is a fully asynchronous RL training paradigm. In synchronous RL, the training step waits for all rollouts to finish, which wastes GPU cycles. Asynchronous training decouples these phases. The v2.0.0 release, dated 2026-07-01, refactors the entire system into a microservice architecture. The repository layout shows separate directories for training, inference, agent, and weight-update services under areal/v2/. This is a major shift from a monolithic design. The benefit is independent scaling: you can run more inference workers without touching the training service. The cost is operational complexity. You now have multiple services to deploy, monitor, and coordinate. The README mentions that v2.0.0 ships the Hermes online RL loop and end-to-end SWE RL examples, which give concrete starting points.

Agentic RL by Replacing base_url: The Key Integration Trick

One of the most practical features for agentic RL is the ability to train black-box agent applications by simply replacing the base_url and api_key. The README gives an example with OpenClaw, where you train your own agent by pointing its API calls at AReaL's RL service. No code changes are needed, and it works with any agentic runtime. This is a significant simplification. Instead of writing custom reward and trajectory logic for every agent framework, you treat AReaL as a drop-in backend. The README's 2026/04/23 news item also announces Scaffoldings integration from NVIDIA TensorRT-LLM, which decouples agent execution, reward calculation, and trajectory acquisition. This suggests a modular approach where you can reuse existing components. For the exact protocol, consult the online_proxy tutorial linked from the README.

Getting Started: Installation and Configuration

The README points to a documentation site for installation and quickstart guides. AReaL-lite, a lightweight version, has a quickstart guide. For the full AReaL, the installation likely requires CUDA-compatible GPUs, and there is an Ascend NPU branch for Huawei hardware. The configuration is YAML-based. Examples include gsm8k_kpop.yaml and gsm8k_icepop.yaml for math tasks. These files set RL hyperparameters and token masking strategies. The KPop feature is configured via rejection_sampling.metric=binary_kl. IcePop uses importance-ratio-based token masking. You would need to clone the repository and follow the documentation to launch training. The presence of example folders like examples/hermes and examples/swe suggests you start by copying one of those and adjusting the YAML. The project is Python-based, and the Apache-2.0 license allows commercial use with attribution.

Token Masking Innovations: KPop and IcePop

AReaL introduces two specific techniques for RL training. KPop, added in June 2026, is a bidirectional binary KL divergence token masking method. It is configured via rejection_sampling.metric=binary_kl. IcePop is an importance-ratio-based token masking approach. The README does not explain the theory behind them. The names suggest a focus on selectively masking tokens during policy updates. For a practitioner, the practical takeaway is that AReaL offers configurable masking strategies that go beyond simple rejection sampling. You can experiment with these settings in the provided GSM8K examples. The existence of these features indicates active research and development, but it also means the codebase is evolving rapidly, and you should verify that your use case maps to the supported metrics.

Scalability Claims and Hardware Realities

The README makes strong claims about scalability, calling it a large-scale asynchronous system with industry-leading speed. It also references AReaL-SEA, a self-evolving data synthesis engine; combined with RL training on AReaL, its 235B MoE model reportedly surpasses GPT 5 on tau^2-bench. These claims come from papers and blog posts linked in the README. You cannot verify them from the repository alone. Training such a model requires an enormous cluster. The project also offers AReaL-lite, which claims to maintain 90% of performance with 80% fewer lines of code and is designed for AI researchers and rapid prototyping.

Maintenance, Community, and License Implications

The repository is active, with the last push on 2026-11-02 and releases as recent as v2.1.0 in August 2026. This suggests ongoing maintenance. The project has a community biweekly meeting, but the README notes that meetings are conducted in Chinese, with English meetings planned for the future. The documentation is available in both English and Chinese. The license is Apache-2.0, which is permissive. You can use, modify, and distribute the code, even commercially, as long as you preserve copyright notices and state changes. There is no copyleft obligation, so you can integrate AReaL into proprietary systems. However, if you use the microservice architecture, you are responsible for maintaining the service orchestration. The v2.0.0 refactor is a major change, and older examples or code from v1.x may not work without migration. Check the release notes for breaking changes before upgrading.

Alternatives: AReaL-lite and General RL Frameworks

The most direct alternative is AReaL-lite, the lightweight sibling. It uses an algorithm-first API design and natively supports fully asynchronous agentic RL. It has 80% fewer lines of code and maintains 90% of AReaL's performance. If you are a researcher focused on algorithm development rather than large-scale deployment, AReaL-lite is the better choice. Another alternative is a general-purpose RL framework like Ray RLlib or a custom loop built on PyTorch. Ray RLlib offers distributed RL but is not specifically designed for LLM agents. The difference is that AReaL is tailored for agentic applications with base_url integration and token masking. General frameworks require you to build the agent-environment interface yourself. The README also announces an integration with Scaffoldings from NVIDIA TensorRT-LLM, which is useful if you are already in the TensorRT-LLM ecosystem. The choice depends on whether you need the specific agentic features or prefer a more generic tool.

Editorial conclusion

Adopt AReaL if you are a research group or engineering team with substantial GPU clusters and a need to train or fine-tune large reasoning or agentic models at scale, especially if you value a fully asynchronous paradigm and microservice flexibility. Do not adopt it if you are prototyping on a single machine or need a quick, lightweight RL loop; consider AReaL-lite for that. Before committing, verify your hardware supports the required CUDA or Ascend NPU environments, confirm that your agent runtime can be integrated via the base_url replacement mechanism, and check the documentation for the specific service orchestration steps in v2.0.0, since the microservice refactor introduces new operational complexity.

Frequently asked questions

What is AReaL used for?

It is reinforcement learning infrastructure designed to bridge foundation model training with agent based applications, built on a fully asynchronous training paradigm and aimed at large scale reasoning and agentic models. It was originally developed by researchers and engineers from Tsinghua IIIS and the AReaL Team at Ant Group.

What are the Python and torch requirements for AReaL?

Python 3.11 or newer and below 3.13, with classifiers listed for 3.11 and 3.12. Torch is asked for at 2.9.1 or newer and below 2.11 on everything except macOS on x86_64, where the requirement is torch below 2.9.1. Transformers is held between 5.0.0 and 5.3.0 inclusive.

How do I use AReaL with an existing agent application?

By replacing the base URL, and the api key where needed, with the reinforcement learning service endpoints. The readme describes a complete example for training an agent that way with no code changes, no complicated dependencies, and compatibility with any agentic runtime, which is how the project frames support for black-box agent applications.

What services does AReaL 2.0 consist of?

Version 2.0 refactored the system into a microservice architecture with independent training, inference, agent and weight-update services. That release also shipped an online reinforcement learning loop example and an end to end software engineering reinforcement learning training example, with a technical report.

Is AReaL production ready?

The packaging metadata sets the development status classifier to 2, pre-alpha, and the comment above the classifier list marks three as alpha, four as beta and five as production and stable. The release history nevertheless shows tagged versions through 2.1.0. The intended audience is listed as developers and science or research, on POSIX and Linux with a CUDA 12 marker.

Official sources

  1. areal-project/AReaL on GitHub
  2. License: Apache-2.0
  3. Project website
  4. README
  5. Releases
Add this badge to your README

If you maintain this project, the badge below links readers to this analysis and shows its maintenance status from the daily GitHub snapshot. Paste the markdown into your README; add ?metric=license or ?metric=stars to the image URL for a different field.

Add this badge to your README

markdown
[![Hysen Labs](https://hysenlabs.com/badge/areal-project-areal.svg)](https://hysenlabs.com/projects/areal-project-areal)

Star history chart for your README

markdown
[![Star history](https://hysenlabs.com/chart/areal-project-areal.svg)](https://hysenlabs.com/projects/areal-project-areal)

Was this useful?