Relax: an asynchronous reinforcement learning engine for omni-modal post-training

What the acronym stands for, and who built it

Relax stands for Reinforcement Engine Leveraging Agentic X-modality, and it is a high-performance reinforcement learning post-training framework open sourced by the Xiaohongshu AI Infra Team for multimodal large language models.

That provenance matters more than usual here. This is not a research prototype written to publish a paper; it is the infrastructure a large platform team built to run multimodal post-training in production and then released. The project went open source on 2026-04-15, and the news table in the readme is a changelog rather than a list of announcements, with entries dated 2026-05-11 for Qwen3.6 series support, 2026-05-26 for the new Hybrid execution mode, 2026-08-18 for LoRA RL with MoE models, 2026-08-19 for multi-agent training, and 2026-08-20 for M2PO, RLOO and two REINFORCE++ variants with runnable recipes and an algorithm reference.

The claim the project makes for itself is that it is one of the few systems capable of end-to-end Omni model post-training, naming Qwen3-Omni, and that it supports end-to-end multimodal RL training from text to images, videos and audio through the TransferQueue data transfer system.

The stack underneath is specific. Ray Serve provides the service layer, Megatron-LM is the training backend, SGLang is the inference engine, and DCS handles NCCL-broadcast weight sync. The licence is Apache 2.0, the language is Python 3.12, there is a paper on arXiv, and the documentation site is at redai-studio.github.io/Relax with a Chinese readme alongside the English one.

Six layers, and every role is a Ray Serve deployment

The architecture is described as a six-layer service-oriented architecture in which every role is deployed as an independent Ray Serve deployment, cleanly separating orchestration, components, engines, backends and distributed capabilities.

The layers are laid out in a table in the readme.

Entrypoints are the train.py script, handling signal handling, CLI parsing, Ray cluster connection and Controller launch. Orchestration holds the Controller, which owns the training loop and global restart, the Service, which manages placement groups and lifecycle, and the Registry, which maps roles and algorithms. Components are the Ray Serve deployments themselves: Actor, Rollout, Critic, ActorFwd, Advantages and GenRM. Engine covers the SGLang rollout engine, pluggable reward functions, the request router and data filters. Backends are Megatron-LM for training with its TP, PP, CP and EP parallelism modes, and SGLang for inference. Distributed covers the Ray actor groups such as RolloutManager and GenRM.

The practical consequence of this structure is that a component failure is a service failure, not a training run failure. The readme names the service level capabilities that follow from it: native service-level elastic scheduling and fault recovery, and a HealthManager for auto-recovery.

The GenRM component is worth singling out. It is the LLM-as-judge reward model, the reward function that scores outputs by asking another model, and it is deployed as its own service with its own actor group rather than being a function inside the trainer. That is a deliberate design choice and it has a cost: judging becomes a network service that has to be scaled and that adds latency to every rollout step.

Asynchrony through TransferQueue, and the Hybrid compromise

The full asynchronous mode is the headline: Rollout, Actor, ActorFwd, Reference and Advantages run on independent GPU clusters with streaming data exchange and configurable staleness. The point of the data transfer system is complete decoupling of training and inference, so a slow rollout does not idle the trainer and a slow trainer does not idle the inference engines.

The trade is staleness. Asynchronous reinforcement learning accepts that the policy generating rollouts is not the policy being updated, and the readme calls the amount configurable. That is the correct framing, because in a synchronous system the two clocks are locked together and the cost of a straggler rollout is paid by everyone.

Then there is Hybrid mode, which is a middle path and arguably the more interesting piece of engineering. It uses separate Actor and Rollout placement groups with TransferQueue streaming, while ref, actor_fwd and advantages run in-process on the actor, and it pairs the --balance-data option with sub-batched forward to minimize GPU waste. The reasoning is straightforward: the reference model, the forward pass used for some of the reference-free algorithms, and the advantage computation are cheap relative to the rollout, so shipping them over the network is overhead with no benefit. Putting them in-process removes that transfer and the associated memory pressure, at the cost of the actor being a bigger job.

Elastic rollout scaling is the other feature that stands apart. It grows and shrinks inference engines mid-training through an HTTP REST API, with same-cluster federation in ray_native mode and cross-cluster federation in external mode. Scaling inference capacity mid-run, rather than sizing it for the peak and paying for it throughout, is the kind of thing that only matters once a cluster is large enough that the waste is visible.

The algorithm suite, and the ops layer around it

The algorithm list is long and named in the highlights: PPO, GRPO, M2PO, RLOO, REINFORCE++ variants, GSPO, SAPO, CISPO and on-policy distillation, with pluggable rewards and a built-in GenRM mode.

That spread is worth reading as a statement about where the field is. The GRPO family, GSPO and CISPO are the current line of work for reasoning model post-training, and the presence of on-policy distillation in the same list says the team is treating distillation as a reinforcement learning problem rather than a separate track. The reference to lcobucci and the rest of the ecosystem is not there, so nothing beyond the algorithm names is claimed.

Agentic RL is the other headline. The readme names multi-turn interaction, loss masking, flexible termination, and VLM multimodal context carry-over for closed-loop execute, observe, decide training. The 2026-08-19 news entry adds multi-agent training with multi-context trajectory export, and the examples directory contains a mini_swe_agent recipe, a search_r1 recipe, a deepeyes set of agentic examples, and a nemo_gym_agentic example, which is a good sample of what the framework is being aimed at.

The operational layer is described as production ready: a HealthManager for auto-recovery, a centralized Metrics Service with WandB, TensorBoard and ClearML backends, and Apprise for real-time notifications.

The dependency list in the requirements file is the honest measure of the scope, and it is a wide one. It includes ray with the serve extra, the ray default extra and sglang-router at 0.2.3 or newer, alongside fastapi pinned at 0.139.1, uvicorn, apprise, clearml pinned at 2.1.3, numpy below 2, wandb, tensorboard, diffusers, transformers, datasets, huggingface_hub pinned at 1.7.2, and av pinned at 17.0.0. There are also the debugging tools, memray, py-spy, pystack, gpustat, ipdb and debugpy. Two entries carry explanatory comments: peft is upper-bounded on purpose because the LoRA backend imports a PEFT internal with no stability guarantee, and boto3 is there for an s3 model downloader.

Packaging, documentation and the state of the project

The packaging is conventional and the version is derived from git.

The build backend is setuptools with versioneer, and the version file is written to relax/_version.py with a v tag prefix and pep440 style. The package name is relax, the author is the Relax Contributors with the Xiaohongshu AI Platform address, and the package is classified as Development Status 4, Beta, with an Intended Audience of Developers and Python 3.10 and 3.11 classifiers, while the readme badge says Python 3.12 and the ruff configuration targets py310.

The optional dependency groups are named after the features they enable: test, gpu, math, vllm, sglang, mcore and transferqueue. That naming is a useful map of the architecture, since each group corresponds to one of the backends or engines in the layer table.

The documentation is a Vitepress site built from a docs directory, with Mermaid diagrams, Swagger UI for the OpenAPI spec, a mathjax3 extension for the equations, and a script that generates the OpenAPI documentation before the build. The package.json for the docs site also holds a script that fixes chunk names after the build, which is the kind of detail that shows the site is deployed somewhere with strict asset naming rules.

The examples directory is where the framework is demonstrated, and it is broad. There are directories for algorithms, code generation, CoT compression, the deepeyes agentic examples, diffusion, elastic rollout, a gagar example, generating a reward model, mini_swe_agent, models, nemo_gym_agentic, on-policy distillation, search_r1, and a sequence classification SFT example.

Only one release has been published, v0.1.0 from 2026-08-12, and the last push was on 2026-10-20, so the code is moving faster than the tags. The repository also carries a CITATION.cff, an AGENTS.md and a CLAUDE.md, a pre-commit configuration with hooks, a gitleaks configuration, and a Codex directory, which is the tooling surface of a team that works this way professionally.

Editorial conclusion

Adopt Relax if you are post-training a multimodal or omni-modal model and you have the GPU cluster, the Ray deployment habits and the patience that a service-oriented RL stack demands, because the decoupling it offers is the point. Do not adopt it for a single-GPU experiment, since the architecture assumes independent Rollout, Actor, ActorFwd, Reference and Advantages deployments on separate GPU clusters, and treat the version line as young, with one tagged release so far. Before you commit, read the algorithm reference and the runnable recipes in the examples directory, check the hardware and Ray assumptions in the documentation, and confirm you can live with a Beta classification on the package.

Frequently asked questions

Official sources

1. License: Apache-2.0
2. Project website
3. README
4. redai-studio/Relax on GitHub
5. Releases