UniRL: a single RL post-training loop for diffusion, autoregressive and unified multimodal models
UniRL is a Framework for Unified Multimodal Model Reinforcement Learning
At a glance
- What is it?
- UniRL is Tencent Hunyuan's Python framework that runs one reinforcement learning loop across image, video, LLM, VLM and unified AR+diffusion models. It ships four entrypoints, Hydra example recipes, and three team-proposed algorithms, but its install path is narrow and its dependency pins are strict.
- Who is it for?
- Adopt UniRL if you already run multimodal post-training on SGLang or vLLM and want diffusion, AR and unified models behind one Hydra-driven loop, and if you can pin Python to 3.12 or 3.13 and the listed dependency versions. Do not adopt it if you need a stable released version, a documented rollback path, or an install that does not pull a pinned SGLang diffusion build.
- Can I use it commercially?
- Check first. The repository uses a licence we do not classify automatically, so read its LICENSE file before any commercial use.
- Is it still maintained?
- Yes. The repository last received commits 1 day ago.
- What is it written in?
- Mainly Python, according to GitHub's language statistics.
Answers come from the project's GitHub data, last synced on September 20, 2026, and from our analysis. They are not legal advice.
DEEP OPEN-SOURCE ANALYSIS
What UniRL covers that separate RL codebases do not
Most RL post-training codebases pick one model family. A GRPO implementation for LLMs handles token sequences; a diffusion RL repository handles denoising trajectories; a unified model that generates images through both an autoregressive path and a diffusion decoder usually needs glue code between the two. UniRL's stated goal is to run one loop across all of them: "generate samples, score them, compute advantages, update the policy, and sync weights back to rollout workers."
The README frames model support and algorithm support as two independent dimensions. Any diffusion algorithm runs on a diffusion model, and AR algorithms run on AR models, so the number of usable combinations exceeds the shipped recipes. That is the real argument for the project: not that it has more algorithms than anyone else, but that the same trainer scaffolding, reward service and weight-sync path apply whether the policy is Stable Diffusion 3, Qwen3, WAN 2.2 or HunyuanImage3.
The audience is therefore narrow and specific. It is for teams that already do multimodal post-training and are tired of maintaining one RL harness per modality. It is not for someone who wants to fine-tune a single LLM with GRPO; the framework's own complexity would be overhead.
Entrypoints, trainers and the Ray DevicePool runtime
The architecture is layered. An entrypoint (`train_diffusion`, `train_ar`, `train_pe`, `train_unified_model`, plus `train_agentic`) loads a Hydra example config covering model, algorithm, rollout, reward, placement and sync, then constructs the matching trainer: `DiffusionTrainer`, `ARTrainer`, `PETrainer` or `UnifiedModelTrainer`.
The trainer coordinates the loop across pluggable rollout engines, algorithms, model bundles, reward services, and a shared distributed runtime built from Ray `DevicePool`, FSDP, a Transfer Queue (TQ), and LoRA or full-weight sync. Config selection uses Hydra's `--config-name=<domain>/<example>` form, so the domain directory is part of the config name itself. The `pe/` domain is worth noting: it trains a prompt enhancer, an AR rewriter scored by a diffusion reward, which is a pipeline shape that does not fit either the pure-AR or pure-diffusion buckets.
The agentic path extends the AR entrypoint with multi-turn tool interaction. Each turn is preserved as a `Sample` lineage, terminal answers are scored through a reward service, and training happens at a colocated rollout barrier. `AgenticTrainer` syncs current weights before every rollout, dispatches sibling trajectories concurrently, and waits for complete GRPO groups before scoring. One public workflow is listed: service-scored multi-turn tool use, with the recipe `examples/deep_research/deep_research_search_judge.yaml`.
What the README does not describe is failure recovery. There is no documented checkpoint-resume story for the agentic barrier, and no rollback procedure if a weight sync fails mid-rollout.
Installing UniRL and running a first recipe
The repository ships an INSTALL.md and a requirements.txt whose header gives the exact install sequence. The requirements file states that the core set is aligned with `pip install -e ".[train,infer,eval]"` and includes the pinned SGLang diffusion dependency. The first command installs the core requirements without build isolation:
pip install -r requirements.txt --no-build-isolationAfter that, the package itself is installed with dependencies skipped, because they were already resolved in the previous step. The requirements header gives this exact form:
pip install -e . --no-deps --no-build-isolationThe same header lists flash-attn as a separate no-build-isolation install at a pinned version. The OCR reward's PaddlePaddle runtime is deliberately excluded from requirements.txt and lives in the `eval` extra, so install it only if you run OCR-based rewards:
pip install -e ".[eval]"Python is constrained by pyproject.toml to `>=3.12,<3.14`, with a comment explaining the cap: st_attn and vsa from `sglang[diffusion]` ship cp310 through cp313 wheels only, with no sdist fallback. If your environment is on Python 3.14, the install will not resolve.
Once installed, a run is a Hydra invocation against an example config name. The README names `diffusion/sd3/sd3_sglang_rollout_colocate` as the diffusion example and `ar/qwen_vl_grpo_geo3k_mc_4x8` and `ar/qwen3_drpo_4b_base_dapo_sglang` as AR examples. The examples directory also contains `run_experiment_single_node.sh` and `run_experiment_multinode.sh`, which the README points to through `examples/README.md` for the full launch guide. Expect the config name to select the domain, and expect the YAML to carry the placement and sync settings, not the command line.
Dependency pins and the torch-not-in-core decision
The pyproject.toml comments are unusually explicit about why each pin exists, and they describe a real constraint rather than a preference. Torch is intentionally absent from the core dependency list. The comment states that torch enters through exactly one engine extra, which is what allows per-engine CUDA stacks. In practice that means the core install is engine-agnostic, and the engine you choose decides which torch build you get.
Other pins are coupled. `transformers` is held to `>=5.6,<5.7`, and `peft` has a floor of `>=0.20` in pyproject.toml because, per the comment, older peft imports transformers cache symbols such as `HybridCache` that were removed in 5.x and fail at import against that transformers pin. `diffusers` is floored at `>=0.38.0` with a note that LTX-2 needs 0.38 for new forward arguments and connector padding behavior, while 0.37 was already required for Qwen-Image RoPE text-length-from-mask. There is also a `cosmos3` extra that raises diffusers to `>=0.39`, so that extra and the base floor can disagree depending on what you install.
Notice that setup.py and pyproject.toml are not identical. setup.py lists `torch>=2.1` and `sglang[diffusion]==0.5.12.post1` directly, with `peft>=0.14.0` and `diffusers>=0.37.0`, while pyproject.toml uses the tighter `peft>=0.20` and `diffusers>=0.38.0` and omits torch. Two install paths with different resolution floors is a maintenance hazard, and the requirements.txt header is the one the repository tells you to follow.
Where UniRL is the wrong tool
The install is the first hard boundary. `sglang[diffusion]==0.5.12.post1` is an exact pin, not a range, and the Python cap exists because of its wheel coverage. If your cluster image ships a different SGLang build, you are either rebuilding the environment or patching the pin and accepting whatever breaks. There is no documented path for running UniRL against a different rollout engine version.
The second boundary is scope. UniRL assumes a reward service and a rollout engine are part of the loop. If your task has a cheap programmatic reward and you only need policy-gradient updates on a text model, the Ray DevicePool, FSDP, Transfer Queue and weight-sync machinery are cost without benefit. A single-process GRPO implementation would be easier to debug.
The third is maturity signals. There are no retrieved releases, and setup.py declares version 0.1.0 while pyproject.toml takes the version dynamically. The README describes three algorithms released in June 2026 with arXiv identifiers, but the documentation does not state a compatibility guarantee between a recipe and a later commit. The README is also silent on rollback: there is no described procedure for reverting a weight sync or resuming an interrupted agentic run. For a framework whose loop includes a distributed weight transfer step, that is a gap worth weighing before you put it in a production training pipeline.
How UniRL differs from verl and other RL post-training stacks
The closest comparison is verl, the widely used RL post-training library for LLMs. Both use a generate-score-update-sync loop and both integrate rollout engines, but they diverge on the model dimension. verl's design centers on autoregressive language models and their token-level algorithms. UniRL adds diffusion and unified models as first-class domains: `DiffusionTrainer`, `PETrainer` and `UnifiedModelTrainer` sit alongside `ARTrainer`, and the model table spans image diffusion, video diffusion, VLM, omni-modality and unified AR+diffusion models such as HunyuanImage3 and Bagel.
The algorithmic emphasis differs too. UniRL highlights three team-proposed methods with their own folders and tutorials: Flow-DPPO for flow matching models, DRPO for token-level LLM RL, and CPPO for position-weighted trust regions. Standard references such as GRPO, DiffusionNFT, DanceGRPO and MixGRPO are also wired in under `unirl/algorithms/`. If you are choosing between the two, the question is not which loop is better but whether your policy produces tokens or denoising steps. For a pure LLM workload, verl's narrower focus is an advantage. For a workload that spans a VLM and a video diffusion model under one reward service, UniRL is the one that already has the trainer split.
Licence and upgrade cost
The repository's LICENSE file is referenced by pyproject.toml as `license = { file = "LICENSE" }`, and the README badge reads Apache-2.0. The repository metadata reports the licence as NOASSERTION, so the two signals do not match, and a team that needs licence certainty should read LICENSE directly rather than trust either label. There is a further wrinkle: setup.py declares package data for `unirl.models.janus_pro.vendor` including `LICENSE-CODE` and `VENDOR_COMMIT.txt`, and pyproject.toml notes that a Boogu-Image DiT is vendored under `unirl/models/boogu_image/vendor/`. Vendored model code can carry terms that differ from the project's own licence, so check those files if you redistribute. This is not legal advice.
Upgrade cost is driven by the exact pins. Moving SGLang, transformers or diffusers forward means re-validating every recipe you depend on, because the comments tie specific model support to specific versions: LTX-2 to diffusers 0.38, Qwen-Image RoPE to 0.37, peft to the transformers 5.x symbol removal. The requirements.txt header itself documents that the optional Geneval and OpenMMLab stack (mmcv, mmdet) is excluded and must be installed separately following `docs/geneval_mmcv_setup.md`, and that PaddlePaddle stays out of the core set because it is a roughly 180 MB second deep-learning framework needed only by `unirl.reward.local.ocr`. Budget for a per-upgrade re-validation pass, not a version bump.
Editorial conclusion
Adopt UniRL if you already run multimodal post-training on SGLang or vLLM and want diffusion, AR and unified models behind one Hydra-driven loop, and if you can pin Python to 3.12 or 3.13 and the listed dependency versions. Do not adopt it if you need a stable released version, a documented rollback path, or an install that does not pull a pinned SGLang diffusion build. Before committing, verify that your GPU stack matches the sglang[diffusion]==0.5.12.post1 pin, that your model appears in the support table, and that the example recipe closest to your workload actually loads under your placement settings.
Frequently asked questions
What is UniRL from Tencent Hunyuan?
UniRL is a Python framework for reinforcement learning post-training of unified multimodal models. It applies one loop (generate samples, score them, compute advantages, update the policy, sync weights to rollout workers) across diffusion, autoregressive, prompt-enhancer and unified AR plus diffusion model families.
Which Python version and dependencies does UniRL require?
pyproject.toml requires Python >=3.12,<3.14, with the cap explained by the cp310 to cp313 wheel coverage of st_attn and vsa from sglang[diffusion]. The core requirements also pin sglang[diffusion]==0.5.12.post1 and transformers>=5.6,<5.7.
Which models can I train with UniRL?
The README's support table lists image diffusion models (Stable Diffusion 3 and 3.5, Qwen-Image, FLUX.2-Klein, Z-Image), video diffusion models (WAN 2.1 and 2.2, HunyuanVideo 1.0 and 1.5, LTX-Video-2 and 2.3), AR models (Qwen-VL, Qwen3, Qwen3-Omni Thinker), the Prompt-Enhancer, and unified models (HunyuanImage3, Bagel, SenseNova-U1.5).
How do I start a UniRL training run?
Runs are Hydra invocations that select an example config by domain, for example diffusion/sd3/sd3_sglang_rollout_colocate or ar/qwen_vl_grpo_geo3k_mc_4x8. The examples directory also ships run_experiment_single_node.sh and run_experiment_multinode.sh, and examples/README.md holds the launch guide.
Does UniRL support multi-turn tool use?
Yes, through the agentic entrypoint train_agentic, which extends the AR path with multi-turn tool interaction, preserves each turn as a Sample lineage and scores terminal answers through a reward service. The README lists one public workflow, service-scored multi-turn tool use, with the recipe examples/deep_research/deep_research_search_judge.yaml.
Community notes