verl-omni separates rollout, reward and actor update for multimodal RL

The gap a text-only RL loop leaves in generative post-training

VeRL-Omni exists because one RL loop cannot cover both text models and generators. The project names four differences rather than one: model structure, I/O patterns, compute characteristics and runtime bottlenecks. A text RL loop optimises against sampled text; a diffusion loop optimises against sampled latents, and a video or omni model has to move pixels, frames and audio through the same step. The scope is three model families. Diffusion generators for image, video and audio are represented by Qwen-Image and Wan2.2. Unified understanding-plus-generation models are represented by BAGEL and HunyuanImage-3.0. Omni-modality models, which handle text, image, audio and video jointly, are represented by Qwen3-Omni. That work used to live inside verl, and it now has its own repository so it can move at its own pace. For a reader, the practical consequence is layout: the Python package sits at the top level next to `docs/`, `docker/`, `scripts/`, `tests/` and an `examples/` tree, and the intended user already owns a diffusion checkpoint and a reward signal and needs the training loop rather than another inference stack.

Rollout correction and async reward move waiting off the critical path

Generation, not the optimizer, is where a diffusion RL step spends its time, and two separate mechanisms target that wait. Rollout correction skips logP recomputation: instead of rebuilding old-policy log probabilities during training, the trainer reuses what the rollout stage already produced. Asynchronous reward computation overlaps the reward phase with the rollout phase, so a reward server scores one batch while the engine is already generating the next. Reward scoring is not welded to one model. Multi-reward serving is listed with HPSv3, GenRM-OCR and UnifiedReward as targets, and an HTTP scorer covers rewards that already sit behind a service endpoint. Add rollout routing, rollout batching and embed caching, and there are four distinct levers on generation cost. Here is the caution. One aggregate throughput number exists for the whole stack, measured on a single reference Qwen-Image FlowGRPO setup, and it does not separate the contribution of each lever. Measure on your own bottleneck before assuming the overlap alone pays for itself.

Two actor backends and three parallelism modes wired through Hydra

Training is decomposed into reward, rollout and actor update, and Hydra composes them, so a run is described by config instead of by branching code. Two actor backends are selectable: VeOmni from ByteDance-Seed and FSDP2. Each combines with USP, TP or DP parallelism for distributed training, and reproducibility is handled by a deterministic RL mode for the end-to-end run. For an operator the failure surface moves rather than disappearing. A wrong parallelism combination now surfaces as a distributed launch failure instead of a model bug, and the reward stage can be a local model, an HTTP service or an async worker without edits to the trainer. One gap is worth naming. The README calls the backends selectable and never says which one a given recipe uses by default, so the backend behind a run has to be read out of that recipe's own config files. Nothing at the repository root answers it for you.

Where the code lives and what the metadata demands first

The README ships no install command. It points at two pages on the documentation site, Installation and Quickstart, and those pages are where a first run is actually specified. What the repository does hand you is packaging metadata, and it is unusually explicit about the floor and the install order.

requires-python = ">=3.11"

Several ceilings exist for unrelated reasons, and each is commented where it is pinned. pyarrow stops below 25.0.0 because that version breaks parquet reads with a column length mismatch, and datasets has its own floor because 2.19 and earlier touch a pyarrow type that newer pyarrow removed.

"pyarrow>=19.0.0,<25.0.0",  # 25.0.0 breaks pd.read_parquet with ArrowInvalid: column length mismatch

Generation is not in the base dependency list at all. The rollout engine sits in an optional extra, and the comment above it fixes the order: gpu stack first, rollout engine second.

# Rollout engine (install in step 2, after gpu step 1).
vllm-omni = [
    "vllm-omni==0.28.0rc1",
    "qwen-vl-utils>=0.0.14",
]

An `audio` extra adds qwen-omni-utils and audioread. A further comment says qwen-tts must be installed apart from the pinned Transformers 5 support revision, because its 0.1.1 source targets a different one. Expect to reconcile that by hand.

The transformers cap exists because of the Qwen3-Omni adapter

The tightest dependency ceiling in the file is deliberate, and the reason is written into the pin as a TODO. transformers stops at 5.14.1 because the Qwen3-Omni adapter and its tests do not yet cope with the FSDP `_no_split_modules` metadata change that Transformers 5.15.0 brings.

"tensordict>=0.8.0,<=0.10.0,!=0.9.0",
"torchdata",
# TODO: Remove this cap after the Qwen3-Omni adapter and tests support the
# FSDP `_no_split_modules` metadata change introduced in Transformers 5.15.0.
"transformers>=5.13.0,<=5.14.1",

Read that comment next to the model's role and it becomes awkward. Qwen3-Omni is the named example of the third family, the model the 2026-08 version note says the multimodal path was stabilised for, and simultaneously the adapter blocking a dependency bump. Upgrading Transformers is therefore gated on adapter work rather than on anything in your own code. Two more pins behave the same way. tensordict is held between 0.8.0 and 0.10.0 with 0.9.0 carved out, and TransferQueue is pinned to one exact version, 0.1.9. Nothing in the packaging file says when any of these ceilings will move, so treat them as live constraints rather than housekeeping.

A dated announcement and a pinned backend disagree on the engine version

Read the news list next to the packaging metadata and the two do not agree about the rollout engine. The 2026-06 entry says the vLLM-Omni rollout backend was upgraded to v0.22 for higher throughput, with the default actor attention backend switched to FA3. The current extra pins a much later build: `vllm-omni==0.28.0rc1`. Both statements can be true, because a dated news entry records a moment and a pin records a later state. The trap is treating the 0.22 figure as current when it describes an engine well behind the one an install resolves to, and treating 0.28.0rc1 as settled when the build itself is a release candidate. Nothing in the repository notes a pin bump between the two. For your own setup, trust the extra over the news list, and expect the generation path to move under you on rc builds. Pin it explicitly in your own environment file before you trust a throughput run.

Eleven trainer directories map the algorithm coverage line by line

The clearest index of scope sits in the `examples/` tree, one directory per algorithm: flowgrpo, flowdppo, mixgrpo, grpoguard, diffusionnft, diffusionopd, dapo, dpo, grpo, gspo and dancegrpo. For Qwen-Image and Qwen-Image-Edit the project table lists FlowGRPO with CPS and SDE variants, Flow-DPPO, MixGRPO, GRPO-Guard and DiffusionNFT, each marked supported. Other models arrive through the news list rather than that table. Wan2.2 is covered for video generation, LTX2.3 for text-to-video with audio, and MiniMax-H3 for T2VA, FL2VA and Ref2VA under both FlowGRPO and DiffusionNFT. Three directories, dapo, dpo and grpo, carry algorithm names the README never explains in this context. One algorithmic detail changes what a training run costs. FlowGRPO with DiNa-LRM scores clean diffusion latents directly and skips VAE decoding, which removes a whole decode step from the loop.

Against flow_grpo and against staying on verl

The nearest comparison in the project's own pages is the diffusers-based `flow_grpo` implementation. On the reference Qwen-Image FlowGRPO setup, VeRL-Omni is reported at roughly 25% higher end-to-end throughput, and the credited causes are vLLM-Omni rollout, the FSDP2 trainer and overlapped reward computation. That figure covers one model, one configuration and one baseline, so it argues that the engine path helps rather than telling you what to plan capacity around. The other alternative is staying on verl, which already covers multimodal and diffusion RL alongside text. Staying means one codebase and one dependency set, at the price of tracking multimodal work inside the same tree. Moving here buys a generation engine built for multimodal output, a second selectable actor backend, and reward serving that can be asynchronous or reached over HTTP. Pick the wrong tool when your reward is a text reward on a language model, when your model already trains fine under verl, or when no cluster of yours can host a vLLM-Omni rollout. Since the extra is optional, an install without it is not the configuration that throughput number describes. The metadata declares Apache-2.0, with the notice and attribution duties that identifier carries and no copyleft terms attached.

Editorial conclusion

Judge it by whether multimodal generation sits inside your training loop. Teams already post-training Qwen-Image, Wan2.2 or Qwen3-Omni against a real reward service have a reason to read the rollout correction, the async reward path and the eleven trainer directories, and the docs site carries the install steps the README omits. Text-only RL work belongs on verl instead, and without a cluster that can host vLLM-Omni it is the wrong shape of tool. Before committing, read two lines of packaging metadata: the transformers ceiling of 5.14.1 that the Qwen3-Omni adapter holds in place, and the rc pin on vllm-omni==0.28.0rc1 in the optional extra.

Frequently asked questions

Official sources

1. License: Apache-2.0
2. Project website
3. README
4. Releases
5. verl-project/verl-omni on GitHub