MARTI: Multi-Agent RL Training and Inference for LLM Workflows

The problem MARTI targets: training the graph, not just the model

Most RLHF tooling optimizes one policy against one reward signal. A multi-agent workflow is a different object. Several models, possibly different checkpoints, exchange messages inside a fixed topology such as a debate, a chain-of-agents or a mixture-of-agents, and the reward arrives once at the end of the episode. The training question is which agent contributed to that reward and how much credit each one deserves. MARTI is built for that question. The README describes the design as centralized multi-agent interaction with distributed policy training, meaning all agent interactions and reward allocation happen in one place while policy updates are distributed across the individual agents. The audience is narrow by construction: research groups and engineering teams who already run PPO, GRPO, REINFORCE++ or TTRL and now want the rollout to span several models rather than one. If your pipeline is a single model answering a single prompt, MARTI adds an orchestration layer you will not use.

Three modules and a split between interaction and training

The architecture in the README is described as three core modules: Multi-Agent World, Centralized Rewarding, and Single Agent Trainer. Multi-Agent World runs the agent graph and produces interaction traces. Centralized Rewarding scores those traces and handles credit assignment and reward shaping, which the feature list names explicitly. Single Agent Trainer consumes the per-agent slices and updates each policy. The split matters because it lets heterogeneous models coexist in one graph: the README states that different models can be trained simultaneously with independent roles, training strategies and dynamic sample filtering per agent, with Qwen3-8B and AreaL-boba-2-8B given as the example pair. The graph topologies listed are debate, chain-of-agents and mixture-of-agents, so the workflow shape is configuration rather than code you write from scratch. On top of this, MARTI-v2 adds multi-agent tree search with adaptive node expansion and refinement, aimed at code generation, plus GSPO loss for sequence-level optimization, TIS correction for the sampling mismatch introduced by vLLM, dynamic data filtering and an overlong buffer for token penalty. The README claims support for sequences up to 32K tokens. That number is the project's own claim; I have not reproduced it.

What the installation and quick start actually give you

The installation section in the supplied README is a single opening code fence with no commands inside it. That is a real gap, not a formatting artifact of this review: the material I have ends at the fence, so I cannot tell you what pip install or conda invocation the project expects. What I can confirm is the surrounding scaffolding. The quick start is organized around two tracks. The first is MARS squared, the multi-agent tree search RL path, with separate subsections for single-agent MCTS training and multi-agent MCTS training. The second is customised async step and workflow, with single-agent training and multi-agent training subsections, and the release notes describe async tool use in agentic RL and async workflow for multi-agent RL as additions from August 2025. The framework is stated to build on OpenRLHF and verl, and to support the vLLM v1 Engine and a Hybrid Engine. Practical consequence: your environment is not just MARTI. You are installing an RL trainer, a serving engine and an inference framework together, and version conflicts between those three are the most likely first failure. Check the repository's install script before trusting any tutorial, because this README does not carry the commands.

Where MARTI is the wrong tool

Two limitations are visible from the material alone. First, there are no retrieved releases. The last push is dated 2026-08-20 and the news entries run from May 2025 through February 2026, but nothing in the supplied data shows a tagged version. That means adopting MARTI means tracking the main branch, and a research framework on main can change its training loop between your runs. For a paper reproduction that is tolerable. For a production fine-tuning pipeline it is a liability you have to budget for. Second, the third-party integrations with AutoGen and CAMEL are labelled experimental in the README itself. If your existing multi-agent code is written against either of those, expect to adapt it rather than plug it in. There is also a scope limit worth stating plainly: MARTI is a training framework. It does not give you serving, evaluation harnesses or dataset curation, and the benchmark section in the README covers AIME-style reasoning and code generation, not general instruction following. If your goal is a chat model that follows formatting instructions, this is the wrong layer to adopt.

How it differs from plain OpenRLHF or verl

MARTI is not a replacement for OpenRLHF or verl. The README says the framework builds on them, and that MARTI-v2 was adapted to the latest OpenRLHF infrastructure. The difference in approach is where the episode lives. With OpenRLHF or verl used directly, you write a rollout function that samples from one policy, scores it, and returns a trajectory; multi-agent structure, if you need it, is something you implement inside that function. MARTI moves the graph into the framework: the Multi-Agent World owns the interaction, Centralized Rewarding owns the credit assignment, and the Single Agent Trainer receives already-sliced per-agent data. The trade is control for structure. You give up the freedom to write an arbitrary rollout loop and you get debate, chain-of-agents and mixture-of-agents topologies plus reward shaping strategies without building them. If you have already written your own multi-agent rollout on top of verl and it works, MARTI's value proposition is weaker for you. If you have not, and credit assignment across agents is the blocker, the framework is doing work you would otherwise do yourself.

Maintenance cost and the MIT licence

Maintenance cost here is dominated by the dependency chain rather than by MARTI's own code. The stack named in the README includes OpenRLHF, verl, vLLM, AutoGen and CAMEL, plus the model checkpoints you train. Each of those moves independently. The TIS correction exists specifically because vLLM sampling introduces a distribution shift during rollout, which tells you the project is coupled tightly enough to vLLM's sampling behaviour that a vLLM upgrade can affect training correctness. Budget for pinning. On licensing, MARTI is MIT, which is permissive and places few obligations on how you redistribute modified source. That applies to MARTI. It does not automatically extend to the dependencies, and OpenRLHF, verl, vLLM, AutoGen and CAMEL each carry their own terms, some of which differ from MIT. Check each one against your distribution model. This is not legal advice; if you are shipping a product built on this stack, have counsel review the dependency licences rather than assuming MIT at the top covers everything underneath.

Who should adopt MARTI, and what to verify first

The fit is a team that already runs OpenRLHF or verl, has a multi-agent workflow where different models play different roles, and needs the reward to be distributed across those roles rather than assigned to one policy. The MARTI-v2 tree search path targets code generation specifically, so if that is your task and you want multi-turn exploration with adaptive node expansion, the framework is aimed at you. The misfit is anyone who wants a stable, versioned dependency, a single-agent RLHF loop, or an out-of-the-box evaluation and serving stack. Three things to verify before you spend GPU time. Confirm the installation commands exist in the repository, since the README section is empty in the material available to me. Confirm whether AutoGen and CAMEL support has moved past experimental, because the README still labels it that way. And identify the exact OpenRLHF commit the MARTI-v2 training scripts expect, since the README says the framework was adapted to the latest OpenRLHF infrastructure and that phrase will not stay true indefinitely.

Editorial conclusion

Adopt MARTI if you are already running OpenRLHF or verl and need graph-structured multi-agent rollouts with credit assignment across agents, and if you can pin a vLLM build and a GPU cluster for multi-turn exploration. Do not adopt it if you want a single-agent RLHF loop, a managed training service, or anything with a stable tagged release. Before committing, verify three things in the repository: whether the installation section resolves to real commands rather than a truncated code block, whether the AutoGen and CAMEL integrations are still marked experimental, and which OpenRLHF commit the MARTI-v2 scripts actually pin.

Official sources

1. Issues
2. License: MIT
3. README
4. TsinghuaC3I/MARTI on GitHub