Dressage: reinforcement learning for agents that use real tools, on top of slime

RL for agents that use real tools

Training an LLM agent with reinforcement learning is harder than training a chat model, because the agent acts: it edits files, runs shell commands, calls retrieval APIs, and the training signal has to survive all of that. Dressage, from Accio Lab, is an agentic reinforcement learning framework built on the slime training library that bridges policy rollouts, sandboxed tool execution and training data conversion through a shared proxy and paddock layer.

Its scope claim covers both paradigms the field currently uses. Whitebox training drives Python tool loops directly, with full gradient flow. Blackbox training treats closed-loop HTTP agents, opencode, openclaw, claude_code and codex are the named ones, as the thing being trained, through a unified interface. The README's news section shows both in production use, with recipes and reproducible scripts rather than only papers.

Quick start: Docker, a checkpoint, one script

The environment ships as a Docker image with GPU passthrough, dated tags such as nightly-dev builds pin it for reproducibility, and the project metadata in pyproject.toml declares the surface: the e2b sandbox client, fastapi and uvicorn for the proxy, ray 2.55.1 for orchestration, transformers 5.12.1, plus optional extras for TransferQueue and Harbor.

Model preparation pulls a Hugging Face checkpoint, initializes the slime submodule, converts the checkpoint into the trainer's torch-dist format with a Megatron tool, and a training example then runs as a single script from examples/scripts. The README documents those steps in full, with all configuration options and troubleshooting delegated to a dedicated quick-start guide, and bundled ALFWorld and HotpotQA whitebox recipes get their own walkthrough.

Two practical notes for anyone planning a run. The scripts directory names the recipe you want, sync, async and partial variants included, so the blackbox Claude Code recipe cited in the news list runs from the same tree as the local async example. And because slime is pinned as a git submodule at version 0.3.0 rather than a dependency, the submodule state is part of your environment, and reproducibility means committing to it like any other pin.

Token-wise evidence: TITO

The technical core is token granularity. Dressage records training evidence per token, token_id, logprob, loss_mask, token_version and token_expert, through a scheme named TITO, token-in-token-out, which encodes only each turn's append delta and splices token IDs incrementally, avoiding the retokenization drift that plagues pipelines rebuilding sequences from text.

The same machinery powers three deeper features. Token-boundary pause and resume through a generation controller lets rollouts stop and continue without losing evidence. Version-aware masking uses token_version to decide which tokens remain valid training signal as context changes. And routing replay records MoE expert routing per token, which is what makes training models with mixture-of-experts layers honest at the token level. For RL practitioners, these are the details that decide whether reported improvements survive reproduction.

Published results, straight from the news list

The README dates and numbers its claims, which makes them easy to restate and easy to check. The SWE-Gym blackbox recipe, an end-to-end Claude Code pipeline with fresh-sandbox evaluation and anti-cheating safeguards, trained Qwen3.5-4B from 32.6 percent to 37.8 percent on SWE-bench Verified, a gain of 5.2 points, with the training script committed. The Claw recipe lifted Qwen3.6-35B-A3B from 41.4 to 55.4 strict Pass^3 on the ClawEval general split, plus 14.0 points.

The systems numbers are stated with the same precision. Rollout step balancing, replacing session-sticky assignment, showed 39.4 to 64.2 percent higher effective tokens per second per GPU and 28.2 to 38.9 percent shorter rollout time in eight-engine experiments. The TransferQueue integration offloaded trajectory storage and cut peak master-node data-plane memory by 67 percent in a two-node experiment and 91 percent in a 32-node capacity analysis. Each claim links its own report, in English and Chinese.

Engineering hygiene around the training loop

The framework's production posture shows in its safety list: atomic trajectory logging, HTTP error redaction, context overflow detection, session artifact archiving and abort safety contracts, so failed rollouts stay diagnosable without corrupting training data. Segment-aware training handles trajectories split by history compaction or tool-schema changes, expanding every segment into a sample while keeping gradient weight honest through prompt-equal denominators.

The slime relationship is stated as integration rather than appropriation: Dressage is built on slime, not maintained as a fork, plugging into upstream through dotted import-path hooks, with the submodule pinned so behavior is reproducible. Additional integrations, Harbor for task execution and evaluation, multi-teacher OPD for training one student against several frozen teachers, and lineage-aware trajectory building for multi-agent runs, extend the same pattern: compose upstream pieces, publish the recipes, keep the numbers reproducible.

Limits and the audience

The requirements are serious by nature. Agentic RL at this scale means GPU clusters, sandboxes, proxy infrastructure and multi-node storage, and the quick start assumes all of it; there are no releases or tags to pin, only commits with a documented Docker tag scheme, and the last push was on 2026-08-27. The Apache-2.0 licence plus a NOTICE file cover the code and its attribution, and thirty-plus test modules back the tree.

The audience is correspondingly narrow: post-training researchers and platform teams who already run slime-shaped pipelines and want agent-shaped rollouts on top, or teams evaluating whether blackbox RL over commercial coding agents is practical for their models. The README equips that audience unusually well, reproducible scripts, dated reports, pinned environments. It is not a managed post-training service, and nothing in it pretends to be.

Against in-house pipelines and hosted post-training

The alternatives bracket the framework. Hand-rolled agentic RL pipelines exist inside every large lab, and they work, but the plumbing Dressage formalizes, token-level evidence, sandbox lifecycle, trajectory storage offloading, engine load balancing, is exactly the part that is expensive to rebuild and hard to trust. Hosted post-training platforms remove the infrastructure work but take the data outside, which is a non-starter for teams whose agent trajectories are their sensitive asset.

Dressage's position is the middle: an open,Apache-licensed framework that assumes your GPUs and your data, formalizes the plumbing, and publishes its recipes with numbers you can chase. Against slime directly, its value is the agent layer; against the hosted route, its value is the boundary. For teams that have decided agentic RL is worth the cluster, reading the two published recipes is the fastest way to judge whether this layer fits.

Editorial conclusion

Dressage fits ML teams already running slime-style RL infrastructure that want to train tool-using agents, whitebox or blackbox, with token-level evidence and reproducible recipes, and who can fund the GPU and sandbox requirements that implies. Skip it if you need a managed post-training service, a single-node setup, or a tagged stable release; the project ships from main with pinned Docker images instead. Verify the claim that matters to you first: read the SWE-Gym recipe report, then run its committed training script at the documented 64K setup, because every number in the README arrives with the script that produced it.

Frequently asked questions

Official sources

1. Accio-Lab/Dressage on GitHub
2. Issues
3. License: Apache-2.0
4. README