RSIAgent: recursive self-improvement for agents, no weight updates required

Self-improvement without a training run

RSIAgent, from AetherLabs, addresses a familiar failure: an agent dropped into an unfamiliar environment, a new OS build, a fresh desktop app, wastes its first attempts learning what any user knows. Fine-tuning fixes that slowly and expensively. RSIAgent's README proposes the other route: recursive self-improvement with model parameters fixed for the entire life of the system.

The learning happens in memory instead. Agents explore, verified successes and failures alike get distilled into procedures, scripts and lessons, and that memory persists across tasks. The paper's phrase for the exploration strategy is broad-then-deep: first gather diverse experience on purpose, then investigate hard cases, hidden constraints and boundary conditions. At execution time memory freezes, and the same agent framework runs your actual task.

Three agents, and a verifier that cannot peek

The loop runs on three roles. The Curriculum Agent picks informative exploration tasks from prior outcomes and decides when further practice stops paying. The Actor Agent does the work through executable Python or Bash programs plus visual observations, then distills its own experience into memory. The Verifier Agent checks task requirements against the resulting environment, and the design gives it one property that does the epistemic work: it cannot read the Actor Agent's private reasoning or memory.

That blindness is the point. Because the verifier grades only what actually happened in the environment, its feedback grounds the learning in observed results rather than in the actor's story about them. The README notes that both grounded successes and grounded failures become lessons, and that official benchmark scores are kept outside the learning loop.

Broad first, then deep, then frozen

The implementation exposes the paper's two exploration stages plus test time as three runtime phases. Phase 1, Broad Recursive Self-exploration, has curriculum agents proposing diverse projects, with actor and verifier pairs executing and checking them in parallel from a shared starting memory; consolidation happens after the whole wave. Phase 2, Deep Recursive Self-exploration, is sequential and targeted: attempts expose gaps and fragile successes, each verified experience updates memory before the next attempt.

Phase 3 is test time. Memory is frozen, curriculum and memory updates are disabled, and the actor works your task through the same action-verification loop, followed by sealed official evaluation. Interaction histories reset between independent attempts; only memory carries over. The details, role interfaces, the wave memory barrier, stopping rules, live in docs/ARCHITECTURE.md.

Installing and the first run

Dependencies are pinned in requirements.txt, which notes that a real desktop run also needs the separately installed OSWorld-V2 environment. With the repository checked out:

pip install -r requirements.txt
cp .env.example .env

The second command follows the instruction in .env.example's own first line: copy it and supply your credential. The template shows exactly which values exist:

# Copy to .env and supply your own credential. Never commit .env.
OPENROUTER_API_KEY=

# Paths default to this checkout and its sibling OSWorld-V2.
# Export path overrides in your shell before invoking a runner:
# RSIAGENT_ROOT=/absolute/path/to/RSIAgent
# OSWORLD_ROOT=/absolute/path/to/OSWorld-V2
# RSIAGENT_ENV_FILE=/absolute/private/path/rsi.env

An OpenRouter key is the one required credential; path overrides stay optional and default to this checkout plus a sibling OSWorld-V2. Two runners sit in the root, run_osworld.py and run_ale.py, one per benchmark family.

Reading the numbers with the README's own caveats

The results table reports mean partial-credit scores with and without RSI: 71.97 to 78.98 on OSWorld 2.0 offline over 82 tasks, and 83.75 to 84.82 on Agents' Last Exam Near-term over 67 tasks. The gains are real in the table, and the surrounding notes deserve as much attention as the numbers.

The RSI column uses 41 recorded RSI entries for OSWorld and 19 for ALE, keeping baseline scores for the rest. It includes selected retries and checkpoints with differing budgets, which the README itself states is not an average over matched repeated runs. ALE adds qualified local regrades and protocol variants. This is unusually candid reporting for a results section, and it means the honest read is: exploration helps clearly on OSWorld, modestly on ALE, with aggregation choices that a careful reader should trace through docs/PAPER.md.

Where the framework will fight you

Three costs are structural. First, the desktop path needs OSWorld-V2 installed as a sibling environment, which pulls in virtual machines or containers and real setup time; the docker pin in requirements.txt hints at it. Second, exploration burns tokens: a broad phase of parallel projects plus a deep phase of targeted retries is a lot of model calls before any of it pays off, so budget the OpenRouter account accordingly. Third, this is research code with research documentation: the architecture and paper notes are required reading, and the results caveats above show the evaluation harness has enough moving parts to demand patience.

The flip side of the training-free design is that none of these costs include GPUs or dataset curation. The whole improvement loop is inference you could stop, inspect, or rerun.

Against skill libraries and fine-tuning loops

The neighboring approaches are easy to name. Skill-library agents, of the kind demonstrated in game environments like Minecraft's Voyager line, accumulate executable skills from play; RSIAgent's memory holds procedures, scripts and failure lessons, but wraps them in a curriculum-verifier loop that decides what is worth learning next. Fine-tuning-based self-improvement collects trajectories and trains on them, which changes the weights and needs a training stack; RSIAgent keeps parameters frozen and leaves the knowledge inspectable as memory files.

The practical difference shows up in iteration speed and auditability. A frozen model with an external memory can be diffed, pruned and reused across model upgrades, since the memory is just data. A fine-tuned checkpoint cannot be inspected that way, and it goes stale with the model it was trained into.

Editorial conclusion

RSIAgent fits research teams and agent engineers who want self-improving behavior on OSWorld-style desktop tasks without a training pipeline, and who will read the architecture docs before running. Skip it if you need a turnkey product, if your environment has no OSWorld-V2 harness, or if per-task latency matters more than accumulated competence. Verify one thing first: work through the results notes in docs/PAPER.md so you know exactly which of the reported gains come from matched comparisons and which come from selected retries.

Frequently asked questions

Official sources

1. AetherLabsAI/RSIAgent on GitHub
2. Issues
3. License: Apache-2.0
4. README