SkillOpt: Training Agent Skills as Text Without Touching Model Weights

The Problem: Skills That Do Not Improve Under Feedback

Most agent skills are either hand-crafted, generated one-shot by a strong LLM, or evolved through loosely controlled self-revision. The SkillOpt README argues that none of these behave like a deep-learning optimizer for the skill itself, and none reliably improves over its starting point under feedback. The tool's target user is an engineer or researcher who runs a frozen LLM agent on a recurring task, such as a coding agent inside Claude Code or Codex, and wants the agent's instructions to get better over time. The core idea: treat the skill document as trainable state, analogous to weights, but in text space. The optimizer model edits the skill document, not the model. This matters because many production agents are frozen for cost, safety, or compliance reasons, and you cannot fine-tune them. SkillOpt gives you a way to train the skill that guides the frozen agent.

The Training Loop: Rollout, Reflect, Aggregate, Select, Update, Evaluate

The v0.1.0 release notes describe a full training loop: rollout, reflect, aggregate, select, update, evaluate. In the default paper-style path, a candidate edit is accepted only when it strictly improves a held-out validation score. This is the key mechanism: it is not a free-form self-revision. The optimizer model turns scored rollouts into bounded add, delete, or replace edits on a single skill document. A textual learning-rate budget controls how much the skill can change per update, and a rejected-edit buffer prevents the optimizer from repeating failed edits. There is also an epoch-wise slow or meta update, which the README says makes skill training stable. The entire process adds zero inference-time model calls at deployment. Once training is done, the deployed artifact is a compact best_skill.md file, typically 300 to 2,000 tokens, that runs against the unchanged target model. This is a genuine optimizer analogy: you have epochs, mini-batch size, learning rate, and validation gates, but the parameters are text.

Getting It Running: pip Install and the CLI

The README states that v0.1.0 is on PyPI and you can install with pip install skillopt. The v0.2.0 release adds the SkillOpt-Sleep CLI, shipped as the skillopt-sleep command. The main CLI keeps conservative defaults and does not expose every experiment-harness control as a flag. For configuration, the documentation points to versioned docs in the repository, including docs/index.md for installation, data preparation, training and eval commands, and configuration. The README mentions YAML configs for benchmarks, and a WebUI dashboard exists for monitoring. To add a new backend, you need to create a skillopt/model/<name>_backend.py module for chat backends, or use the shared codex_harness.py for target-only exec backends, then register through common.py, backend_config.py, and skillopt/model/__init__.py. The openai_compatible backend is a shortcut if your provider implements the OpenAI Chat Completions protocol. This is a real setup cost: you are not just running a single command, you may need to write a backend or benchmark adapter.

Validation Gates and the Rejected-Edit Buffer: The Stability Mechanism

The most distinctive design choice is the strict improvement condition. A candidate edit is accepted only when it strictly improves a held-out validation score. This is a hard gate, not a soft preference. It prevents the optimizer from drifting into skill regressions. The rejected-edit buffer is another safeguard: the optimizer can see what edits were rejected, so it does not repeat them. The textual learning-rate budget limits the magnitude of changes per update, which is analogous to a learning rate in gradient descent. This combination is what makes the training stable, according to the README. However, the strict gate has a downside: if your validation metric is noisy or does not reflect real-world performance, the gate may reject good edits or accept bad ones. The README claims best or tied-best results on all 52 evaluated cells across six benchmarks, seven target models, and three harnesses, but those numbers come from the project's own evaluation, and you should treat them as claims, not independent verification.

SkillOpt-Sleep: Offline Self-Evolution for Local Coding Agents

The v0.2.0 headline feature is SkillOpt-Sleep, a nightly offline self-evolution engine. The README describes it as harvest, mine, replay, consolidate, all behind a held-out validation gate. It is aimed at local coding agents like Claude Code, Codex, and Copilot. The idea is that you run it overnight: it reviews past sessions, replays recurring tasks, and consolidates validated skills. This is a different workflow from the main training loop, which requires explicit rollouts and evaluations. SkillOpt-Sleep is a companion, not a replacement. The release notes say it ships as the skillopt-sleep CLI, and it includes experimental multi-objective, replay, and dream-rollout controls. The integration shells for Claude Code, Codex, Copilot, and Devin are in the repository, not in the PyPI wheel, so if you want those, you need to clone the repo. This is a limitation: the pip package does not include everything the repo offers.

Limitations and Failure Modes: When SkillOpt Is the Wrong Tool

SkillOpt is not for everyone. The training loop requires a scoring function for rollouts. If your task does not have a clear, automated success metric, the validation gate cannot work. You would have to hand-score trajectories, which defeats the purpose. The README mentions six built-in benchmarks, but for a custom task you need to write an adapter, a data loader, a scored rollout helper, and a YAML config. That is a significant integration effort. Another limitation: the strict validation gate can be brittle. If the validation set is small or unrepresentative, the optimizer may overfit to it, and the best_skill.md might not transfer to real-world usage. The README claims transfer across model scales and harnesses, but that is a claim about their benchmarks, not a guarantee for your task. Also, the tool adds zero inference-time calls, but it does add training-time compute: you need to run rollouts, which cost API calls or local execution time. If you have a low-volume task, the training overhead may not be worth it.

Alternatives: One-Shot Generation and Self-Revising Agents

The README itself names the alternatives: hand-crafted skills, one-shot generation by a strong LLM, and loosely controlled self-revision. A real alternative is to use a strong LLM to generate a skill prompt in one shot, with no training loop. That is simpler and cheaper, but it does not improve under feedback. Another alternative is to let the agent itself revise its instructions during deployment, which is what the README calls loosely controlled self-revision. That approach can adapt, but it risks instability and does not have a validation gate. SkillOpt's difference is the discipline: it applies optimizer-style updates with a strict improvement condition and a rejected-edit buffer. If you already have a working one-shot prompt, SkillOpt can potentially improve it, but the cost is the training infrastructure. A third alternative is to fine-tune the model weights, but that is exactly what SkillOpt avoids, and it is not always possible for frozen models.

Maintenance, License, and Upgrade Cost

SkillOpt is MIT licensed, so you can use it commercially with few restrictions, but that is not legal advice. The project is actively maintained, with v0.2.0 released in July 2026 and v0.1.0 in June 2026. The README points to a Changelog for released and unreleased changes, which suggests the project expects ongoing evolution. The upgrade cost from v0.1.0 to v0.2.0 includes the SkillOpt-Sleep CLI and integration shells, but those shells are not in the PyPI wheel, so you must pull from the repository. The main CLI keeps conservative defaults, so upgrading should not break existing workflows, but the experimental controls are not exposed as flags, meaning you may need to edit config files to use them. The documentation is versioned, which helps, but you should plan to track the repository for new backend and benchmark integrations. The project is not archived, and the last push is recent, so maintenance appears active, but that is a snapshot, not a promise.

Editorial conclusion

Adopt SkillOpt if you run frozen LLM agents (like Claude Code or Codex) and want to improve their task performance without retraining or adding inference-time calls, and if you can afford the compute and setup for trajectory rollouts and validation. Skip it if you need immediate deployment without a training loop, or if your tasks lack clear scoring signals, since validation-gated updates require measurable outcomes. Before adopting, verify that your target model and harness are supported (the README lists OpenAI, Azure, Claude, Qwen, MiniMax, and exec backends like codex_exec and claude_code_exec), and that the validation gate aligns with your real-world success metric. Check the v0.2.0 release notes for the SkillOpt-Sleep CLI and integration shells, which live in the repo, not the PyPI wheel.

Official sources

1. Official documentation
2. Official README
3. Project repository
4. Release notes