SkillRL, and what a hierarchical skill bank changes for RL-trained agents

Raw trajectories become a two-level SKILLBANK

SkillRL is a framework for LLM agents that learn reusable behavioural patterns instead of replaying their own history verbatim. The premise is that a memory system which stores full trajectories carries a lot of noise, and that a short strategy or a lesson distilled from a failure is what actually helps the next attempt. The target user is a research team training agents on interactive environments with reinforcement learning, and the reference environments are ALFWorld, WebShop and Search.

The skill library is hierarchical. General Skills hold strategy that applies across tasks, and Task-Specific Skills hold heuristics for a category of task. The distillation step has two halves: successful trajectories are turned into strategic patterns, and failed ones into concise lessons. That second half matters more than it sounds, because the recursive mechanism later reads validation failures to decide when the library needs to change.

One number in the overview is worth quoting with attribution rather than repeating as fact. The project claims 10 to 20 percent token compression relative to raw trajectory storage while, in its words, enhancing reasoning utility. There is no benchmark table in the repository to check that against, so treat it as a claim from the authors.

Template mode and embedding mode decide what enters the prompt

Everything the agent reads from memory passes through a retrieval step, and SkillRL offers two implementations of it.

Template mode uses keyword matching to detect the task category and then injects all skills registered for that category. No embedding model is needed, which keeps the dependency surface small and the behaviour predictable. This is the mode used by the shell scripts that come with the repository.

Embedding mode uses Qwen3-Embedding-0.6B to rank skills by semantic similarity to the task description. General skills and task-specific skills are searched across categories, and only the top-k results are injected. Skill embeddings are computed once at startup, so the cost is a fixed startup hit rather than a per-step search.

The two modes are selected through the same configuration namespace, `env.skills_only_memory.*`, which is Hydra and OmegaConf based. The documented parameters include `skills_json_path`, which is required and points at a skills JSON file, `retrieval_mode` with values `template` or `embedding`, `embedding_model_path` defaulting to `Qwen/Qwen3-Embedding-0.6B`, and `top_k` for the number of skills injected. The trade-off is straightforward: template mode gives you category-level recall with no model to serve, embedding mode gives you semantic recall but adds a model dependency and a ranking step whose output you cannot inspect without reading the JSON yourself.

The library rewrites itself while the policy is still training

Recursive Skill Evolution is the part that distinguishes this from a static memory bank. Instead of freezing the skills before the RL loop starts, the mechanism watches validation failures and updates the library as the policy changes. A skill that was accurate early in training can become wrong later, and a category that is absent from the library entirely can only be learned by something writing into it.

Three configuration keys control this. `enable_dynamic_update` turns the behaviour on. `update_threshold` sets how often a pattern has to recur before a change is triggered, and the shipped script uses 0.4. `max_new_skills` caps how many skills a single update may add, and the shipped script uses 3. That cap is a sensible guard against a library that grows without limit and eats the context window it was meant to protect.

For a user, this changes what a training run means. The final checkpoint is not the only artefact of interest, because the skills JSON is also a product of the run and can be edited, curated, or reused with a different policy. The repository separates the two by making `skills_json_path` a configuration value rather than a hardcoded file. The README does not describe a rollback path for the library, so if a bad update lands mid-run there is no documented way to rewind it.

Installing means installing verl, vLLM and three environment packages

The setup path is long and environment specific. The base install pulls the Python dependencies, pins a specific vLLM and flash-attn build, installs the project in editable mode and adds the OpenAI client:

git clone https://github.com/aiming-lab/SkillRL.git
cd SkillRL
pip install -r requirements.txt
pip install vllm==0.11.0
pip install flash-attn==2.7.4.post1 --no-build-isolation --no-cache-dir
pip install -e .
pip install openai

Then each environment has its own package. ALFWorld needs alfworld plus gymnasium==0.29.1 and stable-baselines3==2.6.0, and the game assets and pre-trained MaskRCNN detector are fetched with a separate command:

pip install alfworld
pip install gymnasium==0.29.1
pip install stable-baselines3==2.6.0
alfworld-download -f

WebShop and Search each have their own setup script or local package, under agent_system/environments/env_package/. Inference calls go to an Azure OpenAI endpoint, configured with two environment variables:

export AZURE_OPENAI_API_KEY="..."
export AZURE_OPENAI_ENDPOINT=""

Two things follow from that list. A machine running this needs a GPU, a working CUDA and flash-attn toolchain, and network access to an Azure OpenAI deployment, so a laptop-only evaluation is not on the table. And the environment setup is per benchmark, so switching tasks means running a different setup path rather than a flag.

Memory generation, then SFT, then RL with skills

The training pipeline has three stages, and the order matters because the skill library is built from data the model itself produced.

The first stage is memory data generation. The base model runs against the environment to produce trajectories that become the agent's initial experience. The prompt that drives this generation is a file in the tree, at memory_data/prompt/prompt.txt, and the project asks you to read it rather than to guess what the agent was told.

The second stage is supervised fine-tuning, used to give the model basic task ability and instruction-following alignment before any RL happens. LLaMA-Factory is the framework used for this stage, and the generated SFT dataset was published separately on Hugging Face as SkillRL-SFT-Data, with the generation code under examples/sft_data_generation/.

The third stage is RL with the skill bank, run through the GRPO trainer scripts shipped for each environment:

export MODEL_PATH=YOUR_SFT_CKPT
bash examples/grpo_trainer/run_alfworld_skills.sh
bash examples/grpo_trainer/run_webshop_skills.sh
bash examples/grpo_trainer/run_search_skills.sh

Those scripts set the skill flags directly, including the skills JSON path, `top_k=6`, dynamic updates, the 0.4 threshold and the cap of 3 new skills. The matching checkpoints are published on Hugging Face as SFT and RL pairs for each of the three tasks, which is the cheapest way to evaluate the idea without running a training job.

requirements.txt and setup.py disagree about the same packages

The repository ships two dependency manifests and they are not in sync. requirements.txt pins transformers==4.51.1 and tensordict<=0.6.2, while setup.py allows transformers<=4.57.3 and requires tensordict>=0.8.0,<=0.10.0 with 0.9.0 excluded. The vLLM constraint in setup.py is vllm>=0.8.5,<=0.11.0, which happens to include the 0.11.0 that the installation steps pin, and the tensordict ranges do not overlap either. Whichever manifest you install from, you will not get the set the other one describes. This is the first thing to resolve before any debugging, because a transformers or tensordict mismatch surfaces as an obscure runtime error rather than a clear message.

The packaging metadata raises a similar question. pyproject.toml names the project `verl` and describes it as "verl: Volcano Engine Reinforcement Learning for LLM", the repository root contains a verl/ directory, and setup.py is headed with a Bytedance copyright and the Apache License 2.0. The repository is tagged MIT, the licence is Apache-2.0 in that file header, and pyproject.toml points licence at a file named LICENSE. There is a Notice.txt in the root as well. If you intend to ship a derivative, read those files rather than the licence tag, and settle attribution obligations with whoever maintains the project.

What changes against a plain verl training run

The examples directory is the clearest sign of the base this project sits on. It carries the full verl trainer zoo: grpo_trainer, ppo_trainer, dapo_trainer, gspo_trainer, rloo_trainer, gigpo_trainer, gigpo_dynamic_trainer, ray, slurm, split_placement, env_server, generation and data_preprocess, plus a separate gigpo/ directory at the repository root. The SkillRL-specific addition is the skill memory layer and the pipeline around it: skill_generation/ holds the distillation code, memory_data/ holds the generated experience and the prompt, and agent_system/environments/env_package/ holds the three benchmark environments.

So the difference is not a new reinforcement learning algorithm. The optimisers, the rollout engines and the trainer structure are verl's, and the same repository can be used without the skill layer at all by leaving `env.use_skills_only_memory` off. What SkillRL changes is the representation of the agent's memory and the point in training at which that representation is allowed to change. Against the other common alternative, which is keeping raw trajectories in the prompt and letting the model reread them, the trade is fewer tokens for a lossy summary, plus a library that can drift away from what the policy actually does.

That last point is the honest limit of the approach. A distilled skill is a claim about behaviour written in words, and nothing in the framework verifies that the word still matches the policy except the failure counter that triggers updates.

Editorial conclusion

SkillRL fits research teams that already run verl-style RL on agent environments and want to measure what a skill memory buys them in context length and transfer. It does not fit production agent stacks that need a stable, small dependency set, a single supported model provider, or a package that installs cleanly. Before adopting it, resolve the dependency conflict between requirements.txt and setup.py yourself, since one file pins transformers==4.51.1 and tensordict<=0.6.2 while the other allows transformers<=4.57.3 and requires tensordict>=0.8.0, and check the licence file in the repository root rather than trusting the MIT tag alone, because setup.py carries an Apache-2.0 header. The paper is on arXiv at 2602.08234 and the repository has no tagged releases, so the last push on 2026-09-25 is the only anchor you have.

Frequently asked questions

Official sources

1. aiming-lab/SkillRL on GitHub
2. Issues
3. License: MIT
4. Project website
5. README