SkillZero: in-context RL that trains skills into the weights

The problem SkillZero targets: skills that live in the prompt instead of the policy

Agentic systems are commonly given skills as text. A prompt or a context window carries the procedure, and the model follows it at inference time. That works, but it makes the skill part of the input rather than part of the model, so every rollout pays the context cost and the behaviour depends on the prompt being present and correctly formed. SkillZero is the reference code for a paper that frames the alternative as skill internalization: training a policy so that the skill is absorbed into the weights through in-context reinforcement learning rather than retrieved from the context at each step. The stated evaluation targets are ALFWorld and Search-QA, and the README claims substantial improvements over a standard RL baseline on both. The audience is narrow and identifiable. This is for researchers who train agent policies with RL, who already work in one of those two environments, and who want to reproduce or extend a specific method rather than drop a component into a production agent. If you are building an assistant that needs tool use next week, nothing in this repository is aimed at you.

What the repository actually contains: two environment stacks and a scripts directory

The layout is a training repository, not a library. All training entry points live under scripts/, and the README states that those scripts assume the repository root as the working directory and cd there automatically. Two named scripts are given: scripts/train_alfworld_skillzero_3b.sh and scripts/train_search_skillzero_3b. Checkpoint merging is handled separately, with scripts/model_merger.py described as containing FSDP and Megatron merge examples that use paths under ./checkpoints/. The environment side is split into agent_system/environments/env_package/search/third_party for the search environment and a separate data preprocessing path in examples/data_preprocess/. Two figures in docs/skillzero/ carry the method and motivation, and a third carries the metrics, so the conceptual detail sits in the paper at arXiv:2604.02268 rather than in prose in the repository. That distribution of information matters when you evaluate it: the code tells you how to run the method, and the paper tells you why the method is shaped the way it is. Anyone reading only the repository will see the plumbing without the argument.

Installation: pinned versions, a second conda environment, and a downloaded index

The Python environment is created with conda create -n skillzero python=3.12 -y, followed by pip install vllm==0.10.0, pip install flash-attn==2.7.4.post1 --no-build-isolation --no-cache-dir, and pip install -e . from the repository root. Both heavy dependencies are pinned exactly, which is the honest choice for a research release and also the first thing that will break on a newer CUDA stack. WandB logging is optional but wired into many scripts through trainer.logger=['console','wandb'], and the README shows exporting WANDB_API_KEY before running. ALFWorld needs gymnasium==0.29.1, stable-baselines3==2.6.0, and the alfworld package, then alfworld-download -f to fetch PDDL and game files plus a pre-trained MaskRCNN detector into ~/.cache/alfworld/. The search path is heavier. You install the third-party environment package under agent_system/environments/env_package/search/third_party, add gym==0.26.2, and run examples/data_preprocess/preprocess_search_r1_dataset.py, which the README says writes to ~/data/searchR1_processed_direct. Retrieval is a separate conda environment named retriever on Python 3.10, with numpy==1.26.4, torch==2.6.0 and matching torchvision and torchaudio from the cu124 index, transformers, datasets, pyserini, huggingface_hub, faiss-gpu==1.8.0 from the pytorch and nvidia channels, and uvicorn plus fastapi. The index is fetched with examples/search/searchr1_download.py --local_dir ~/data/searchR1, concatenated from part_* files into e5_Flat.index, and the wiki-18.jsonl.gz file is gunzipped in place. The retrieval server is then started with bash examples/search/retriever/retrieval_launch.sh, and the README explicitly recommends redirecting its output to a file because writing to the terminal was observed to spike server response times. That detail is worth taking seriously: it is a documented operational quirk, not a suggestion.

The retriever is a hard dependency with its own failure surface

Search training does not degrade gracefully if the retrieval side is unhealthy. The retrieval server is a local flat e5 index served over FastAPI and uvicorn, and the training script talks to it. Nothing in the README describes a fallback, a health check, or a retry policy. The practical consequence is that a server that is slow, half-started, or logging to the terminal will show up as bad training behaviour rather than as an obvious connection error, and you will be debugging the wrong layer. Combined with the fact that the index must be assembled by concatenating part files and decompressing a JSONL dump, the search pipeline has several steps where a partial success is possible and silent. If you only care about the skill internalization method and not about search specifically, the ALFWorld path is the cheaper way in: three pip installs and one download command, with no second environment and no index. That asymmetry is the single most useful thing to know before you start.

Where SkillZero is the wrong tool

There are no releases. The README lists news entries that are all paper and sibling-project announcements, with the most recent being another repository rather than a versioned artifact of this one. There is no changelog, no versioning scheme, and no statement about backward compatibility. The dependency pins are exact, and the search stack requires an older Python than the main environment, so upgrading either torch or vLLM means re-validating the whole pipeline yourself. The README is also visibly truncated in the acknowledgement section, and the training section is short enough that the flags passed by the shell scripts are the real documentation. If your requirement is a maintained dependency with a support commitment, this is not it. It is also the wrong choice if your environment is not ALFWorld or Search-R1: the repository ships two environment packages, and adapting a third means writing your own environment integration against an interface that is documented only by example. Finally, if you need multi-node or large-scale training guarantees, nothing in the supplied material addresses them.

How it compares to the sibling projects in its own news list

The README's news section is effectively a map of alternatives, all from overlapping author groups. SDAR, at ZJU-REAL/SDAR, is described as self-distilled agentic reinforcement learning for skill internalization, which is the closest neighbour and the same lab. The difference in approach is where the training signal comes from: SkillZero's framing is in-context reinforcement learning, while SDAR's framing is self-distillation, and the two later projects extend that line further. OPID is described as on-policy skill distillation, SEED as self-evolving on-policy distillation beyond skill internalization, SkillRise as cross-task skill evolution via agentic RL, Skill1 as evolving skill-augmented agents in one unified policy, and AgentOPSD as recursive credit update for SDAR. Read as a set, these are successive attempts at the same underlying question, and SkillZero is the earliest of the group in this list. If you are choosing between them, the honest selection criterion is not which is newest but which training signal matches the data you have: reinforcement learning against environment reward, distillation from a stronger policy, or cross-task transfer. The repository itself does not provide a comparison table or shared benchmark numbers across the siblings, so that choice has to be made from the papers.

Maintenance cost, licensing, and what to check before committing

The licence is Apache-2.0, which permits commercial use and modification and includes an explicit patent grant, with the usual requirements around retaining notices and stating changes. That is the most permissive outcome you could expect from an academic release, and it removes the licensing question from the adoption decision. It does not remove the maintenance question. With no releases, the tracked state is the main branch, and the last push recorded is 2026-08-12, so any fork you build on should be pinned to a commit rather than followed. The exact pins on vllm and flash-attn mean an upgrade is a project, not a bump. Expect to own the environment installation, the retrieval server, and the checkpoint merge path yourself, because scripts/model_merger.py is offered as an example rather than as a supported tool. Before committing, check three things in order: that the pinned vLLM and flash-attn wheels resolve on your hardware, that alfworld-download -f has populated ~/.cache/alfworld/ with both the game files and the detector, and that the retrieval server is answering requests before training starts. Those three checks cover the failure modes this repository is most likely to hand you.

Editorial conclusion

Adopt SkillZero if you already run agentic RL experiments on ALFWorld or Search-R1 and want a reproducible reference implementation of skill internalization, with the paper as the primary source of intent. Do not adopt it if you need a supported library, a stable API, or a pipeline that installs without a second conda environment and a downloaded FAISS index. Verify first that vllm==0.10.0 and flash-attn==2.7.4.post1 build against your CUDA and PyTorch versions, that the ALFWorld assets have landed in ~/.cache/alfworld/ after alfworld-download -f, and that examples/search/retriever/retrieval_launch.sh responds before you launch scripts/train_search_skillzero_3b.sh, because a dead retrieval server will surface as a training failure rather than a setup error.

Official sources

1. Issues
2. License: Apache-2.0
3. Project website
4. README
5. ZJU-REAL/SkillZero on GitHub