upskill: turning a strong model into a skill a weak model can reuse

Upstreaming a strong model's habits to a cheap one

The gap upskill targets is a cost gap, not a capability gap. A large model can do a task, but it is slow and expensive to run on every request. A small model is cheap but loses the task. The project's own description puts it as the instruction to "create skills with teacher models (expensive/slow) that student models (cheap/fast) can use to perform harder tasks reliably."

The artifact that carries the knowledge across is a skill directory containing a SKILL.md file, the same kind of file that coding agents such as Claude Code, Open Code and OpenAI Codex are built to load. upskill does not run those agents directly as a product. It produces and grades the skill files, so the workflow fits a team that already has its own agent harness and wants a repeatable way to author the skills that harness consumes.

That framing matters for judging scope. upskill is a generator and an evaluation harness, not an agent runtime. If your problem is that your cheap model cannot follow your commit message conventions, upskill can try to encode those conventions into a skill and then measure whether the cheap model follows them more often with the skill than without it. If your problem is that you lack a place to run agents at all, upskill is the wrong tool.

Four phases behind the upskill command

The project separates work into four phases with explicitly different model roles: skill generation, which creates or refines the SKILL.md; test generation, which produces synthetic evaluation cases; evaluation, which runs those cases against one or more evaluator models; and benchmark, which repeats evaluation across several runs or models.

Each phase can draw on a different model, and the command line exposes that separation rather than hiding it behind a single default. `upskill generate` takes a `--model` flag for the model that writes the skill and a `--test-gen-model` override for the model that writes the tests. It also accepts `--eval-model` for a cross-model pass in which a different model grades the result. `upskill eval` takes `-m/--model` as a repeatable flag, and the same repeatable pattern appears on `benchmark`, while on `runs` and `plot` the model flag only filters historical results.

A skill therefore becomes a portable artifact tied to a model pair rather than a single prompt. The CLI treats the phases as separate because the whole premise is that the model that writes the skill should not have to be the model that runs it.

Installing the CLI and writing your first skill

The package ships on PyPI and the project suggests uv for installation. Running it without installing gives a one-off invocation:

uv pip install upskill

uvx upskill

A first skill needs only a task description. The command below writes the result to ./skills/{skill-name}/ by default, which is where later commands expect to find it:

upskill generate "write good git commit messages"

If you already have a recorded agent trace, or a skill directory you want improved, the same command accepts it through `--from`, and the path form is auto-detected:

upskill generate "document the pattern" --from ./trace.md

To see whether the generated skill actually helps a cheaper model, name the generating and evaluating models separately. The generation phase writes the skill with sonnet, and the evaluation pass runs it against haiku:

upskill generate "write good git commit messages" --model sonnet --eval-model haiku

Evaluation is then a separate step against a skill directory, and repeating the `-m` flag is what turns an evaluation into a benchmark:

upskill eval ./skills/git-commit-messages/ -m haiku -m sonnet

Run logs are written to ./runs by default, which `--runs-dir PATH` can override, and they can be read back later by skill name:

upskill runs --skill git-commit-messages

Why --no-baseline disappears in benchmark mode

upskill eval has two modes that behave differently enough to trip up first-time users. In simple evaluation mode the command compares the skill against a baseline, and `--no-baseline` turns that comparison off. Once you pass more than one `-m` value, or set `--runs` above 1, the command enters benchmark mode, and in that mode baseline comparison is always off. The documentation is blunt about the consequence: `--no-baseline` is redundant there.

This matters because the baseline is the more informative half of a single-model result. A score on its own says the skill is present, but the difference between the baseline bar and the with-skill bar is the claim you actually want to defend. The generate output shows that shape, with a baseline bar, a with-skill bar and a token count that moves from 1200 to 800 in the documented example. A benchmark run across haiku and sonnet gives you relative standing instead, which is the right question when you are choosing a model but the wrong one when you are arguing that a skill earns its place.

The same command family also has a `plot` subcommand, which reads historical results rather than producing new ones.

Remote evaluation on Hugging Face Jobs

The v2 tip in the project documentation points out that the recommended default configuration now runs evaluations on Hugging Face Jobs, which means the remote path is no longer an exotic option but the recommended one. Two things have to be in place. The environment needs an HF_TOKEN, and the job creation and result capture need a dataset repository named through `--artifact-repo`.

The flag itself is listed as required with the jobs executor, so the combination is not optional:

upskill generate "parse invoices" --executor jobs --artifact-repo <user>/upskill-tests

The execution backend is selected per command through `--executor [local|jobs]`, and it overrides whatever the configuration file says. `--max-parallel N` sets the concurrency of evaluation executions and also overrides the config. The repository root holds both `upskill.config.yaml` and `fastagent.config.yaml`, so configuration is expected to live in checked-in files rather than only in flags.

The Makefile shows the same idea from the shell side, with targets named hf-go-check, hf-go-smoke, hf-go-prompt and hf-go-batch, an ARTIFACT_REPO variable, and a check that requires the scripts to be executable and that runs `hf auth whoami` before proceeding. The remote path therefore assumes both a Hugging Face account and a repository you are willing to store run artifacts in.

Test generation is the weakest link in the chain

The evaluation cases are synthetic, and they are produced by a model. A separate `--test-gen-model` override exists for every command that may generate tests, which tells you that the project treats test quality as a variable worth controlling, but the cases are still machine-written and they are written before the evaluation model ever sees the skill.

That ordering creates a blind spot. If the test generator misreads the task, the evaluation measures the wrong behaviour, and the reported improvement between baseline and skill will look clean while being irrelevant. The output block in the documentation shows exactly this shape of evidence, a baseline at 60 percent and the skill at 100 percent for the same generated cases, and those percentages are only as meaningful as the cases behind them.

Two mitigations are visible in the interface rather than guaranteed by it. `upskill eval` accepts `-t, --tests PATH`, a test cases JSON file, so a hand-written or reviewed case set can replace generated cases. And `-v, --verbose` shows per-test results, which lets you read the individual cases instead of trusting the aggregate bar. Neither is a default, so anyone relying on a headline number should check which of the two they used.

Hand-written SKILL.md compared with a generated one

The alternative worth naming is not another package but the manual route: write the SKILL.md yourself, hand it to the agent, and judge the result by reading transcripts. The difference in approach is the part that matters. The manual path costs you writing time and gives you a skill whose scope is exactly the case you thought of, with no measurement attached and no way to say whether it helped. upskill inverts that cost. It spends teacher model calls and test generation calls to produce a skill and, more importantly, a number and a run history that let you compare against a no-skill baseline.

That inversion has a failure mode of its own. A generated skill is a model artifact, so its wording, scope and example choices may not match the conventions you want, and fixing them means going back through `--from` with an existing skill directory rather than editing a file by hand. The generated-and-evaluated route wins when you have many models and want a comparable picture; the manual route wins when you have one model, one task, and strong opinions about the wording.

A third option, running the evaluation stage alone with a skill you wrote yourself, gets you the measurement without paying for generation. `upskill eval` accepts a skill directory path and a JSON test file, which makes it usable as a benchmarking tool for hand-written skills too.

Licence, dependencies and the cost of keeping it running

upskill is released under Apache-2.0, and the package metadata shows a narrow dependency set: click, fast-agent-mcp, pydantic, python-dotenv, pyyaml and rich, with Python required at 3.13.5 or later and below 3.14. The version in the metadata is 0.2.1, and the repository has no GitHub releases, so there is no release history to point at and no published changelog to read.

The repository was not archived, and the last push was on 2026-10-14. There is no stated release cadence, and the version number alone is not evidence of one. A pre-1.0 version with a pinned Python range means an upgrade can be as simple as reinstalling the package and as disruptive as a new Python.

For contributors, the project enforces a fairly wide static gate: ruff format for formatting, ruff check across style, imports, modernization, bugbear, simplify and import hygiene rules, cyclomatic complexity capped at 15 through Ruff's C90 rules, ty type checking across src, tests and scripts, pmd cpd for duplicated code in src, and pytest. The helper script runs the whole sequence:

uv run scripts/check.py

The same checks run in CI on pushes and pull requests targeting main.

Editorial conclusion

upskill fits teams that already run several models at once and can afford teacher model calls, since the value of a generated skill depends entirely on the evaluation loop that measures it. It does not fit a single-model setup or anyone unwilling to let a model write the tests that judge another model. Before adopting it, check that your runtime sets HF_TOKEN and passes --artifact-repo if you plan to use the jobs executor, and confirm from your own runs under ./runs that the skill you get actually beats the baseline on your prompts.

Frequently asked questions

Official sources

1. huggingface/upskill on GitHub
2. Issues
3. License: Apache-2.0
4. README