skill-up: Evaluating and Evolving Agent Skills from the Command Line
An evaluation and evolution tool for Agent Skills.
At a glance
- What is it?
- Alibaba's skill-up turns Agent Skill evaluation into a declarative YAML suite that runs across several agent engines and produces structured reports. The companion skill-upper closes the loop by reading failures and repairing the evals. The design is sound, but the tool is only as useful as the cases you write.
- Who is it for?
- Adopt skill-up if you already maintain SKILL.md files and want evaluation to live in version control next to them, especially if your team runs more than one agent engine and needs comparable reports. Skip it if your skill quality bar is still being defined by hand, since the tool will faithfully measure the wrong cases.
- Can I use it commercially?
- Yes. Apache-2.0 is a permissive licence: you can use, modify and sell software built on it, as long as you keep its copyright and licence notices.
- Is it still maintained?
- Yes. The repository last received commits 1 day ago.
- What is it written in?
- Mainly Go, according to GitHub's language statistics.
Answers come from the project's GitHub data, last synced on September 15, 2026, and from our analysis. They are not legal advice.
DEEP OPEN-SOURCE ANALYSIS
The gap skill-up fills between writing a SKILL.md and trusting it
An Agent Skill is a directory with a SKILL.md file that tells an agent how to behave for some task. The hard part is not writing the file. It is knowing whether a change to it made the agent better or worse. The official Agent Skills evaluation guide, linked from the skill-up README, describes the loop: write realistic cases, run with and without the Skill, grade the outputs, aggregate, iterate. In practice that loop is usually run by hand, with ad hoc run folders and no way to compare two runs a week apart. skill-up targets exactly that gap. It is a Go CLI that takes a declarative eval suite and executes it against one or more agent engines, then emits machine-readable reports. The intended user is an engineer or platform team that maintains skills as artifacts and wants the same discipline applied to them that would be applied to any other code change. It is not aimed at someone who wants a one-off sanity check of a prompt.
eval.yaml plus cases: the declarative suite format
The unit of work is a directory of YAML. The README describes the format as eval.yaml plus cases/*.yaml. The eval.yaml carries the evaluation environment, the engine, and the model. The case files carry the individual scenarios. That split matters because it means the expensive, environment-specific choices (which engine, which model) live in one file, while the scenarios live in many small files that are easy to add, delete, and review in a pull request. The README's own directory sketch shows evals/eval.yaml and evals/cases/<case-id>.yaml sitting next to SKILL.md, with results written to a separate workspace directory under iteration-1/result.json. Keeping inputs and outputs in separate trees is a small decision with real consequences: you can commit the evals and gitignore the workspace, so iteration history does not pollute the repository. The cost is that the case format is something you have to learn. The README points at a Writing Evals guide rather than inlining the schema, so plan on reading the documentation before your first suite compiles.
Three judge strategies and what each one can actually decide
skill-up supports rule_based, script, and agent_judge evaluation strategies. These are not interchangeable, and the choice is the most consequential part of authoring a suite. A rule_based judge is deterministic and cheap, but it can only check things that can be expressed as rules, such as whether the output contains a required string or matches a pattern. A script judge hands the check to your own code, which is the escape hatch when the criterion is deterministic but too awkward to express declaratively. An agent_judge uses a model to grade the output, which is the only one of the three that can assess qualities like whether a summary is faithful to a source. It is also the one whose verdicts will drift when the judge model changes. The README does not state which judge model the agent_judge strategy uses by default, so that is something to confirm in the user configuration documentation before you rely on a score being reproducible. Mixing strategies within one suite is the sensible pattern: cheap rule checks for the mechanical requirements, an agent judge for the parts that need reading comprehension.
Engines, transports, and the custom engine escape hatch
The built-in engines are Qoder CLI, Claude Code, and Codex, and the README elsewhere lists qwen_code among the supported engine names. That plurality is the main architectural argument for skill-up over a bespoke script: the same case can be run under more than one client, which surfaces skills that only work because of one agent's quirks. The README is explicit that user-defined agents go through engine.custom and that this uses a local transport, pointing at docs/design/custom-engine.md for the details. The word local is worth pausing on. It means a custom engine is something running on the same machine as the CLI, not a remote service, which constrains how you can wire skill-up into a hosted evaluation farm. If your agent is only reachable over the network, the material supplied here does not describe a path for that, and you should treat it as unverified rather than assume a remote transport exists.
Installing the CLI and getting a first suite to run
There are two entry points. The recommended one is skill-upper, the Agent Skill shipped in the repository, installed with npx skills add against the skills/skill-upper path, with flags -g for global and -a codex or -a claude-code for the target agent. The README notes you normally do not need to install skill-up first, because skill-upper checks for the CLI at runtime and walks the agent through installation if it is missing. The manual path is a single curl piped to bash from install.sh on the main branch. Piping a remote script into a shell is a supply chain decision, so if that matters to you, download install.sh, read it, and run it locally instead. Once installed, the workflow is to open a project containing your SKILL.md and ask the agent to read the file, derive the important behaviours, create cases with appropriate judges, validate the configuration, and run skill-up. The README also documents skill-up import for bringing in an Anthropic-style evals.json, and an --auto flag that auto-detects that format. The build requires Go 1.25 or later, per the badge in the README, which is a recent toolchain and worth checking against your CI base image before you plan a pipeline.
Reports: grading.json, benchmark.json, JUnit XML, and HTML
Output is where skill-up is most concrete. It writes Anthropic-compatible grading.json and benchmark.json, a human-readable benchmark.md, a result.json, JUnit XML, and an HTML report. The JUnit XML is the piece that makes CI integration straightforward, because most CI systems already know how to display test results in that format. The Anthropic-compatible JSON files are the piece that makes migration cheap in both directions: you can import an existing evals.json and you can hand your results to tooling that already understands that shape. The HTML report is for the human reading a failure at 5pm. This is a wider report surface than most internal eval harnesses produce, and the compatibility claim is specific enough to be checkable. What the material does not say is whether the report schema is versioned or stable across releases. Given that the project shipped v0.9.0, v0.9.1, and v0.10.0 within about a month, anything downstream that parses grading.json should pin the skill-up version it was written against.
The evolution loop, and why it is the part to be sceptical about
skill-upper is described as reading failed reports, diagnosing whether the Skill or the eval is wrong, repairing or expanding the case suite, and rerunning skill-up until the suite evolves. That is an appealing loop, and it is also the part of the design with the least mechanical guarantee. An agent that can edit both the artifact under test and the test suite can, in principle, make failures disappear by weakening the test rather than fixing the skill. The README's own prompt template asks the agent to determine whether the Skill or the eval is wrong, which acknowledges the ambiguity rather than resolving it. The practical mitigation is process, not tooling: review eval changes with the same scrutiny as SKILL.md changes, and treat a case deletion in a pull request as a signal rather than a cleanup. Nothing in the supplied material describes a guard that prevents the agent from loosening a judge or deleting a case, so if that risk matters to your team, you are relying on code review to catch it.
Where skill-up is the wrong tool, and what to use instead
skill-up assumes you have a SKILL.md and a set of behaviours worth pinning down. If you are still exploring what the skill should do, writing eval cases first will freeze a design you have not settled on, and you will spend the loop repairing evals instead of improving the skill. The other poor fit is a single-engine shop with one or two simple assertions. If your check is that the agent outputs valid JSON, a shell script and a jq call will do it in ten lines, with no YAML schema, no workspace directory, and no Go toolchain to install. The genuine alternative for teams that want evaluation inside the agent rather than beside it is a framework like promptfoo, which centres on a config file of prompts and assertions and runs providers through a Node toolchain. The difference in approach is where the artifact lives. promptfoo treats the prompt as the thing under test and keeps everything in one config; skill-up treats a skill directory as the thing under test and separates the environment declaration from the individual cases, which is why it can run the same suite across claude_code, codex, and qodercli and emit Anthropic-shaped reports. If cross-engine comparison is not something you need, that separation is overhead rather than a feature.
Licence, maintenance, and the upgrade cost of a fast-moving CLI
skill-up is Apache-2.0, which permits commercial use, modification, and redistribution provided you keep the licence and notice files and state significant changes. That is a permissive licence and it is compatible with the common practice of vendoring a tool into an internal build. It is not legal advice, and if you plan to redistribute a modified binary you should have someone read the actual LICENSE file rather than this paragraph. On maintenance: the release cadence visible in the supplied material is three releases between 2026-08-12 and 2026-09-01, with the repository last pushed on 2026-09-10. That is an actively changing tool. The maintenance cost that falls on you is not patching the CLI, it is the report consumers. Anything that parses grading.json, benchmark.json, or the JUnit XML is coupled to a schema that has had three releases in a month. Pin the version in CI, and treat a skill-up upgrade as a change that requires re-running the suite and diffing the report shape before it lands.
Editorial conclusion
Adopt skill-up if you already maintain SKILL.md files and want evaluation to live in version control next to them, especially if your team runs more than one agent engine and needs comparable reports. Skip it if your skill quality bar is still being defined by hand, since the tool will faithfully measure the wrong cases. Before committing, verify that the Go 1.25 toolchain is available in your CI image, that your chosen engine is one of the built-in ones or works through engine.custom, and that the judge strategy you pick can actually express your pass criteria.
Community notes