agent-skills-eval: a before-and-after runner for Agent Skills
A test runner for agentskills.io-style AI agent skills
At a glance
- What is it?
- The project runs each eval twice, once with a SKILL.md in context and once without, then has a judge model grade both sides. It is a fair way to ask whether a skill earns its place, and it inherits every weakness of LLM-as-judge grading.
- Who is it for?
- Adopt it if you already maintain skills in the agentskills.io layout and want a repeatable with_skill versus without_skill number in CI, driven by the agent-skills-eval.yaml config and the evaluateSkills SDK entry point. Do not adopt it if you need deterministic scoring, because the README describes judge-graded pass/fail with no rubric beyond the eval's own assertions.
- Can I use it commercially?
- Yes. MIT is a permissive licence: you can use, modify and sell software built on it, as long as you keep its copyright and licence notices.
- Is it still maintained?
- Yes. The repository last received commits 41 days ago.
- What is it written in?
- Mainly TypeScript, according to GitHub's language statistics.
Answers come from the project's GitHub data, last synced on September 15, 2026, and from our analysis. They are not legal advice.
DEEP OPEN-SOURCE ANALYSIS
The gap between shipping a SKILL.md and knowing it helped
The README frames the problem in one line: Agent Skills make it easy to ship a SKILL.md and assume your agent is now better at the task, and the hard part is proving it. That is the whole reason the project exists. A skill file is a prompt artifact. Once it is in a repository, nothing about the file itself tells you whether the model's answers improved, or whether you wrote 400 words of context that the model would have produced anyway. The tool is aimed at people who author skills rather than people who consume them: skill maintainers, agent platform teams, and anyone who has to justify a skill's existence to a reviewer. The unit of work is the skill directory, not the agent runtime, and the README states this separation explicitly, saying it works wherever your skills do.
Why every eval runs twice
The mechanism is a paired run. For each eval defined in a skill, the runner sends the same prompt to the target model along two paths: with_skill, where SKILL.md sits in context, and without_skill, a baseline with the skill stripped. Both outputs go to a judge model, which sees the eval's expected_output and assertions and grades each side independently, returning pass or fail per side. The comparison is what carries the signal, and the README is direct about the switch that enables it: the --baseline flag is what enables the comparison, and without it you only get the with_skill run. That default matters. A single-sided run tells you the model passed your assertions, which is not the same claim as the skill caused the pass. If you skip --baseline you have a smoke test, not an eval. The judge is a separate model from the target, configurable independently, and the README describes it as any chat model, which means you can grade a small target with a stronger judge or the reverse.
The workspace layout is the actual product
Output lands in a workspace directory with an iteration-N layout. Inside an iteration you get meta.json for run metadata, benchmark.json with rolled-up pass/fail per skill, one directory per eval containing with_skill and without_skill subdirectories with output, timing and judge grading, and a report directory holding a static index.html. That last piece is the one that changes how a team uses the tool. The report is a file, not a hosted service, so it can be published to any static host or attached to a pull request. The artifacts are JSON and JSONL throughout, which means a run today can be diffed against a run tomorrow without re-running anything. I would treat benchmark.json as the integration point and index.html as the artefact for humans. The README's claim that you get a real, evidence-backed answer is fair only if the assertions behind the grading are written tightly enough to distinguish a good answer from a plausible one.
Running it: CLI flags, then a YAML file
The quickstart is a single command. You point the tool at a folder of skills, name a target model and a judge model, and turn on the baseline: npx agent-skills-eval ./skills --target gpt-4o-mini --judge gpt-4o-mini --baseline --strict. Past that, the README recommends a config file at the project root, agent-skills-eval.yaml, with keys for root, workspace, baseline, target, judge, baseUrl, apiKeyEnv, include, exclude, concurrency, layout, strict, and report, plus nested blocks for logging, targetParams and judgeParams. Credentials come from the environment via apiKeyEnv, so the key itself never sits in the file. The README states that CLI flags always override config values, which is the behaviour you want for a CI job that raises concurrency or flips strict on a release branch. The provider layer is OpenAI-compatible by default, and the README lists OpenAI, Together, Groq, Anthropic through OpenAI-compat layers, and local Llama servers as targets, so baseUrl is the field that decides where requests actually go.
The SDK is where custom pipelines start
The package exports OpenAICompatibleProvider, consoleReporter and evaluateSkills. You construct a provider with baseUrl, apiKey, model and providerName, then call evaluateSkills with root, workspace, baseline and concurrency, and read the returned result. That is enough to wire the runner into a build step, a multi-skill rollup, or a dashboard that reads benchmark.json rather than parsing console output. The README also mentions custom providers as an SDK use case, though the supplied material does not document the provider interface beyond the OpenAI-compatible implementation, so anyone swapping in a non-compatible backend should expect to read the source. The same gap applies to tool-call assertions: the feature list mentions deterministic checks for agents that call tools, but no example of the assertion shape appears in the README, which is the first thing I would go looking for in the docs site.
Judge grading is the weak joint
Every pass/fail in this tool comes from a model reading two outputs and deciding. The README describes the result as pass/fail with cited assertions, not vibes, and the citation requirement helps, but the grading is still a model's judgement of text. Three consequences follow. First, the same run can produce different verdicts across judge models, so a skill that passes under one judge may fail under another, and the tool gives you no cross-judge agreement number. Second, a weak assertion set produces a weak eval: if the assertion is vague, the judge will pass both sides and the skill's lift will look like zero. Third, temperature settings for target and judge live in targetParams and judgeParams, and the README's example sets both to 0, which reduces variance without eliminating it. None of this is a defect unique to this project, but it does mean the number in benchmark.json is an estimate, and treating it as a hard gate in CI will eventually block a merge for a reason nobody can reproduce.
Where a paired runner is the wrong tool
The with_skill versus without_skill design assumes the skill's effect is a change in a single model's output for a fixed prompt. That assumption breaks in two common cases. If your skill mainly affects tool selection, the text output may look identical on both sides while the tool calls differ, and only the deterministic tool-call assertions can catch it, which puts the burden on assertions the README does not document. If your skill is one of several loaded together, stripping it changes the prompt in a way that no longer matches production, and the baseline becomes a strawman. For plain unit testing of a prompt template with exact-match scoring, a general eval library with string or code-based graders is a better fit, because the difference in approach is fundamental: those tools compare output to a reference, while agent-skills-eval compares two model outputs to each other through a third model. The paired design is the point here, and it is also the constraint.
Licence, upgrades and what to check first
The project is MIT licensed, which permits commercial use and modification with the copyright notice retained; the repository's LICENSE file is the authority, and this is not legal advice. Maintenance cost is mostly your own eval suite rather than the tool. Skills are validated against the agentskills.io specification, including SKILL.md frontmatter rules and the evals/evals.json format, so a spec revision upstream is the event that would force a version bump on your side. The repository shows no retrieved releases in the supplied material, so installs resolve to whatever npm serves for the version range you request; pinning an exact version in package.json is the only way to keep a CI run stable across installs. The first things to verify on a real skill: that evals/evals.json assertions are specific enough for a judge to grade, that --baseline is actually set in your config rather than assumed, and that the judge model differs from the target model so you are not grading a model with itself.
Editorial conclusion
Adopt it if you already maintain skills in the agentskills.io layout and want a repeatable with_skill versus without_skill number in CI, driven by the agent-skills-eval.yaml config and the evaluateSkills SDK entry point. Do not adopt it if you need deterministic scoring, because the README describes judge-graded pass/fail with no rubric beyond the eval's own assertions. Before trusting a run, verify that your evals/evals.json assertions are specific enough to grade, and that the judge model you pass to --judge is not the same model you are testing against.
Community notes