Skillgrade: running agent skills through repeated trials instead of one lucky prompt
"Unit tests" for your agent skills
At a glance
- What is it?
- Skillgrade is a TypeScript CLI that scaffolds an eval.yaml beside a SKILL.md file, runs an agent against fixture workspaces in Docker, and scores each trial with deterministic scripts or an LLM rubric. The judgement: it is a trial-runner for skill discovery and tool use, not a general LLM benchmark harness, and its usefulness depends entirely on the graders you write.
- Who is it for?
- Adopt Skillgrade if you maintain a SKILL.md-based skill and want a repeatable pass rate rather than a single manual prompt, and if you are willing to write deterministic graders, since the LLM rubric path depends on a grader model you have to configure. Skip it if you need to evaluate a general agent workflow with no skill directory, or if you cannot run Docker and have not checked whether the local provider covers your case.
- Can I use it commercially?
- Yes. MIT is a permissive licence: you can use, modify and sell software built on it, as long as you keep its copyright and licence notices.
- Is it still maintained?
- Yes. The repository last received commits 20 days ago.
- What is it written in?
- Mainly TypeScript, according to GitHub's language statistics.
Answers come from the project's GitHub data, last synced on September 15, 2026, and from our analysis. They are not legal advice.
DEEP OPEN-SOURCE ANALYSIS
The problem: a skill that works once is not a skill that works
Agent Skills are directories with a SKILL.md file that an agent is supposed to discover and then follow. The failure mode is not a crash. It is an agent that never notices the skill, or notices it and uses the wrong tool from it, or uses the right tool in the wrong order. A single manual prompt in a chat window tells you almost nothing about that, because the model, the agent CLI and the sampling all vary between runs. Skillgrade exists to turn that anecdote into a number. The README describes it as "Unit tests" for your agent skills, and the framing is accurate in one narrow sense: the unit under test is the skill directory, and the assertion is whether the agent's run satisfies a grader. It is aimed at people who publish or maintain skills, not at people benchmarking foundation models. If you do not have a SKILL.md, there is nothing here for you to point the tool at.
What actually runs: eval.yaml, a container, and two kinds of grader
The unit of work is a task declared in eval.yaml. A task has an instruction, an optional workspace of files copied into the container, and one or more graders with weights. The agent receives the instruction inside that workspace and does whatever it does. After the agent process exits, Skillgrade builds the grading input. For a deterministic grader it serialises the task context into a single JSON document in the SKILLGRADE_INPUT environment variable and runs your script, with the working directory set to the workspace. For an llm_rubric grader it puts the rubric in a prompt to a grader model. The README is explicit that Skillgrade "delivers expected, it never interprets it": comparison against the answer key is the grader's job, and the expected block is never shown to the agent. That separation is the most interesting design decision in the project. It means one small script can serve many tasks, and it means the tool does not pretend to know what a correct answer looks like. The example in the README reads SKILLGRADE_INPUT, parses answer.json from the workspace, and prints a JSON object with a score and a details string. A shell grader can do the same with jq. Both channels are constructed after the agent has exited and neither writes to the workspace, which is stated as a contrast with run: and rubric: values, which are staged into the agent's environment.
Trials, presets and the pass rate that comes out the other end
Skillgrade does not report a single boolean. It runs each task N times and reports a pass rate, and the presets table maps N to a stated purpose: --smoke is 5 trials for a quick capability check, --reliable is 15 for a reliable pass rate estimate, --regression is 30 for high-confidence regression detection. --trials=N overrides the count and --parallel=N runs trials concurrently. The README does not publish variance figures or confidence intervals for those presets, so treat the labels as intent rather than statistics. What the presets do give you is a vocabulary for a pull request: a smoke run before merging, a regression run before a release. The --ci flag turns the pass rate into an exit code, non-zero when it falls below --threshold, which defaults to 0.8 in the options table and is also settable per suite in defaults. That is the piece that makes this usable in a pipeline rather than as a local curiosity. Reports land in $TMPDIR/skillgrade/<skill-name>/results/ unless you pass --output=DIR, and skillgrade preview prints a CLI report while skillgrade preview browser serves a web UI on port 3847.
Getting it running: init, edit, run, review
The prerequisites are Node.js 20 or newer and Docker. Installation is npm i -g skillgrade. From inside a skill directory that contains SKILL.md, you run skillgrade init with one of GEMINI_API_KEY, ANTHROPIC_API_KEY or OPENAI_API_KEY set in the environment. With a key present, init generates an eval.yaml with AI-powered tasks and graders; without one it writes a well-commented template. --force overwrites an existing eval.yaml. You then edit the file, and run it with skillgrade --smoke, again with an API key exported. The agent is auto-detected from whichever key you set, and --agent=claude overrides that. The agent list is wider than the three API-key paths: gemini, claude, codex, acp, opencode and command. The command agent takes --command=CMD, and the ACP path takes --acp-command=CMD, with the README giving gemini --acp as the example. There is also --opencode-agent for build, plan or explore, and --opencode-model in provider/model form. The defaults block in eval.yaml covers agent, model, provider, trials, timeout in seconds, threshold, grader_model, grader_provider, the docker block with base and a setup shell string, and an environment block with cpus and memory_mb. Tasks can override agent, model, grader_provider, trials and timeout individually. Two flags are easy to miss: --grader=TYPE restricts a run to deterministic or llm_rubric graders, and --eval=NAME[,NAME] restricts it to named tasks.
Where the model is thin: grader quality is the whole product
Skillgrade cannot tell you whether your graders are any good, and the README's own answer to that is --validate, which verifies graders using reference solutions. That flag is the honest admission at the centre of the project: the tool measures agreement between an agent and your grading logic, and if the grading logic is wrong you get a confident number that means nothing. The llm_rubric grader is the weaker of the two channels by construction. It asks a grader model to judge a free-text rubric, and the model is configurable through grader_model and grader_provider, with per-grader provider and model overrides. That is a second model in the loop whose behaviour you did not pin down unless you set those keys. Weighting compounds the problem: a task with weight 0.7 on a deterministic script and 0.3 on a rubric is mostly measuring the script, and nothing in the tool stops you from getting that ratio backwards. The Docker path is the other constraint. provider accepts docker or local, and the docker defaults shown are node:20-slim with a setup hook for extra packages, plus cpus and memory_mb limits. Any skill whose dependencies are not expressible as a base image plus a setup shell string will need work. And a skill that depends on a live service, a private registry or a GPU is not addressed by anything in the supplied material.
How it differs from a prompt-evaluation framework
The obvious comparison is a general prompt-evaluation library, and the difference is the unit of packaging. A prompt-eval framework takes a prompt, a dataset and a scoring function, and assumes you will bring your own harness. Skillgrade takes a directory containing SKILL.md, auto-detects it unless you set the skill key explicitly, and builds the trial around an agent CLI that must be installed and reachable. The container is the boundary: files listed under workspace are copied in, with src, dest and an optional chmod such as "+x", so a fixture can place an executable at /usr/local/bin/superlint and the agent can call it. That is a different shape of test from a dataset row. The answer key lives in the task as expected, with metadata labels such as tier, form and tags that are recorded with results and usable by a filter. A prompt-eval framework would model that as a column. Neither approach is better in the abstract, but if your artefact is a skill directory rather than a prompt template, the packaging here removes a layer of glue you would otherwise write yourself. The cost is that you inherit the agent CLI matrix and the Docker requirement.
Maintenance, licence and what changes under you
The licence is MIT, which permits commercial use and modification, though the usual caveat applies: this is a description of the licence identifier, not legal advice, and you should read the LICENSE file in the repository for the actual terms. The maintenance surface is larger than it looks for a tool this size. The agent integrations are the fragile part, since each of gemini, claude, codex, acp and opencode is an external CLI whose flags and output can move independently of this project. Pinning --model and grader_model in defaults is the practical defence, because otherwise the README states the default is whatever the agent CLI is configured to use, which is a moving target across machines and across CI runners. The eval.yaml format carries version: "1", so schema changes are at least signposted. There are no releases retrieved in the supplied material, so there is no changelog to read and no version to pin the CLI itself to; you install from the global npm package and take whatever is current. If you need a frozen toolchain, that is a gap you have to close yourself, for example by installing skillgrade as a project dependency rather than globally and committing the lockfile.
Editorial conclusion
Adopt Skillgrade if you maintain a SKILL.md-based skill and want a repeatable pass rate rather than a single manual prompt, and if you are willing to write deterministic graders, since the LLM rubric path depends on a grader model you have to configure. Skip it if you need to evaluate a general agent workflow with no skill directory, or if you cannot run Docker and have not checked whether the local provider covers your case. Before trusting a number, run skillgrade --validate against reference solutions to confirm the graders actually discriminate, then run --smoke and inspect the JSON under $TMPDIR/skillgrade/<skill-name>/results/ yourself.
Community notes