Eval Skills: agent skills for building product-specific AI evals
Skills that guide AI coding agents to help you build product-specific AI evals.
At a glance
- What is it?
- A set of skills that steer AI coding agents through eval work on your own traces and outputs, not foundation model benchmarks. The install is one npx command; the hard part is that the skills only cover the parts of eval work that generalize.
- Who is it for?
- Adopt Eval Skills if you already have traces or an eval pipeline and want an agent to run error discovery or audit what you have, and if you accept that the skills stop at the generalizable parts. Do not adopt it for production monitoring, CI/CD regression suites or cost optimization: the README says those are outside its scope.
- Can I use it commercially?
- Not without permission. GitHub finds no licence file in the repository, and without a licence all rights are reserved by default: you may read the code but not reuse it. Check the README, or ask the authors, before using it.
- Is it still maintained?
- Yes. The repository last received commits 2 days ago.
- What is it written in?
- GitHub does not report a main language for this repository.
Answers come from the project's GitHub data, last synced on September 17, 2026, and from our analysis. They are not legal advice.
DEEP OPEN-SOURCE ANALYSIS
What problem Eval Skills targets, and who it is for
Most teams writing evals start by writing a judge prompt. The README argues the order is wrong: you should only write evals after doing error discovery, meaning qualitative and quantitative analysis of your traces to find failure modes. The project packages that sequencing, plus a set of related habits, as skills that an AI coding agent loads and follows.
The audience is narrow and technical. You need a dataset of LLM outputs or traces in JSONL, CSV or JSON format, and you need to be working inside an agent that supports skills. The README frames the skills as guards against "easily avoidable footguns" in evals, citing experience helping 50+ companies and teaching an AI Evals course. That is the project's own claim about where the patterns come from; there is no published methodology behind the number.
The skills are explicitly not for foundation model benchmarks. They target product-specific evals, the kind that measure whether your application does the thing your users need.
How the skills are structured and how routing works
The repository is a skill bundle, not a library you import. The top level holds .agents/, .claude-plugin/, .codex-plugin/, skills/ and README.md, which is the layout you would expect from a plugin that ships skill definitions to more than one agent host. Each skill lives in its own directory with a SKILL.md file, for example skills/error-discovery/SKILL.md.
evals-start is the entry point. It inspects your situation and routes you to the right skill. The README says most of the time it sends you to eval-audit if you already have an eval pipeline, or error-discovery if you have traces but have not analyzed them yet. That routing step is the mechanism that keeps an agent from jumping straight to judge design.
Seven other skills sit behind the router: generate-synthetic-data, write-judge-prompt, validate-evaluator, evaluate-rag and build-review-interface, plus the two above. Note that the search phrase "what are the 12 skills" does not match this repository. The README's table lists nine.
Inside error-discovery: a stdlib review app and a sampling loop
error-discovery is the skill the README calls the most important, and it is the one with the most concrete pipeline. Point the agent at a JSONL, CSV or JSON file of LLM outputs or traces, and the README says the skill does five things in order.
It reads the dataset and infers the content type (articles, agent traces, code, structured output). It designs visual encoding from what varies in the data, using Gestalt principles: color for categories, spacing for hierarchy, opacity for importance. It builds a single-file HTML review app served by a Python stdlib server, with no dependencies. It clusters the data and picks a diverse initial sample of cluster representatives plus random picks. Then it runs an interactive loop that monitors annotations, categorizes failure modes and proposes new samples to increase coverage.
The division of labour is the interesting design choice. You read and leave free-text notes; the agent sorts those notes into failure modes, tracks coverage and picks new samples to fill gaps. Keeping the human on free text and the agent on taxonomy means the failure-mode categories stay editable, but it also means the quality of the categories depends on how well you write notes.
Installing Eval Skills and running a first error analysis
The README gives one install path, through npx skills from vercel-labs. Run it against the repository URL:
npx skills add https://github.com/ai-evals-course/evals-skillsThat pulls the whole bundle. If you want only the error analysis skill, the README shows the --skill flag with the skill name:
npx skills add https://github.com/ai-evals-course/evals-skills --skill error-discoveryUpdates are handled by two separate commands, which is worth noting because they do different things:
npx skills check
npx skills updateOnce installed, you do not invoke a binary. You point the agent at your data in natural language, as the README's example does:
Can you help me do error analysis on traces.jsonl?What you should see, per the README, is the agent reading the file, standing up the single-file HTML review app on a Python stdlib server, and presenting an initial sample drawn from clusters. You then annotate in that app. The README does not document which port the server binds to, so treat that as something to observe at runtime rather than configure in advance.
Where Eval Skills stops being the right tool
The README is unusually direct about scope. The skills cover the parts of eval work that generalize across projects. Production monitoring, CI/CD regression suites and cost optimization do not generalize, and the README says they are covered by the AI Evals course instead. If your immediate need is a regression gate that blocks a deploy, this repository does not give you one.
The second limitation is the one the README states about its own audit: eval-audit "isn't a complete solution, but it will catch common problems." Treat it as a linter for eval setups, not a certification.
The third is a consequence of the design. Skills encode mistakes that generalize, and the README says skills grounded in your own data and domain will outperform them. So the bundle is a starting point that expects to be extended, and the README points at Matt Pocock's writing-for-ai-agents as a resource for that. A team that installs this and treats the nine skills as a finished eval process will get generic coverage, not coverage of their product's failure modes.
How this differs from eval frameworks and from writing your own skills
The obvious comparison is an eval framework such as a Python library that defines metrics, runs them over a dataset and reports scores. The difference in approach is where the work sits. A framework gives you executable evaluators and expects you to have already decided what to measure. Eval Skills gives an agent instructions for deciding what to measure, and for most skills the artifact is guidance plus a review interface rather than a scoring library. The one place the two overlap is validate-evaluator, which the README describes as calibrating LLM judges against human labels using data splits, TPR/TNR and bias correction.
The second comparison is writing your own skills. The README takes a position here: start with these, then write your own, because domain-grounded skills beat generic ones. That makes the repository closer to a template collection than to a dependency. If your team already has a house style for eval work, the value of adopting nine external skills is mostly in the routing logic and the error-discovery loop, not in the individual prompts.
Maintenance, licensing and what the repository does not state
The last push to the default branch was on 2026-09-16, one day before this writing, and the repository is not archived. That is the only maintenance signal available here. There are no retrieved releases, so there is no version history to reason about, and no changelog in the top-level entries. Upgrades run through npx skills check and npx skills update, which means update cost is tied to whatever the npx skills tool resolves rather than to a pinned version you control. For a skill bundle that an agent loads, that matters: an update can change agent behaviour without a version number changing in your repository.
The license is not stated in the repository metadata available, and the README does not mention one. That is a gap to close before adopting, because the skills are text an agent reads and reproduces into your project, and the terms under which you may reuse that text are not documented here. Nothing in this repository indicates what obligations, if any, come with redistribution. Check the repository's license file directly rather than inferring from the README.
Editorial conclusion
Adopt Eval Skills if you already have traces or an eval pipeline and want an agent to run error discovery or audit what you have, and if you accept that the skills stop at the generalizable parts. Do not adopt it for production monitoring, CI/CD regression suites or cost optimization: the README says those are outside its scope. Before relying on it, read skills/error-discovery/SKILL.md, confirm which skills npx skills actually installs, and check that the Python stdlib server it starts can read your JSONL, CSV or JSON file.
Frequently asked questions
What are evals in skills?
In this project, skills are instruction files that an AI coding agent loads to guide eval work, and evals are product-specific evaluations of your own LLM outputs or traces rather than foundation model benchmarks. The README states that you should only write evals after doing error discovery on those traces.
What are the 12 skills in Eval Skills?
The README's table lists nine skills: evals-start, eval-audit, error-discovery, generate-synthetic-data, write-judge-prompt, validate-evaluator, evaluate-rag and build-review-interface. No list of twelve appears in the README.
What are evaluation skills?
The repository describes them as skills that guard against common mistakes in evals, drawn from patterns the authors say generalize across projects. Each skill lives in its own directory under skills/ with a SKILL.md file.
What do evals mean in AI?
The README distinguishes product-specific AI evals from foundation model benchmarks, and says error discovery on your traces should come before writing any eval. The eval-audit skill inspects an existing pipeline and surfaces problems with prioritized severity.
Community notes