NVIDIA SkillEvaluator: Three Independent Tiers for Grading Agent Skills
Multi-tier framework for evaluating AI agent skills with quality gates, semantic overlap detection, synthetic evaluation dataset generation, and live agent evaluation that measures how skills affect agent behavior.
At a glance
- What is it?
- SkillEvaluator packages deterministic validation, semantic deduplication and sandboxed live agent runs into one CLI for skills that follow the Agent Skills specification. The tiers are independent entry points, which is the design decision that matters most when you plan a pipeline.
- Who is it for?
- Adopt SkillEvaluator if you author skills that follow the Agent Skills specification and want a keyless first gate before anything leaves your machine, or if you need a measured answer to whether a skill changes agent behaviour. Skip it if you have no SKILL.md directory to point at, if you cannot run Docker or a sandbox for untrusted code, or if a hosted-model bill per run is unacceptable.
- Can I use it commercially?
- Yes. Apache-2.0 is a permissive licence: you can use, modify and sell software built on it, as long as you keep its copyright and licence notices.
- Is it still maintained?
- Yes. The repository last received commits 1 day ago.
- What is it written in?
- Mainly Python, according to GitHub's language statistics.
Answers come from the project's GitHub data, last synced on September 15, 2026, and from our analysis. They are not legal advice.
DEEP OPEN-SOURCE ANALYSIS
The problem is that a skill folder has no build step
A skill is a folder of instructions and supporting files, defined by the Agent Skills specification. Nothing compiles it. Nothing type-checks the prose. A malformed SKILL.md, a leaked API key in a helper script, or two skills that tell the agent the same thing in different words all reach the agent unchanged. SkillEvaluator exists to put gates in front of that. It is aimed at people who publish skills into a shared catalogue, or who maintain a set of them for one agent runtime and need to know which ones actually change behaviour. The README frames the project as part of the NVIDIA Verified Skills pipeline and lists the tiers by question rather than by feature: is it safe and well-formed, does it overlap with what already exists, and does it help the agent. Those are three different jobs, and the tool keeps them separable.
Three tiers that do not depend on each other
The README states plainly that tiers are independent entry points and nothing requires running earlier ones first. That is unusual for a pipeline diagram, and it changes how you wire the tool. Tier 1 covers validation with commands named validate, quality-check, security-scan, pii-scan, lint-scripts and rubric-eval. Tier 2 covers deduplication with context-optimization-check and similarity-check. Tier 3 covers live evaluation with create-eval-dataset, a tier3 evaluate subcommand and compare. Each tier has its own credential requirements, listed in the overview table. The deterministic Tier 1 checks need no API key at all. Tier 2 needs an embeddings provider, and intra-skill analysis additionally needs a chat model. Tier 3 needs the agent CLI with its own credential plus a sandbox. Because the entry points are independent, a team that only wants PII scanning can install the tool and run one command without ever configuring a model provider.
What the first keyless run actually checks
The quickstart installs every evaluation extra through uv and then runs a scoped validation. The install command is uv tool install --python 3.13 "skillevaluator[all] @ git+https://github.com/NVIDIA/SkillEvaluator.git". The run is skillevaluator validate ./my-skill --checks schema,pii,license,quality,unicode,lint --no-dedup. The README notes this first result needs no API key, no Docker daemon and no repository clone. The target is any directory containing a SKILL.md. The scoped check list is what keeps the run keyless, and the --no-dedup flag keeps Tier 2 out of the path. Two practical details sit alongside it. If the shell cannot find the command after installation, the README points to uv tool update-shell followed by a new terminal. And the complete Tier 1 security scan is a different, heavier thing: it uses external Semgrep, SkillSpector and Gitleaks, plus the security extra, which the scoped first run deliberately avoids.
Provider configuration and the embedding gap
One credential can cover chat and embeddings if you pick the right provider. The README gives NVIDIA Build as the free starting point: set SKILL_EVAL_LLM_PROVIDER=nv_build and NVIDIA_API_KEY, then run skillevaluator models --limit 10 to see what is available. The default model there is nvidia/nemotron-3-nano-30b-a3b, overridable with SKILL_EVAL_LLM_MODEL. OpenAI, Anthropic and Amazon Bedrock are supported through SKILL_EVAL_LLM_PROVIDER values of openai, anthropic and bedrock with their standard credentials, and a local or hosted OpenAI-compatible endpoint uses openai-compatible with SKILL_EVAL_LLM_BASE_URL, SKILL_EVAL_LLM_MODEL and SKILL_EVAL_LLM_API_KEY. The pinned chat defaults are gpt-5.6-sol for OpenAI, claude-opus-5 for Anthropic and us.anthropic.claude-opus-5 for Bedrock, with gpt-5.4-mini documented as the lower-cost OpenAI alternative. The constraint worth reading twice is the embedding gap: Anthropic and Bedrock do not provide embeddings, so Tier 2 needs a separate OpenAI, NVIDIA Build or OpenAI-compatible embedding provider even when chat is already configured. Auto-selection only happens when exactly one of the three well-known API keys is present in the environment.
Deeper runs, autopilot, and what blocks an exit code
The deeper commands are skillevaluator context-optimization-check ./my-skill and skillevaluator similarity-check ./skills. The first checks one skill for repeated guidance and therefore needs both chat and embedding providers; the second needs embeddings only. Before a full run the README asks you to install Semgrep, SkillSpector and Gitleaks, then verify the runtime with skillevaluator doctor --agents codex --env-mode docker, then run skillevaluator validate ./my-skill --full --agents codex --env-mode docker. The --full flag runs all three tiers and enables autopilot. Autopilot behaviour is specific: if the skill has no accepted evaluation source it creates one initial case at evals/evals.json, and if that file already exists SkillEvaluator reuses it. A wider four-bucket dataset comes from skillevaluator create-eval-dataset ./my-skill --full, which the README says should be generated and reviewed first. Gating is where the tiers differ. Tier 1 always gates validate. Tier 2 gates by default, and --no-block-on-dedup keeps its scan and reports but makes findings advisory. Tier 3 is advisory by default, and --block-on-agent-eval promotes its findings, including invalid task-source evidence, into the exit gate. That flag is the single most consequential switch in the tool, because it turns a live agent run into a pass or fail condition.
Where the design costs you
Tier 3 is the part to approach with caution. The README calls live evaluation experimental and restricts it to trusted skills and workspaces, directing untrusted code to Docker or cloud sandboxes instead of local mode. Cost is not fully avoidable: live model calls and managed sandboxes can incur charges, and local mode only avoids the managed sandbox portion, not hosted model charges. The README directs readers to the Tier 3 guide before scaling a run and suggests starting with one agent and a small dataset. There is also a correctness trap in the deterministic tier. Missing scanner evidence makes validation incomplete, so a run that skips Semgrep, SkillSpector or Gitleaks can look clean while covering less than you think. The autopilot default compounds this: a single generated case at evals/evals.json is a starting point, not a benchmark, and the README's own recommendation is to generate the broader dataset and review it rather than accept the one-case fallback. Finally, the material describes a v0.1.0 release from August 2026 with a push in September 2026, so the surface is young and the pinned model defaults may move.
How it differs from running an agent benchmark
Harbor, the open-source agent evaluation framework, powers the sandboxed agent runs inside Tier 3. That relationship is the clearest way to see the difference in approach. A general agent benchmark asks how well a model or agent performs on a task suite; the skill under test is not the variable. SkillEvaluator asks a narrower question with the skill as the independent variable: does this folder of instructions change what the agent does. That is why the tool needs a baseline comparison (the compare subcommand) and why Tier 2 exists at all, since a benchmark suite has no notion of two tasks overlapping semantically. The trade-off is scope. SkillEvaluator will not tell you whether an agent is good at coding; it will tell you whether a skill is well-formed, redundant, or behaviourally inert. If your question is model selection, this is the wrong tool.
Licence, maintenance and what to verify before adopting
The repository is Apache-2.0, which permits commercial use and modification and includes an explicit patent grant; redistribution generally requires keeping the licence and notices, but read the LICENSE file rather than this summary, which is not legal advice. Maintenance signals in the material are limited to a single v0.1.0 release in August 2026 and a repository push in September 2026. The dependency surface is the real ongoing cost: uv for installation, Python 3.12 or 3.13, three external scanners for full Tier 1 coverage, an embeddings provider that Anthropic and Bedrock users must source separately, the agent CLI itself for Tier 3, and a Docker, local or cloud sandbox. Each of those can break independently of SkillEvaluator. The concrete first step is to run skillevaluator doctor --agents codex --env-mode docker against your chosen runtime and confirm it reports the scanners and sandbox as present before you trust a --full result.
Editorial conclusion
Adopt SkillEvaluator if you author skills that follow the Agent Skills specification and want a keyless first gate before anything leaves your machine, or if you need a measured answer to whether a skill changes agent behaviour. Skip it if you have no SKILL.md directory to point at, if you cannot run Docker or a sandbox for untrusted code, or if a hosted-model bill per run is unacceptable. Before trusting a green result, install Semgrep, SkillSpector and Gitleaks and check the doctor output, because the README states that missing Tier 1 scanner evidence makes validation incomplete.
Community notes