Skill Scanner: Cisco's Best-Effort Detector for Agent Skill Packages
Security Scanner for Agent Skills
At a glance
- What is it?
- Skill Scanner is a Python security scanner for Agent Skills that layers YAML and YARA-X signatures, AST and dataflow analysis, an optional LLM judge, and a bounded CEL decision layer. Its own benchmark numbers show why you should treat it as one signal in a review pipeline rather than a gate you trust on its own.
- Who is it for?
- Adopt skill-scanner if you already review Agent Skills by hand and want a repeatable first pass that emits SARIF into GitHub Code Scanning or runs as a pre-commit hook. Do not adopt it as an approval gate: with every bundled CEL rule in shadow mode and a source-disjoint F1 of 13.74 percent, a clean scan is not evidence of safety.
- Can I use it commercially?
- Check first. The repository uses a licence we do not classify automatically, so read its LICENSE file before any commercial use.
- Is it still maintained?
- Yes. The repository last received commits 10 days ago.
- What is it written in?
- Mainly Python, according to GitHub's language statistics.
Answers come from the project's GitHub data, last synced on September 15, 2026, and from our analysis. They are not legal advice.
DEEP OPEN-SOURCE ANALYSIS
The problem: skills ship executable intent, not just text
An Agent Skill is a package of instructions and code that an agent loads and then acts on. That makes the skill itself an attack surface. A markdown file can carry a prompt injection that redirects the agent, a script can exfiltrate environment variables, and a tool definition can quietly widen what the agent is allowed to touch. Reviewing these by eye does not scale once a repository holds dozens of skills from different authors.
Skill Scanner targets that review step. It reads a skill package and produces findings about prompt injection, data exfiltration, and malicious code patterns, according to the README. The intended audience is the person who has to decide whether a skill is safe to install: a security engineer reviewing a third-party skill repo, or a platform team gating contributions to an internal skill registry. It is not a runtime sandbox and it does not execute the skill to observe behaviour. Everything it reports comes from reading files.
Four detection layers plus a CEL decision step
The architecture is a pipeline of analyzers feeding a decision layer. The first layer is pattern matching over YAML rules and YARA-X signatures. The second is AST and dataflow analysis in Python, which the README describes as behavioural dataflow analysis: instead of matching a string, it traces how data moves through code to catch flows that a signature would miss. The third is an optional LLM-as-a-judge that reads the skill semantically. The fourth is described as cloud-based scanning. A meta-analyzer sits above the findings, correlating and prioritising them, and can filter some out.
The interesting design choice is where the CEL layer sits. The README states that the core scanner uses the official cel-go v0.32.0 runtime to correlate bounded facts after deterministic detection and before optional LLM analysis. So CEL is not another detector. It is a policy expression layer that consumes typed facts produced by the detectors and decides what to do with them. The word bounded matters: the CEL rules operate over a fixed fact schema rather than arbitrary data, which keeps evaluation deterministic. The README reports that five CEL-shadow runs were exact and deterministic and that CEL evaluated 154 candidates without proposing a suppression or falling back. That determinism is the property worth having in a layer that can suppress findings.
The published numbers are the most useful part of the README
Most security tools ship a feature list. This one ships a benchmark table, including the result it failed. On a development benchmark of 5,256 malicious and 1,338 benign MaliciousSkillBench packages, the README reports F1 rising from 32.92 percent to 47.73 percent and recall from 19.88 percent to 31.43 percent against origin/main, with the benign false-positive rate falling from 3.59 percent to 1.05 percent and precision at 99.16 percent.
Then the weaker result. On a locked source-disjoint split the scanner records TP=65, FP=42, TN=503, FN=774. That is 60.75 percent precision, 7.75 percent recall, 13.74 percent F1, and 7.71 percent false-positive rate. The README states plainly that this improves F1 over origin/main (7.40 percent) but regresses FPR (3.67 percent), so it does not pass the promotion gate. The consequence is stated in the same paragraph: every bundled CEL rule remains in shadow mode, and the change does not promote any CEL suppression.
Read those two paragraphs together and you have the honest picture. High precision on a corpus that resembles the training data, and recall in the single digits once the test set is source-disjoint. A scanner with 7.75 percent recall on unseen sources will miss most of what it is looking for. That is not a reason to ignore the tool. It is a reason to size your expectations before you put it in a build pipeline.
Getting it running: pip, pre-commit, and a lenient flag
The package is published on PyPI as cisco-ai-skill-scanner, so installation is a pip install. The README's badge states CPython 3.11 through 3.14. The repository links a Quick Start guide under docs/getting-started/quick-start.md and an API reference under docs/user-guide/, which is where the exact CLI flags live; the README excerpt does not reproduce a full command line, so check those pages rather than guessing at argument names.
Two integration paths are documented. For CI, the project emits SARIF for GitHub Code Scanning and ships a reusable GitHub Actions workflow described in docs/github-actions.md, with exit codes available for failing a build. For local work, it integrates with the standard pre-commit framework so skills are scanned before each commit. The scanner follows the Agent Skills specification and supports OpenAI Codex Skills and Cursor Agent Skills formats. There is also a --lenient flag that extends scanning to non-standard layouts such as Claude Code .claude/commands/*.md files and flat markdown skill repositories. If your skills do not live in a spec-compliant directory, that flag is the difference between a scan and an error.
Tuning runs through a scan policy. The README points to docs/user-guide/custom-policy-configuration.md for presets and tuning, and docs/reference/policy-quick-reference.md as a compact reference for policy sections and knobs. Custom detection rules are covered in docs/architecture/analyzers/writing-custom-rules.md, which describes adding signature, YARA, and Python rules. That is the extension point if your threat model includes patterns the bundled rules do not cover.
Where it fails, and where it is the wrong tool
The README is unusually direct about this, and it is worth repeating without softening. No findings does not mean no risk. A clean scan indicates that no known threat patterns were detected, not that the skill is secure. Coverage is described as inherently incomplete, with the explicit note that no automated tool detects every technique, particularly novel or zero-day attacks. False positives and false negatives can occur, and no configuration eliminates all incorrect classifications.
Beyond the stated limitations, the architecture implies a few more. Because the scanner reads files rather than running them, a skill that assembles malicious behaviour at runtime from benign-looking fragments can pass every static layer. The LLM judge is optional, so a default configuration without it loses the semantic layer entirely. The CEL layer, the component that could suppress findings, is in shadow mode across all bundled rules, which means its suppression logic is being observed rather than applied. And the recall figures above mean the wrong tool for anyone who wants a scanner to certify a skill as safe. It is also the wrong tool if your skills are not in a supported format and you cannot use --lenient, since the scanner will not see them at all.
What it replaces, and what it does not
The obvious alternative is a general-purpose SAST tool such as Semgrep or Bandit pointed at the skill directory. The difference is the unit of analysis. A general SAST tool reasons about code in a language and has no concept of a skill as a package with instructions, a manifest, and a trust boundary. Skill Scanner's rule packs and threat taxonomy (docs/architecture/threat-taxonomy.md) are written against prompt injection and exfiltration patterns specific to agent skills, which a generic linter will not flag because the offending content may be prose rather than code.
The trade-off runs the other way too. A mature SAST tool has years of language-specific taint tracking and a large rule ecosystem. Skill Scanner's dataflow analyzer is one component of a young project, and the benchmark numbers above describe its current state, not a settled one. Running both is defensible: a general SAST pass for the code the skill ships, and Skill Scanner for the skill-shaped risk around it. What Skill Scanner does not replace is manual review of high-risk skills, which the README states should be paired with scanner results for production deployments.
Maintenance cost, licence, and what to verify first
Release cadence is active: 2.0.13 on 2026-08-03, 2.0.14 on 2026-09-01, and 2.1.0 on 2026-09-05, with the last push to main on the same day as 2.1.0. Three releases in roughly five weeks means you should expect to track versions rather than pin once and forget. The CEL runtime is pinned to cel-go v0.32.0, so upgrading the scanner may move that dependency.
Licensing needs a caveat. The repository metadata reports NOASSERTION, while the README carries an Apache 2.0 badge. Those two signals disagree, and the badge is not the licence file. If you are adopting this inside a company with licence review, read the LICENSE file in the repository and get your own answer rather than relying on either the badge or the metadata field. Nothing here is legal advice.
Before you wire it into a pipeline, three things are worth checking against your own material. First, run it on your skill corpus and compare the findings to what a human reviewer would have flagged, because the published recall figures are corpus-specific. Second, read docs/development/detection-evaluation-rollout.md, which the README cites for methodology, provenance, confidence intervals, and limitations, and confirm whether the promotion gate result affects the rule packs you intend to enable. Third, decide explicitly whether you are running the LLM analyzer, since leaving it off removes a detection layer that the benchmark results depend on.
Editorial conclusion
Adopt skill-scanner if you already review Agent Skills by hand and want a repeatable first pass that emits SARIF into GitHub Code Scanning or runs as a pre-commit hook. Do not adopt it as an approval gate: with every bundled CEL rule in shadow mode and a source-disjoint F1 of 13.74 percent, a clean scan is not evidence of safety. Before wiring it into CI, run it against your own skill corpus and read docs/development/detection-evaluation-rollout.md to confirm which rule packs and rollout modes your policy actually activates.
Community notes