SkillForge: skill creation that measures behaviour instead of documents

Quality as a property of behaviour

SkillForge's stated core principle is that skill quality is a property of behavior, not documents, and that a skill is done when a fresh agent demonstrably does better with it than without it.

That sentence reframes the whole category. Most tooling for writing agent skills produces a document, and the review of that document is where quality is argued about. The review is subjective, the reviewer has to hold the intended behaviour in their head, and a document that reads well can still fail to change what the agent does.

SkillForge replaces the argument with an experiment. Before a skill is written, a fresh subagent attempts the task without it. If the subagent succeeds anyway, there is no skill to create, which is what Phase 0b, the RED GATE, is for. After the skill exists, the same task is run with the skill present, and the with-skill run has to clear the failures recorded in the baseline. Only then does the work move on to review and shipping.

The tool routes any skill-related request, whether to use, improve, create or compose a skill, and it also maintains the health of the whole skill ecosystem. The audience is therefore a team that has already adopted coding agents seriously enough to have a roster of skills and a problem with how they overlap.

The most recent release, v6.0.0, is described as a ground-up rework following a deep external audit, and that audit ships in the repository as SKILLFORGE_AUDIT.md.

The pipeline and its two gates

The pipeline is documented as a sequence of phases:

Phase 0  TRIAGE      index + word-boundary matching -> USE | IMPROVE | CREATE | COMPOSE | CLARIFY
Phase 0b RED GATE    fresh subagent attempts the task WITHOUT the skill; no failure = no skill
Phase 1  ANALYSIS    load-bearing lenses (Inversion, Pareto, Root Cause), failure-form matching
Phase 2  SPEC        tiered (minimal default / full for infrastructure), decisions + WHY
Phase 3  GENERATE    fresh-context subagent receives ONLY the spec + baseline failures
Phase 4  GREEN GATE  with-skill runs must clear the recorded baseline failures; trigger tests
Phase 5  REVIEW      lint (validate_skill.py) + ONE adversarial reviewer charged to refute
Phase 6  SHIP        with evals/ - a per-skill regression suite runnable forever after

Three design choices stand out.

The first is the RED GATE before any writing. It is not a review step, it is a precondition. If a fresh agent can already do the task, the pipeline returns CLARIFY and no skill gets written. That is a claim a lot of skill tooling would be better without, because most of it would have created a duplicate of something that already works.

The second is the GREEN GATE, which is the same task with the skill present, and the requirement that it clears the recorded baseline failures rather than merely looking better.

The third is the review phase. Instead of a panel that reads the skill and agrees, the README describes one adversarial reviewer charged to refute, with the falsifiable checks moved into the lint script validate_skill.py. That is a deliberate split between what a script can check and what only a reading model can judge, and it is the change the v6 comparison table highlights: v5 had a four-agent unanimous synthesis panel reading the skill, and a self-scored timelessness rating of at least seven for approval.

The generation step also constrains the generator: the fresh-context subagent receives only the spec and the baseline failures, not the original conversation, so the skill is written from the requirement rather than from the history of the requirement.

Deduplicate before creating anything

The first thing the pipeline does is look at what already exists. Phase 0 uses an index of every skill across runtimes, matched on word boundaries, to decide whether a request is really a use, an improve, a create, a compose or a clarify.

The README lists this as the first of the things no other skill creator has, and the reason is practical. Once a team has thirty skills written by different people at different times, the most common failure is not a missing skill, it is a fourth overlapping one. An index that answers should this exist first is the cheapest possible defence.

The index itself changed in v6. The comparison table says the v5 index missed the Claude Code plugin cache entirely, and v6 has cross-runtime discovery covering personal, Codex and the Claude Code plugin cache, deduped, with auto-refresh.

The two opt-in features are the ones with the clearest privacy dimension. Friction mining, through mine_skill_friction.py with the --consent flag, finds skill gaps in your own local session history. The v5 behaviour it replaced was personal-directory scanning on by default with hardcoded GitHub handles, and v6 makes personal context strictly opt-in with recorded consent and no shipped defaults.

The proactive advisor is the other one. In v5 it was a launchd background advisor that analysed the root directory and queued suggestions into a file nothing read. In v6 it is delivered through Claude Code hooks on SessionStart and UserPromptSubmit, with caps enforced, opt-in, and never auto-invoked.

Both changes are the same lesson: a tool that watches your machine has to be asked, and advice that nobody reads is not advice.

Skills ship with their own regression tests

The most transferable idea in the project is that a skill is packaged with its tests. The evals/ directory holds trigger queries and behavioural scenarios, and run_skill_evals.py executes them.

That changes what a skill is. Instead of a markdown file in a directory that someone occasionally edits, a skill becomes a unit with fixtures. The eval suite is the record of what the skill was supposed to fix, which means a later edit that breaks the behaviour has something to fail against.

The scaffolding tool takes this into account from the start: init_skill.py scaffolds a new skill and includes evals/. And the CI instruction is a copy operation: copy assets/templates/github-workflow-skill-ci.yml into the .github/workflows/ directory of any skill repository. That is the whole deployment story for continuous integration, and it is deliberately a template you copy rather than a package you install.

The doctor is the other half of the maintenance story. skillforge_doctor.py produces an ecosystem health report that finds trigger collisions between skills, duplicates, stale file references, budget violations and pinned models across the whole roster. Trigger collisions are the failure mode worth noticing, because two skills whose descriptions overlap will have the agent choosing between them unpredictably.

Together, the evals and the doctor are the argument for treating a skill roster as a codebase. It has unit tests, it has a linter, it has a dependency graph, and it can rot.

Installing it, and the toolbox it ships

The install path assumes Claude Code. The README clones the repository into a temporary directory, copies it into the skills directory, and then removes the parts that should not ship with the skill, including the README, the licence, the context file, the docs, the git directory, the assets and the tests, plus the audit document:

git clone https://github.com/tripleyak/SkillForge.git /tmp/skillforge
cp -r /tmp/skillforge ~/.claude/skills/skillforge
cd ~/.claude/skills/skillforge && rm -rf README.md LICENSE CONTEXT.md docs .git .gitignore .skillignore index.html assets/images scripts/tests SKILLFORGE_AUDIT.md
cp /tmp/skillforge/commands/skillforge.md ~/.claude/commands/skillforge.md   # optional /skillforge command

The optional advisor and hooks are installed separately:

python3 ~/.claude/skills/skillforge/scripts/install_skillforge.py

Requirements are Claude Code or Codex CLI, and Python 3.8 or later with the standard library only, with PyYAML used if it happens to be present.

The toolbox itself is a set of scripts you run directly:

python3 scripts/discover_skills.py
python3 scripts/triage_skill_request.py "<request>" --json
python3 scripts/validate_skill.py <dir>
python3 scripts/run_skill_evals.py <dir> [--live]
python3 scripts/skillforge_doctor.py

There are also init_skill.py to scaffold, compile_skill.py to cross-runtime compile with a target of claude, codex or agentskills, package_skill.py to produce a .skill package into a dist directory, and mine_skill_friction.py with --consent.

One thing the comparison table makes clear is the scale of the change in v6. A five thousand word SKILL.md with details-block progressive disclosure became a 1,158-word SKILL.md, with depth moved into references/ that are loaded on demand. The reason given for the description change is worth remembering: descriptions became trigger conditions only, because workflow summaries make agents skip the body.

MIT, the release line and what you cannot verify

The licence is MIT, with the LICENSE file at the top of the tree, and the licence field on the repository says the same.

The release history shows a fast-moving project with one significant rework. v6.0.0, titled Evidence-Driven Skill Creation, was released on 2026-07-29, preceded by v5.1.0 for Compatibility and Safety Hardening and v5.0.0 for a Context-Efficient Redesign, both on 2026-02-25. The repository is not archived and the last push was on 2026-07-29, the same day as the v6 release.

That means the code you install is a recent major version with a documented rationale, and that the rationale is unusually available. The README says the audit that drove the rework ships in the repository as SKILLFORGE_AUDIT.md, and the v6 comparison table sets out the before and after for each design decision. Projects that rework this way often leave the reasoning behind, and this one does, which makes the design arguable rather than merely announced.

What you cannot verify from the outside is the central claim. The pipeline is supposed to show that a skill improves behaviour, and the evidence for that is a before and after run in your own environment with your own tasks. There is no published benchmark in the README, and there could not be, because the claim is relative to the agent model, the task and the baseline. Treat it as a method to run on your own roster rather than as a performance figure.

The other documented change worth flagging is the parser. v5 shipped a hand-rolled YAML parser that failed on SkillForge's own SKILL.md, and v6 has one shared typed parser in scripts/frontmatter.py with more than a hundred unit tests, plus a regression test in which SkillForge validates itself. A project that ships its own failing case as a fixed regression test is telling you something about how it intends to be maintained.

Editorial conclusion

SkillForge suits a team already writing agent skills for Claude Code or Codex who has noticed that skill quality is hard to argue about and wants the argument settled by a before and after run rather than by review. It does not suit a team with no skills yet, since the pipeline assumes an existing roster to deduplicate against, and it does not suit a team that cannot run a subagent in CI, because the gates are behavioural. Before adopting, install it into a scratch skills directory, run scripts/discover_skills.py and scripts/skillforge_doctor.py against your real roster to see what the collisions and duplicates look like, and read SKILLFORGE_AUDIT.md, since the v6 design is the direct answer to that document.

Frequently asked questions

Official sources

1. Issues
2. License: MIT
3. README
4. Releases
5. tripleyak/SkillForge on GitHub