AgentOps: A Judgment Layer for Coding Agents That Cannot Grade Their Own Work

The gap AgentOps targets: an agent grading its own homework

Most coding-agent workflows end when the agent says the work is finished. The same context that wrote the code decides whether the code is correct, which means the failure mode is not a wrong answer but an unexamined one. AgentOps inserts a separate step. According to the README, a fresh context reads the exact change and returns PASS, FAIL, or NOT_PROVEN. The third value is the interesting one. A binary pass or fail forces a reviewer to guess when evidence is missing; NOT_PROVEN gives the reviewer a place to stand when the change cannot be judged from what was supplied.

The target user is an engineer already running a coding agent inside a repository that has a tracker and a Git history. The README is explicit that AgentOps does not replace either: your tracker keeps the work, Git keeps the history, and the coding agents keep executing. AgentOps joins them as a federated integration graph and adds the judgment step. That framing rules out a large class of users. If you want a single pane of glass that owns your tickets and your branches, this is not that product. If you want the missing review gate between an agent's claim and a merge, the scope is deliberately narrow.

One RPI traversal: charter, on-demand plan, implement, fresh validate

The standard path is a single traversal the README writes as: RPI charter, on-demand Plan, Implement and checks, fresh Validate, finish. The charter is the controlling artifact. Per the README, the lean RPI charter owns an authorized outcome through finish, plans on demand, repairs understood failures directly, and runs cheap checks plus a fresh final judgment. Evidence can revise the approach as long as the outcome and scope stay unchanged. A clear small edit needs no planning or memory worksheet at all, which matters because the alternative design (mandatory planning for every change) makes the process cost more than the edit.

Intent is stored rather than remembered. The README states that Plan writes BDD acceptance criteria and DDD ubiquitous language into a bead, and that Implement builds against it. Validate then judges a hashed snapshot under .agents/ao/intents/sha256/. That path is the mechanism: the intent is frozen as bytes, hashed, and the verdict is rendered against that hash rather than against a description of the intent. If you do not use beads, Plan shapes the caller's issue or chat text and the runtime snapshots those bytes the same way. The tracker is optional; the snapshot is not.

The default reviewer is same-family. Validate runs in a fresh context from the author's model family, so Codex reviews Codex work and Claude reviews Claude work. Cross-family review requires an explicit --cross-model [model] flag in Validate or RPI, and the README notes that an unavailable requested leg leaves the combined result unproven. That is a defensible default: same-family review is cheap and catches most mechanical errors, while cross-family review is opt-in because it costs a second provider. It is also a real limit, since a shared blind spot between author and reviewer survives the default path intact. Review time is bounded by the caller or the native runtime, and the README states there is no fixed ten-minute cap.

Install paths and the ao dependency the skills declare

The universal install is one command: npx skills@latest add boshu2/agentops --all -g. The README says this installs the skill bundle into every coding agent you use, and the skills then run inside that agent. You invoke them by typing /rpi in the agent's chat, or by asking for plan, implement, validate, and learn by name.

For a managed bundle that tracks releases, the README gives marketplace commands. In Claude Code: claude plugin marketplace add boshu2/agentops, then claude plugin install agentops@agentops-marketplace. In Codex: codex plugin marketplace add boshu2/agentops, then codex plugin add agentops@agentops-marketplace. A third path exists for contributors: checkout plus ao skills link, which the README describes as source-tracked symlinks, documented in docs/install-day2-ops.md. The three paths differ in a way that matters. The npx route copies skills you can edit; the plugin route is a read-only bundle that stays current with the repo; the symlink route is for people changing the skills themselves.

Not every skill is self-contained. The README carries a table of skills that need more than the coding agent. The rpi skill needs ao conditionally, because it delegates exact-subject checks to Validate and only persists verdict.v2 when requested. validate needs ao to derive exact subject identity and to run ao provenance store-verdict when persistence is requested. fitness is described as an entire procedure that runs one ao goals subcommand. using-gc runs ao gc prepare and ao gc check. reverse-engineer needs python3 because its first phase runs scripts/reverse_engineer.py. The README also states that the plugin and the npx command install all 54 skills today regardless of whether you have python3 or ao, so a missing dependency surfaces at invocation rather than at install. Budget for that: a skill that installs cleanly can still fail on first use.

Admission-control hooks and what the policy dispatcher actually blocks

AgentOps ships a PreToolUse policy dispatcher that is on by default. The README describes it as deterministic guards that block a small set of known-destructive commands and route the caller to the correct tool instead. The specific blocks listed are staging the private bead ledger, hand-editing the hash-chained provenance ledger, and overwriting installed skill copies. The design claim is that it is silent on every clean call and that every block is one line.

Activation depends on how you installed. Claude Code plugin installs have it active automatically. For npx or skills.sh copies you run ~/.claude/skills/cc-hooks/scripts/install-hooks.sh once. For a git clone or brew install you run scripts/install-policy-dispatch.sh once. Disabling is possible either through /plugin disable agentops or by removing the two PreToolUse matchers from settings. The policy list lives in skills/cc-hooks/SKILL.md.

Two things are worth stating plainly. First, the guard set is small by design, so this is not a security boundary against a hostile agent; it is a guardrail against a careless one. Second, the hooks are runtime-specific. If your agent is not Claude Code, the README does not describe an equivalent automatic activation, and the install scripts it names write into ~/.claude and a local scripts directory. The portability claim applies to the skills; the hook layer is more tightly coupled to one runtime.

The single-writer default and the cost of switching to a fleet

The README states the default is one agent, one writer. Multi-agent work is a separate configuration: swarm, agent-native, ntm, and using-gc orchestrate it, and the README adds a constraint that matters more than the list. They dispatch; they do not own the verdict. The judgment step stays with Validate even when four skills are coordinating the execution.

That separation is the design's strongest idea and also its main coordination cost. If you run a fleet, you now have two things to reason about: which agent produced which change, and which fresh context judged it. The exact-subject identity that validate derives with the ao helper is what ties the two together. Where the README is thin is the fleet case itself. It names the skills and the boundary but does not describe, in the material available, how conflicting dispatches are resolved or how a verdict on one subject interacts with a partially completed sibling change. Treat fleet mode as documented at the level of intent, not at the level of failure handling.

Memory is scoped the same way. The README describes on-demand recall plus separately budgeted mining and curation over reviewed caller-selected external Markdown topic pages, with the rule that you update an existing page and preserve support, limits, and invalidation. Two statements in that section deserve attention. Learning may remove rules, so the memory store is not append-only. And saved pages do not prove benefit; only later task evidence does. That is an unusually honest constraint to put in a README, and it means the memory feature should be judged by downstream task outcomes rather than by how much it accumulates.

Where AgentOps is the wrong tool, and what to compare it against

The README rules out several cases itself. The lean path uses public or already-cleared inputs and claims no native restricted-source enforcement. If your work involves protected external drafts, the README points to ADR-0016 for state tiers, review before Git, and legacy evidence preservation, which tells you this is a routing and process question rather than a toggle. If you need the tool to enforce source restrictions for you, it does not claim to.

The other boundary is the software factory. The README states that AgentOps supplies skills and evidence contracts, not another software factory. So the honest comparison is not against a competing review tool but against the review process you already have. A pull request plus a human reviewer plus CI produces a verdict too, and it is a verdict from a context that did not write the code. The difference in approach is what gets frozen. In a conventional PR review, the reviewer reads a diff and a description; the description is prose and can drift from the change. AgentOps hashes the intent into .agents/ao/intents/sha256/ and renders the verdict against that hash, which makes the reviewed subject exact rather than approximate. The trade is that you must keep the intent artifact alive and coherent, which is work a human reviewer absorbs informally.

A second comparison is against orchestration frameworks that own both dispatch and completion. Those give you a single completion signal, which is simpler to wire into CI. AgentOps deliberately splits dispatch from judgment, so you get a verdict object you have to route yourself. If your pipeline wants one boolean, the split is friction. If your pipeline has been burned by a completion signal that nobody could audit, the split is the point.

Maintenance, versioning, and the Apache-2.0 boundary

The repository is active, not archived, with a last push of 2026-09-10 and releases at roughly two-week intervals across v3.4.0, v3.5.0, and v3.6.0 in the July and August 2026 window. The release cadence matters for the plugin install path specifically, because the README describes the plugin bundle as read-only and staying current with the repo. The npx path copies skills you can edit, which means local edits survive but do not track upstream. The symlink path tracks source and is aimed at contributors. Pick the path that matches how much you intend to modify, because switching later means re-resolving where your customizations live.

The upgrade surface is larger than a single binary. There are skills, a Go tool (ao), Python scripts invoked by several skills, hook installers, and a verdict schema the README calls verdict.v2. A version bump can move any of those. The schema name carries a version suffix, which is a signal that the verdict shape is expected to change across major lines; if your CI parses verdict.v2, pin the release and read the notes before moving.

The licence is Apache-2.0, which permits commercial use and modification and includes an explicit patent grant. It also requires that you preserve copyright and licence notices and state significant changes in modified files. That last clause is the one teams forget when they fork the skills and edit them in place. This is a description of the licence text, not legal advice; if you redistribute a modified bundle, have counsel read the notice requirements against your distribution model. Note also that the README references external projects, including beads under a separate install, and their licences are their own.

A concrete adoption test before you commit

The README asks for feedback in a specific form: run it, then open an issue and paste the verdict.v2 if you asked validate to persist one. That is a usable acceptance test for your own evaluation, and it exercises the parts most likely to break. Install via npx skills@latest add boshu2/agentops --all -g, confirm ao is on PATH, run /rpi on a small change, and request persistence so a verdict artifact lands on disk. Then read the artifact. If the verdict is NOT_PROVEN, the reason is the useful output, because it tells you which evidence your process is not capturing.

Do this on a change you already understand. The point is not to see whether the agent can code; it is to see whether the judgment step produces a verdict you would act on. If the verdict is indistinguishable from what your existing reviewer would have said, the extra artifact is overhead. If it catches a case where the author's context was wrong about its own change, the hashed-intent path is earning its keep. Either result is a fact you can carry into the decision, which is more than a feature list gives you.

Editorial conclusion

Adopt AgentOps if you already run Claude Code, Codex, or Cursor and want a hashed intent snapshot plus a fresh-context verdict sitting between implementation and merge. Do not adopt it if you need a hosted dashboard, a fleet-first orchestrator, or a tool that enforces restricted-source handling, because the README states the lean path claims no native restricted-source enforcement. Verify three things before wiring it into a pipeline: that ao is present and on PATH for the skills that declare it, that the verdict.v2 schema your CI parses matches what ao provenance store-verdict writes, and that your tracker is either beads or a durable source the runtime can snapshot into .agents/ao/intents/sha256/.

Official sources

1. boshu2/agentops on GitHub
2. License: Apache-2.0
3. Project website
4. README
5. Releases