Waza CLI for Testing and Comparing AI Agent Skills

What Waza Solves for Agent Skill Authors

Waza addresses the gap between authoring an agent skill and knowing whether that skill works across tasks and models. An AI agent skill commonly centers on a `SKILL.md` file, but the file alone does not establish whether the agent follows the intended procedure, produces useful output, respects task constraints, or remains consistent after a prompt changes. Waza gives that problem a CLI workflow: it scaffolds skills and evaluation projects, creates tasks and fixtures, executes evaluations, grades saved output, and compares result files.

The intended users are engineers, evaluation specialists, and teams maintaining reusable agent instructions. It is also relevant to repository owners who want to turn an informal prompt into a tested skill definition. The repository contains examples for code explanation, custom agents, rubrics, repository resources, required skills, scenarios, controlled model comparisons, and preflight checks. That range suggests Waza is designed for both individual skill maintenance and repository-level regression workflows.

Waza does not replace the agent that performs the task. Instead, it sits around evaluation execution and analysis. The README describes executions against skills and custom agents, named model comparisons, shared graders, specification verification, token counting, snapshots, and adversarial packs. This makes it closer to a test and measurement framework than to an autonomous agent framework. The distinction matters when selecting it: Waza helps answer whether an existing or proposed agent workflow meets defined checks, rather than supplying a complete application runtime for every agent action.

Waza's Evaluation and Grading Pipeline

A normal Waza workflow begins with an evaluation specification. Commands such as `waza run eval.yaml` and `waza grade eval.yaml --results results.json` establish two visible stages: execution against tasks and grading of generated results. Tasks and fixtures live under the initialized project, while an evaluation file ties the suite together. For example, `waza preflight` accepts an evaluation file and a context directory, which indicates that local fixture material can be supplied as task context before execution.

The repository is organized as a Go CLI under `cmd/`, with implementation packages under `internal/`. Its dependency on `github.com/spf13/cobra` fits the command hierarchy shown in the README, where `init`, `new`, `run`, `grade`, `compare`, and other commands operate on projects and files. Schemas under `schemas/` provide structured contracts for project artifacts, while the `web/` tree and embedded dashboard assets indicate that the binary can also expose evaluation-related visual output.

Grading is not limited to a generic pass or fail result. Waza exposes shared registry graders, evaluator-only references, rubrics, specification verification, and an `assure` command intended to challenge native file and output checks offline. The README also documents `waza spec verify`, which compares a `SKILL.md` with an evaluation file. That is useful when an evaluation suite has drifted away from the requirements stated by the skill.

Results can flow into separate analysis commands. `waza compare` accepts two result files and is intended for model comparison. Snapshots add another path: `waza run --snapshot` captures execution data, and `waza replay` inspects it offline. Offline grading and replay are useful for debugging or CI, but they do not measure changes in the live model service. The README explicitly cautions that recorded or mocked outcomes are not evidence of live agent quality.

Installing Waza and Creating the First Evaluation

The recommended installation path uses the project's shell installer. It supports macOS, Linux, and Windows Bash environments, including Git Bash, MSYS2, and Cygwin. The script selects a standalone binary, verifies its checksum, and installs it in `/usr/local/bin` or `~/bin` when the former is not writable.

Run this installer on macOS or Linux:

curl -fsSL https://raw.githubusercontent.com/microsoft/waza/main/install.sh | bash

The documentation says the installer detects Linux, macOS, or Windows and amd64 or arm64 architecture. A native Windows PowerShell user can instead download and run the PowerShell installer:

irm https://raw.githubusercontent.com/microsoft/waza/main/install.ps1 | iex

The Windows scripts and Bash installer execute code fetched over the network, so readers who require a controlled software supply chain should inspect the script or use a GitHub Releases asset instead. After installation, initialize a project and create a skill:

waza init my-project && cd my-project
waza new skill my-skill

This should create the project structure used by the quick start. The skill definition belongs in `skills/my-skill/SKILL.md`. The README then directs the user to add evaluation tasks under `evals/my-skill/tasks/` and fixtures under `evals/my-skill/fixtures/`.

Once those files exist, run the evaluation and its readiness check:

waza run my-skill
waza check skills/my-skill

The first command executes the skill evaluation. The second checks whether the skill is ready for submission. If starting from an existing skill definition rather than a generated example, `waza new eval skill-name` creates an evaluation scaffold from `SKILL.md`.

Source Builds, Copilot Assets, and Container Use

Installing from source has stricter prerequisites than installing a release binary. The README requires Go 1.26 or later, Make, Node.js, and npm. It also says `go install` cannot be used because the project contains Git LFS artifacts. A source build clones the repository, retrieves those artifacts, builds the dashboard, and embeds the generated assets in the Go binary.

Use the documented source-build sequence:

git clone https://github.com/microsoft/waza.git
cd waza
git lfs install
git lfs pull
make build
./waza --version

The last command was not given as an exact README command, so it should not be treated as required project syntax. The documented Makefile target is `make build`; the binary can then be invoked with the actual Waza command required by the workflow.

The `copilot-sdk` executor has another installation boundary. Waza bundles the GitHub Copilot CLI and extracts it, along with runtime assets, into a versioned directory in the user's local cache on first use. `COPILOT_HOME/cache` can override that location, while `COPILOT_CLI_PATH` forces a specific Copilot CLI binary. If installation fails, Waza reports the failure rather than silently switching to a `PATH` executable. `COPILOT_CLI_INSTALL_VERBOSE=1` enables installation diagnostics.

A Dockerfile is included for CI or container environments. It builds the web dashboard with Node.js, compiles the Go binary in a Go 1.26 Alpine builder, and copies the result into an Alpine 3.19 runtime image with CA certificates. The default container command is `waza --help`, so users must supply a Waza subcommand when invoking it for evaluation work. The image does not by itself define model credentials or evaluation files.

Snapshots, Preflight, and Failure Boundaries

Waza provides several controls for reproducing or investigating evaluation behavior, but each has a defined scope. `waza preflight` is intended to validate an evaluation and its context before a run. `waza spec verify` checks evaluation coverage against a skill, and `--fail --format github-actions` turns failures into a format suitable for GitHub Actions. `waza migrate eval.yaml` checks whether a schema artifact requires migration, which can prevent an old suite from silently using outdated fields.

Snapshots address consistency after execution. The documented sequence captures output into a directory and later replays a task snapshot:

waza run eval.yaml --snapshot ./snapshots/
waza replay ./snapshots/my-task-run1.json

Replay is useful for inspecting stored behavior without making the snapshot equivalent to a fresh model call. Model output can vary because of provider behavior, configuration, context, and external state. The repository's warning about recorded outcomes is therefore important: snapshot replay can validate tool and evaluation mechanics, but it cannot establish current live-agent quality.

The bundled Copilot CLI is also a failure boundary. A team using `copilot-sdk` must account for first-use extraction, cache permissions, runtime assets, and platform compatibility. Waza deliberately does not fall back to another Copilot binary on `PATH`, so this failure is visible. That choice reduces accidental executor substitution, but it also means an environment without the expected bundled assets cannot complete that path.

Waza is the wrong tool when the requirement is simply to invoke a single prompt, build an agent application, or guarantee production quality without a representative task suite. A passing `waza check` confirms repository-defined readiness checks, not universal correctness.

Waza Compared with Prompt Testing and CI-Only Checks

A lightweight alternative is a script that sends a few prompts to a model and checks output strings. That approach may be quicker for a prototype, but it usually leaves task construction, fixture loading, grading, result persistence, snapshot handling, and cross-model comparison to the surrounding script. Waza places those operations behind named CLI commands and repository conventions such as `eval.yaml`, `tasks/`, `fixtures/`, `results.json`, and `waza.lock`.

Another alternative is CI-only validation that checks files, schemas, or known commands but does not execute the agent. Waza is more useful when the team needs an evaluation artifact that can be run locally, in automation, and against multiple result files. Commands such as `waza coverage`, `waza tokens count`, and `waza tokens compare` also add repository-level visibility beyond a single test invocation. Coverage can reveal which requirements have tasks, while token comparison can identify skill files whose size differs from a reference branch.

Waza still does not replace a live acceptance process. Its evaluations depend on task quality, grader design, model settings, and context fixtures. An offline adversarial pack can expose weaknesses in defined failure scenarios, but it cannot cover every production request. The framework is consequently best understood as a repeatable measurement layer: more structured than a one-off prompt script, but narrower than a full agent orchestration platform.

Release Cadence, Upgrades, and MIT Licensing

The repository is not archived, and its last push was on 2026-10-15. Recent releases include `v0.38.10` on 2026-10-09, `azd-ext-microsoft-azd-waza_0.38.10` on 2026-10-09, and `v0.38.9` on 2026-10-05. Those facts indicate recent repository activity, but they do not guarantee compatibility between every release and every provider or custom agent configuration.

Waza checks for updates in the background. The README says the check is non-blocking, cached for 24 hours, and can be disabled with `--no-update-check` or `WAZA_NO_UPDATE_CHECK=1`. An explicit update uses `waza update`; scripted environments can use `waza update --yes` to skip the confirmation prompt. The command downloads and executes the official OS-specific installer, so automation should account for network access, installer policy, and the possibility that release behavior changes between versions.

The project uses the MIT license. That permits broad use and modification under the license's conditions, including preserving the license notice and disclaimer. MIT status does not settle questions about model-provider terms, hosted API costs, Copilot CLI distribution, or data handled by an external service. Teams embedding Waza in CI or a product should review those dependencies separately from the repository license.

The source build has a higher maintenance cost than the binary installation because it requires Go, Node.js, npm, Make, Git LFS, dashboard generation, and Copilot-related assets. Release binaries avoid some of that source-build work, but upgrades still change CLI behavior and can affect saved evaluation files.

Editorial conclusion

Adopt Waza when an AI agent skill needs a maintained evaluation suite, structured grading, snapshot replay, or cross-model comparison. Do not adopt it as a general agent runtime or as proof that a skill performs well in live production. Verify first that the evaluation tasks, graders, model configuration, context fixtures, and required Copilot CLI assets match the workflow you intend to change.

Frequently asked questions

Official sources

1. License: MIT
2. microsoft/waza on GitHub
3. Project website
4. README
5. Releases