agent-qa: a natural language test harness with execution memory

The problem agent-qa targets: tests that break on UI drift

End-to-end suites fail for two different reasons. A real regression, and a selector that moved. agent-qa is aimed at the second category. The README describes the harness as self-healing: when any sub-action such as click, fill, or select fails, agent-qa re-observes the UI and tries a different path in the same run. That is the core claim, and it changes what a test file has to contain. Instead of binding to a CSS selector or a test id, the test states an action in human language and the agent resolves it against visible roles, labels, and screen state at run time. The audience is software teams already running browser or mobile automation who spend maintenance time on broken locators rather than on writing new coverage. It is not a unit test framework and it does not replace type checking or contract tests. It sits at the top of the pyramid, where the assertions are about what a user can see and do.

How the memory loop actually closes

The mechanism the README describes has three stages. First, the agent plans an action against the current screen. Second, if a sub-action fails, it re-observes and picks another path. Third, and this is the part that distinguishes it from a retry wrapper, the harness records what happened. The README states that with every test run agent-qa builds execution memory from product, suite, and test observations, and adds that context to future runs, and that it curates memory from steps that were healed during execution. So a healed click is not just a one-off recovery, it becomes a hint for the next run. A separate action cache reuses validated plans across similar subsequent runs, which the README frames as reducing planner work, token usage, and runtime overhead. Read together, there are two persistence layers: a cache of plans that worked, and a memory of observations that shape future planning. The README also says tests, configs, hooks, memory, and suite logic all live as version-controlled code. That means the memory is a diffable artifact, not an opaque service-side store. Whether that is good depends on your review culture. A memory file that grows on every run will show up in pull requests, and someone has to decide whether a change in it is noise or a signal that the UI moved.

Getting it running: the commands the README gives

The install path is npm. The README shows npm install -D agent-qa, and for Codex or Claude Code subscription auth an additional npm install -D @vostride/agent-qa-subscription-auth. Initialization is npx agent-qa init. Browser support is installed separately with npx agent-qa install-browsers --chromium, and mobile projects use npx agent-qa install-mobile-drivers --all. There is a dashboard, started with npx agent-qa dashboard --open, which the README describes as the place to complete auth and run tests from the UI. The CLI path is a single command against a test file: npx agent-qa run tests/hacker-news-top-story.yaml. Two things stand out in that list. Tests are YAML files, not TypeScript modules, which keeps the natural language assertions readable but moves any real programming into hooks. And authentication is a setup step you perform once, either through the dashboard or through the subscription-auth package. The README does not show the shape of the YAML, the configuration keys, or an example memory file. Those live in the linked docs at vostride.com, which I have not read. If you are evaluating this, the configuration page is the first thing to open, because model selection, endpoint configuration, and cache behaviour are all going to be config keys rather than CLI flags.

Hooks run in Docker, and that is a hard dependency

The hook system is the most concrete engineering decision in the README. Hooks can be written in Node, Bun, Python, or Bash, and they run in isolated Docker containers to set up environments, call APIs, seed fixtures, tear down state, or pass structured outputs back into the active test run. The isolation is real: a hook cannot reach your host filesystem or your shell environment unless you give it a way in. The cost is also real. Docker is required, and the README says so plainly, telling you to install Docker before using hooks. That rules out a plain CI container without a Docker socket, and it rules out most hosted runners that do not allow nested containers. If your test pipeline runs inside a locked-down environment, the hook feature is unavailable and you are limited to whatever the agent can do through the browser alone. Seeding a database or minting a session token for a logged-in journey usually needs a hook. So the practical question is not whether the agent can drive a browser, it is whether your CI can start a container per hook invocation.

Where agent-qa is the wrong tool

A model in the loop makes every run non-deterministic in a way that a conventional runner is not. Two runs of the same YAML file can resolve the same natural language step through different paths, and the memory layer means run ten is not the same as run one. That is the design, and it is also the failure mode. If a step is ambiguous, the agent may heal it into a path that passes while testing something you did not intend, and the memory will then reinforce that path on later runs. The README's own framing, that memory is curated from steps that were healed, cuts both ways: healing that masks a genuine regression becomes a stored habit. There is no mention of a strict mode, a locked plan, or a way to forbid healing on a given step. Teams with regulatory or audit requirements that demand a reproducible, reviewable pass/fail with no model inference in the decision should look elsewhere. The same applies to anyone whose flakiness is in the application itself rather than in the locators, because self-healing will paper over a race condition instead of surfacing it. And the licence is listed as NOASSERTION on the repository, with a LICENSE.md and a NOTICE.md referenced from the README. That is not a licence you can plan around until you read those two files. I have not read them, so I cannot tell you the terms, and neither should anyone else guess.

The comparison that matters: Playwright with a healing layer

Playwright is the obvious baseline, and the difference is architectural rather than cosmetic. A Playwright test is a program. The locator strategy is written down, the assertion is written down, and the runner either matches or it does not. agent-qa inverts that: the test is a description, and the resolution happens at run time against the live DOM, with a planner deciding which element satisfies the description. Playwright's own answer to churn is the role and label based locator, which is stable as long as the accessibility tree is stable, and it fails loudly when it is not. agent-qa's answer is to try another path and remember it. The trade is determinism for survival. If your team already maintains accessible markup and your suite fails because a designer renamed a button, Playwright locators plus a code review will fix that in one commit. If your suite fails because a third-party widget renders differently on Tuesdays, agent-qa's re-observation is doing work that a locator cannot. There is also a scope difference: agent-qa ships mobile drivers and a dashboard, which Playwright does not, so the comparison is only clean for the web half. For the mobile half, the relevant baseline is a driver-level framework plus your own retry logic, and the README does not give enough detail on the mobile path for me to compare it properly.

Maintenance cost and the licence question

There are no releases retrieved for this repository, which means the versioning story is the npm package version rather than a changelog you can read on the releases page. The last push date is recent, so the project is active, but active is not the same as stable, and a harness that runs a model on every step will need to track upstream model and endpoint changes. Budget for that: model deprecations, endpoint behaviour shifts, and prompt-level regressions are all things you will have to debug from the harness side. The memory directory adds a second maintenance surface. It is version-controlled, so it accumulates, and someone has to prune it when a healed path is no longer valid. The README does not describe a pruning command or a memory size limit. On licensing, the repository metadata reports NOASSERTION, and the README links to LICENSE.md and NOTICE.md. The NOTICE file suggests there may be third-party attributions to carry, which matters if you redistribute the harness inside a product. Read both files before you put this in a commercial pipeline. That is not legal advice, it is just the two files the README points at.

Editorial conclusion

Adopt agent-qa if your suite is a set of user journeys that break on selector churn and you can accept a non-deterministic runner whose memory lives in git. Do not adopt it if you need a signed licence, a reproducible pass/fail with no model in the loop, or a fixed browser target that Playwright already covers. Before committing, run npx agent-qa install-browsers --chromium, open the dashboard, and check whether the memory directory the harness writes after the first run is something your reviewers will accept in pull requests.

Official sources

1. Issues
2. Project website
3. README
4. vostride/agent-qa on GitHub