Passmark: AI-driven Playwright steps, cached and cross-checked
The open-source Playwright library for AI browser regression testing with intelligent caching, auto-healing, and multi-model verification.
At a glance
- What is it?
- Passmark wraps Playwright tests in natural-language steps executed by AI models, with Redis step caching, auto-healing, and multi-model assertion checks. It is a good fit for teams already fluent in Playwright who want to reduce selector churn, and a poor fit for anyone unwilling to send test traffic through hosted model APIs.
- Who is it for?
- Adopt Passmark if your team already writes Playwright specs and the recurring cost is flaky selectors rather than missing coverage. Skip it if you cannot send test traffic to Anthropic, Google, OpenAI, or a gateway, or if your CI must run without external model calls.
- Can I use it commercially?
- Check first. The repository uses a licence we do not classify automatically, so read its LICENSE file before any commercial use.
- Is it still maintained?
- Yes. The repository last received commits 4 days ago.
- What is it written in?
- Mainly TypeScript, according to GitHub's language statistics.
Answers come from the project's GitHub data, last synced on September 15, 2026, and from our analysis. They are not legal advice.
DEEP OPEN-SOURCE ANALYSIS
The selector churn problem Passmark targets
Playwright tests break for boring reasons. A button gets a new class, a wrapper div appears, a test id is renamed, and a spec that passed yesterday fails today even though the product works. The usual fixes are page objects, role-based locators, and patience. Passmark takes a different route: instead of asserting against a selector, the test states what should happen in plain language and lets a model decide which element to act on.
The README frames this as covering browser regression testing end-to-end and catching regressions early. The intended user is a QA or frontend engineer who already runs Playwright and wants the test file to survive markup changes without edits. That is a narrower audience than "anyone who wants AI testing". If you have no Playwright suite, Passmark gives you a new one to maintain rather than a shortcut. If your suite is stable and cheap to fix, the model calls add cost without removing much pain.
How runSteps turns descriptions into browser actions
The entry point is a single function, runSteps, imported from passmark alongside test and expect from @playwright/test. You hand it the Playwright page object, a userFlow string, an array of steps, an assertions array, and the test and expect fixtures. Each step has a description, an optional data object for values such as a color or size, and an optional waitUntil string.
By default the library works from ARIA accessibility snapshots, per the README. That choice matters: the model reasons over the accessibility tree rather than pixels, so it sees roles, names, and structure. A second mode, mode: "cua", switches to OpenAI's computer-use agent, which is screenshot-driven. The README states CUA uses gpt-5.5 plus the built-in computer tool, and that the CUA model is currently locked and not user-configurable.
Assertions are handled separately from steps. The example passes a single assertion string, "You can see My Cart with Acme Circles T-Shirt", and the README describes multi-model assertion verification as a core feature. The mechanism it implies is that more than one provider judges the same assertion, which is why the setup section insists on at least one Anthropic model and one Google model.
Caching, healing, and what the README does not explain
Two mechanisms get named in the project description and only partly explained in the README. The first is intelligent caching. The README mentions Redis step caching and notes it is skipped in CUA mode because coordinate actions are not portable across viewport sizes. That is a sensible constraint and it tells you caching operates on step outcomes, but the README supplied here does not document how the cache is keyed, how entries are invalidated when the page changes, or which Redis environment variables to set.
The second is auto-healing. The description lists it and the README says your tests stay stable without needing to update AI prompts or retrain models. What is missing is the failure path: what happens when healing picks the wrong element, whether the run reports that a step was healed, and whether a healed step can be promoted back to a deterministic locator. Treat both features as real but under-documented until you read the source or the docs site at passmark.dev. A caching layer you cannot inspect is a debugging liability when a test passes locally and fails in CI.
Getting a first test running
The README's quick start scaffolds a project with npm init playwright@latest passmark-project, selecting the default options and TypeScript, then cd passmark-project and npm install passmark. Because the Playwright config must read .env, you add dotenv yourself: npm install dotenv, then in playwright.config.ts import dotenv and path and call dotenv.config with path.resolve(__dirname, '.env').
Keys go in .env. Direct provider access needs ANTHROPIC_API_KEY and GOOGLE_GENERATIVE_AI_API_KEY. If you route through a gateway instead, the README names AI_GATEWAY_API_KEY for Vercel, OPENROUTER_API_KEY for OpenRouter, and OPENCODEZEN_API_KEY for OpenCode Zen. Cloudflare AI Gateway is described as a proxy rather than a reseller, so it needs CLOUDFLARE_ACCOUNT_ID and CLOUDFLARE_AI_GATEWAY, plus CLOUDFLARE_AI_GATEWAY_API_KEY if the gateway has authentication enabled, and it still needs the upstream Anthropic and Google keys.
Gateway selection happens in code, not config files: configure({ ai: { gateway: "vercel" } }), with "openrouter", "opencodezen", or "cloudflare" as the other accepted values. Run a spec with npx playwright test example.spec.ts --project chromium and inspect results with npx playwright show-report, which the README says includes an AI summary supplied by Passmark. The example test also raises the timeout to 60 seconds, which is a fair hint that model round-trips do not fit Playwright's default budget.
Per-step model overrides and the precedence rule
The most interesting design decision in the README is that the same ai object accepted by configure() can also be passed at the runSteps call level and on individual steps. Precedence runs step.ai, then call-level ai, then the global configure(). The stated purpose is hybrid runs: cheap snapshot steps through a gateway such as OpenRouter, expensive visual steps through CUA on direct OpenAI access.
The example shows a slider drag, "Drag the price slider to $40", carrying ai: { mode: "cua", gateway: "none" } while surrounding steps inherit the global OpenRouter setting. Two constraints come with this. CUA requires gateway: "none" because the Responses-API computer tool is only exposed on direct OpenAI access, and the README explicitly says the Vercel, OpenRouter, and Cloudflare gateway values are not compatible with CUA. You also need an OPENAI_API_KEY whose account has access to the CUA model and the computer tool. Mixing modes is therefore a real capability, but it forces you to hold two billing relationships and two sets of credentials.
Where Passmark is the wrong tool
The clearest limitation is environmental. Every run depends on external model APIs. If your CI runs in an air-gapped network, or your security policy forbids sending internal URLs and page content to third parties, Passmark does not fit, and no configuration in the README changes that. The accessibility snapshot of a logged-in admin page is still page content leaving your network.
The second limitation is determinism. A model choosing which element to click is not the same as a locator resolving to one node. When a step fails, the failure may be a model misjudgement rather than a product regression, and the README does not describe a mechanism for distinguishing the two. That ambiguity is expensive during an incident.
The third is cost predictability. The README advises a 60 second timeout for AI execution and offers caching as the mitigation, but caching is skipped entirely in CUA mode. A CUA-heavy suite pays full model cost on every run, on every viewport, because the README says coordinate actions do not survive viewport changes. Finally, the license is listed as NOASSERTION in the repository metadata while the README badge points to FSL-1.1-ALv2. That mismatch is worth resolving before adoption, and I am not in a position to interpret either term.
How it differs from plain Playwright and from visual diff tools
Against plain Playwright, the difference is where intent lives. A standard spec encodes intent as selectors: page.getByRole('button', { name: 'Add to cart' }). Passmark encodes intent as a sentence and resolves it at runtime through a model. You trade determinism and offline execution for tolerance of markup changes. Plain Playwright also gives you trace viewer artifacts and a stable failure taxonomy; Passmark adds a model layer whose failure modes the README does not enumerate.
Against screenshot-diff tools such as Playwright's own toHaveScreenshot, the difference is what counts as a regression. A visual diff flags any pixel change, including a font rendering shift on a different OS, and says nothing about whether the flow completed. Passmark's assertions are semantic: the example checks that "My Cart" is visible with a named product. That is closer to user intent and further from byte-level comparison, but it depends on a model agreeing with you, which is exactly why the multi-model consensus exists. Neither approach replaces the other, and the README does not claim Passmark does visual diffing.
Maintenance, licence, and what to check before adopting
Maintenance cost here is mostly external. Model names, gateway compatibility, and provider APIs move faster than test suites do. The README already documents one hard constraint of that kind: the CUA model is locked and not user-configurable, so a provider-side change is not something you can route around with configuration. The library is TypeScript and the last push recorded in the metadata is 2026-07-30, with no releases retrieved, so pin the passmark version in package.json rather than tracking latest.
On licensing, the repository metadata reports NOASSERTION while the README badge and the link to LICENSE.md indicate FSL-1.1-ALv2. Functional Source License terms are not the same as MIT, and the ALv2 suffix refers to a future licence conversion. Read LICENSE.md yourself and get your own advice; I will not characterise the terms beyond noting the discrepancy. The practical checks before you commit: confirm the licence text matches what your organisation permits, confirm your .env is loaded by the dotenv call in playwright.config.ts, and confirm both an Anthropic and a Google key are present, because the README states the multi-model consensus needs one of each. If you plan to use CUA, verify separately that your OpenAI account has access to the computer tool on the Responses API.
Editorial conclusion
Adopt Passmark if your team already writes Playwright specs and the recurring cost is flaky selectors rather than missing coverage. Skip it if you cannot send test traffic to Anthropic, Google, OpenAI, or a gateway, or if your CI must run without external model calls. Before committing, verify three things: that the LICENSE.md terms permit your use, that your Playwright config loads .env through dotenv, and that your model keys actually reach the models named in the README, since a missing Google key breaks the consensus assertions the library is built around.
Community notes