Pa11y CLI and Node.js accessibility testing: install, runners, and exit codes

What Pa11y solves, and who it is built for

Automated accessibility checks are easy to run once and hard to keep running. Pa11y turns a page into a pass or fail signal that a script can act on. It drives a real browser through Puppeteer, injects a test runner into the rendered page, and prints the issues it finds.

The audience is narrow and practical: developers who already have a test or deploy pipeline and want accessibility to be part of it. The README describes the tool as running "accessibility tests on your pages via the command line or Node.js, so you can automate your testing process." That sentence is the whole pitch. There is no editor plugin, no hosted dashboard, and no manual review workflow in the repository.

Because it runs headless, Pa11y sees the DOM after JavaScript has executed. That matters for single-page applications where the markup that ships from the server is not the markup a user receives. The trade-off is that a browser has to start on every run, which is the main reason CI jobs using Pa11y are slower than a static HTML linter.

How the runners, standards and reporters fit together

A Pa11y run has three configurable parts: a runner that supplies the rules, a standard that selects which rules apply, and a reporter that formats the output.

The default runner is HTML_CodeSniffer, listed in the README as htmlcs. The alternative is axe, backed by the axe-core dependency in package.json. The two can run together in a single invocation, and the README shows the repeated flag form for that. The --standard flag is documented as being used only by htmlcs, with WCAG2AA as the default and WCAG2AAA and WCAG2A as the other accepted values. That is a real constraint: switching to axe does not change the standard, because axe applies its own rule set.

Reporters cover cli, csv, html, json and tsv. Custom reporters are CommonJS modules, and resolution follows a documented order: for --reporter rainbows, Pa11y first looks for an installed package named pa11y-reporter-rainbows, then for a module file at <cwd>/rainbows. Both go through Node's require, so a reporter that is not resolvable fails at load time.

Severity is split into error, warning and notice. Only errors trigger a failing exit code by default, and --include-notices and --include-warnings control whether the lower severities appear in the report at all. The axe-only flag --level-cap-when-needs-review caps the severity of issues that require manual review, defaulting to error.

Installing Pa11y and running a first test

Pa11y 10 requires an even-numbered version of Node.js 22.13.0 or above; the engines field in package.json states ^22.13.0 || >=24. Older Node.js releases can be used with Pa11y 9 or below, which the README points to under support and migration.

Install it globally with npm:

npm install -g pa11y

Then point it at a URL. The default runner is HTML_CodeSniffer, so no runner flag is needed for a first run:

pa11y https://example.com

The CLI reporter prints a human-readable list. If the page has issues of type error, the process exits with code 2. You can confirm what Pa11y thinks it is running against, including the environment details it will use, before trusting a CI result:

pa11y --environment

To run axe instead, or both runners in one pass, repeat the flag:

pa11y https://example.com --runner axe --runner htmlcs

Local files work too, but the README is explicit that paths must be absolute, not relative. A config file is optional and defaults to pa11y.json in the current directory; command-line options take priority over values in that file.

Exit codes are the interface that matters in CI

Pa11y's output is for humans. Its exit codes are for scripts, and the README defines three: 0 for a successful run with no errors, 1 for a technical fault, and 2 for a successful run that found errors. The distinction between 1 and 2 is the useful part. A crashed browser and a page with a missing alt attribute should not be treated the same way in a pipeline, and Pa11y keeps them apart.

What counts as a failure is controlled by --level, which accepts error, warning, notice or none. Setting it to none makes the command always exit 0, which is a reasonable way to collect data before enforcing anything.

The second control is --threshold, which permits a fixed number of issues before failing. The README's example is concrete: with --threshold 10, a page with 9 errors exits 0 and a page with 10 or more exits 2. This is the pragmatic escape hatch for teams adopting the tool on an existing codebase. It is also the setting most likely to be abused, because a threshold that never moves quietly turns a quality gate into decoration.

Individual issues can be suppressed with --ignore, either as a semicolon-separated string or by repeating the flag. Suppressions are by issue code, so they survive page changes that introduce new issues of a different code.

Where Pa11y gives you a false sense of coverage

Pa11y checks what a rule engine can check in a rendered DOM. It does not tell you whether the page is usable with a keyboard, whether focus order makes sense, or whether a screen reader announces a custom widget correctly. Those are the failures that matter most to real users, and no runner in this repository addresses them.

The runner choice compounds the problem. HTML_CodeSniffer and axe do not implement identical rule sets, so a page can pass with the default runner and fail under axe. Running both is the honest configuration, and it costs roughly two passes over the same page.

Severity inflation is the other trap. Notices and warnings are excluded from the report unless you ask for them, and excluded from failure unless you change --level. A team that leaves the defaults in place is testing for errors only, which is a narrower claim than "we test accessibility."

Finally, there is operational fragility. Every run starts a browser through Puppeteer. The dependency is pinned to a caret range on puppeteer ^25.9.0, and the README does not document rollback behaviour for a failed browser launch. The repository does include a TROUBLESHOOTING.md file at the top level, which is where a browser-startup problem would be addressed, but the README itself does not walk through it.

Pa11y against axe-core and Lighthouse

The comparison people search for is pa11y vs axe. It is not quite a like-for-like pairing, because axe-core is the rule engine that Pa11y can load as a runner. Using axe through Pa11y gives you axe's rules plus Pa11y's browser control, reporters, configuration file, ignore list and exit codes. Using axe-core directly gives you the rules and leaves the surrounding harness to you. If you already have a Puppeteer or Playwright setup, adding axe-core to it is less machinery than adding Pa11y on top.

Lighthouse is the other common reference point. It is a broader audit that includes accessibility as one category among performance, SEO and best practices, and it produces a score. Pa11y produces a list of issues and an exit code, and does nothing else. If your goal is a single number in a report, Lighthouse fits. If your goal is a build that fails on a specific class of defect, a tool that exits 2 on errors is easier to wire up than a score threshold.

The repository layout also shows an example/puppeteer/ directory alongside example/actions/, example/basic/, example/configs/ and example/multiple/. The actions example points at a capability the README does not cover in the truncated text: driving page interactions before the audit runs. That matters for anything behind a login or a modal, and it is worth reading the example directory rather than assuming Pa11y only handles static URLs.

Licence and upgrade cost

Pa11y is licensed LGPL-3.0-only, stated in both the README badge and the license field in package.json. This is not the permissive MIT or Apache-2.0 that most JavaScript tooling uses, and it is worth flagging to whoever handles licence review before the tool lands in a distributed product. Invoking the CLI or requiring the package as a dependency is a different situation from modifying Pa11y and shipping the modified library, and only your own review can settle which applies. Nothing here is legal advice.

The upgrade path has a real cost. Moving to Pa11y 10 means moving to Node.js 22.13.0 or above on an even-numbered release, which may be a larger change than the Pa11y upgrade itself. The repository carries a MIGRATION.md file at the top level, and the README links to a support and migration section, so the breaking changes between major versions are documented rather than left to release notes.

Maintenance is current: the last push to the repository was on 2026-09-21, and version 10.0.0 was released on 2026-08-28. The prior release, 9.1.1, dates to 2026-02-26, so the project does not ship on a tight cadence. Expect major versions to arrive with Node.js runtime changes attached.

Editorial conclusion

Adopt Pa11y if you want a scriptable WCAG check that fails a build on errors and can run both htmlcs and axe in one pass. Skip it if you need visual or keyboard-flow verification, or if you are pinned below Node.js 22.13.0, since version 10 requires an even-numbered release of 22.13.0 or above. Before wiring it into CI, run pa11y --environment on the build agent to confirm the bundled Chromium starts, then set --threshold to a number you can actually defend.

Frequently asked questions

Official sources

1. License: LGPL-3.0
2. pa11y/pa11y on GitHub
3. Project website
4. README
5. Releases