Model or dataset
antiwork/shortest avatar
antiwork/shortest

antiwork/shortest: natural language end-to-end tests on Playwright and Claude

QA via natural language AI tests

5,699 stars340 forksTypeScriptMIT

At a glance

What is it?
Shortest turns plain-English test descriptions into browser runs driven by the Anthropic Claude API on top of Playwright. It suits teams who already accept per-run model cost and want assertions back in TypeScript.
Who is it for?
Adopt Shortest if your team already pays for Anthropic API access, and wants test intent expressed in English while keeping assertions in TypeScript. Skip it if you need deterministic, offline, zero-marginal-cost suites, or if your main flows sit behind CAPTCHAs and bot detection.
Can I use it commercially?
Yes. MIT is a permissive licence: you can use, modify and sell software built on it, as long as you keep its copyright and licence notices.
Is it still maintained?
Yes. The repository last received commits 101 days ago.
What is it written in?
Mainly TypeScript, according to GitHub's language statistics.

Answers come from the project's GitHub data, last synced on November 15, 2026, and from our analysis. They are not legal advice.

From daily snapshots

Star history

  • antiwork/shortest5,699 ★ · +20
Star history
Repository2026-09-242026-11-15
antiwork/shortest5.7k5.7k

20 Stars in the last 52 days, measured from the daily GitHub snapshot. Compare with other projects

Editorial analysis

What Shortest replaces, and for whom

A conventional Playwright suite encodes intent twice: once in the developer's head and once as selectors, waits and assertions. When the UI shifts, the selector breaks and the test fails for a reason that has nothing to do with the behaviour under test. Shortest attacks that specific cost. A test is a string such as "Login to the app using email and password", plus optional context, and the framework decides how to drive the browser.

The audience is narrow but real: teams who already run Playwright, who are comfortable sending page state to a third-party model, and who have an Anthropic API key. The README's web app development section lists React >=19.0.0 and Next.js >=14.0.0 as prerequisites, and warns that using the package with React 18 in Next.js 14+ projects may cause type conflicts with Server Actions and useFormStatus.

How a plain-English test becomes a browser run

The config file sets the shape of the run: a baseUrl, a browser.contextOptions object passed through to Playwright's browser context, a testPattern such as **/*.test.ts, and an ai block whose provider is anthropic. Credentials are read from SHORTEST_ANTHROPIC_API_KEY or ANTHROPIC_API_KEY, and the README notes they can be overridden with ai.config.apiKey.

Test files import shortest from @antiwork/shortest and call it with a description and a context object. The context values are substituted into the run, which is how credentials reach a login flow without being written into the sentence. Execution is Playwright underneath, so the same browser context options, headless flag and page object apply.

The interesting part is what happens after the browser work. A .after callback receives the Playwright page, so you can read localStorage, query a database through Drizzle, and assert with expect. Shortest also exposes beforeAll, beforeEach, afterEach and afterAll hooks that receive the page, and the README's example uses them for Clerk sign-in via email_code and sign-out. In other words, the model drives the browser, but the hard assertions can stay in TypeScript where you control them. That split is the most defensible design decision in the project.

Installing Shortest and running a first test

The README points at the init command as the supported path. It installs @antiwork/shortest as a dev dependency if missing, writes a boilerplate shortest.config.ts, generates .env.local with placeholders such as ANTHROPIC_API_KEY, and appends .env.local and .shortest/ to .gitignore.

bash
npx @antiwork/shortest init

After that, edit shortest.config.ts. The README's example sets headless to false, points baseUrl at http://localhost:3000, ignores HTTPS errors in the browser context, matches **/*.test.ts, and selects the anthropic provider.

typescript
import type { ShortestConfig } from "@antiwork/shortest";

export default {
  headless: false,
  baseUrl: "http://localhost:3000",
  browser: {
    contextOptions: {
      ignoreHTTPSErrors: true
    },
  },
  testPattern: "**/*.test.ts",
  ai: {
    provider: "anthropic",
  },
} satisfies ShortestConfig;

Then write a test file that matches the pattern. The README's example is a login flow whose credentials come from environment variables rather than from the sentence itself.

typescript
import { shortest } from "@antiwork/shortest";

shortest("Login to the app using email and password", {
  username: process.env.GITHUB_USERNAME,
  password: process.env.GITHUB_PASSWORD,
});

Run everything, a single file, or a single test by line number. The README also shows a headless flag for CI use.

bash
pnpm shortest                   # Run all tests
pnpm shortest login.test.ts     # Run specific tests from a file
pnpm shortest login.test.ts:23  # Run specific test from a file using a line number
pnpm shortest --headless        # Run in headless mode using

The example config points baseUrl at http://localhost:3000, so the app under test needs to be reachable there.

Chaining, API assertions and the GitHub 2FA path

Shortest accepts an array of descriptions as a sequential chain, so "user can login with email and password" can be followed by "user can modify their account-level refund policy" in one call. Because the array is ordinary JavaScript, the README shows reusable flow constants combined with the spread operator, which is a neat way to reuse a login step across several scenarios without a page object model.

API testing gets two forms. One constructs an APIRequest with a baseURL and passes req.fetch({ url, method, params }) into shortest alongside the assertion sentence. The other skips the object entirely and puts the whole thing in a template string, including the URL and the query parameter. The second form reads well but hides the request shape inside prose, which is harder to review than the first.

GitHub 2FA is handled with a TOTP secret rather than a live authenticator app. The README walks through pulling the setup key from GitHub's authenticator configuration, storing it as GITHUB_TOTP_SECRET, and running the CLI with the github-code and secret flags to complete enrollment. The .env.example file lists GITHUB_TOTP_SECRET, GITHUB_USERNAME and GITHUB_PASSWORD alongside the Anthropic key, and Mailosaur variables for email validation. That is a lot of secrets concentrated in one file, which is worth noting before you wire it into CI.

Where Shortest is the wrong tool

The README states plainly that execution is AI-powered through the Anthropic Claude API. Every run therefore depends on a network call to a model provider, consumes tokens, and can in principle interpret the same sentence differently across runs. If your suite must be deterministic, offline, or free at the margin, this is the wrong layer to build on. Playwright alone gives you that, at the cost of writing selectors.

There is a second, sharper limitation. Tests that depend on visual stability, canvas rendering, or anti-bot defences are poor fits, because the agent has to perceive the page to act on it. The repository ships an examples/api-failure.test.ts file.

The React and Next.js notes are narrower than they look: React >=19.0.0 and Next.js >=14.0.0 are prerequisites in the web app development section, and the warning concerns type conflicts when the package is used with React 18 in Next.js 14+ projects.

Shortest versus plain Playwright and versus hosted AI test runners

The honest comparison is with Playwright itself, since Shortest is built on it and inherits its browser engine. Plain Playwright costs nothing per run, runs offline, and fails loudly with a stack trace pointing at a selector. Shortest trades those properties for resilience against UI churn: when a button moves or its label changes, the sentence still describes the goal. The trade is explicit, and a team should decide which failure mode it prefers. A flaky selector is annoying but cheap to diagnose; an ambiguous model decision is harder to reproduce.

Against hosted AI test platforms, the difference is where the code lives. Shortest is an npm package, @antiwork/shortest, that runs inside your own repository and CI pipeline, with tests as .test.ts files next to your application code and assertions in your own TypeScript. Hosted tools typically keep the test definitions in their own environment. The README's CI section is a GitHub Actions workflow in the repository, and it tells you to put the Anthropic API key in pipeline secrets, so the execution stays in your infrastructure even though the reasoning does not.

Maintenance, licensing and upgrade cost

The repository is not archived, and the last push was on 2026-08-06. The most recent tagged release listed is v0.4.9 on 2025-04-01, with v0.4.8 and v0.4.7 in the weeks before it. Those two facts sit awkwardly together: the code has moved more recently than the last tag, so anyone pinning to a release number is behind the default branch. If you adopt Shortest, expect to track main or to wait for tags that arrive irregularly.

The package is MIT licensed, and the repository carries a license.md file. MIT is permissive, so the licence itself is unlikely to be the constraint. The real recurring cost is different: every test run makes Anthropic API calls, so suite size and CI frequency translate directly into spend. The README's CI guidance mentions adding the key to pipeline secrets but does not discuss budgets, caching or rate limits, so that accounting is yours to do.

Upgrades carry a second kind of cost. Because test behaviour depends on a model rather than on a fixed selector, a change in the underlying provider can alter how a passing test behaves without any change to your repository.

Editorial conclusion

Adopt Shortest if your team already pays for Anthropic API access, and wants test intent expressed in English while keeping assertions in TypeScript. Skip it if you need deterministic, offline, zero-marginal-cost suites, or if your main flows sit behind CAPTCHAs and bot detection. Before committing, run npx @antiwork/shortest init on a throwaway branch, confirm the generated shortest.config.ts matches your dev server port, and check that the ANTHROPIC_API_KEY you plan to use in CI is scoped to a budget you are willing to spend on every pull request.

Frequently asked questions

What is antiwork/shortest?

It is an AI-powered natural language end-to-end testing framework built on Playwright, with test execution driven by the Anthropic Claude API. Tests are written as English descriptions in TypeScript files that match the testPattern in shortest.config.ts.

How do I install and set up Shortest in an existing project?

Run npx @antiwork/shortest init. According to the README, that installs @antiwork/shortest as a dev dependency if needed, creates a default shortest.config.ts, generates .env.local with placeholders such as ANTHROPIC_API_KEY, and adds .env.local and .shortest/ to .gitignore.

Does Shortest need an Anthropic API key?

Yes. The README says the key defaults to the SHORTEST_ANTHROPIC_API_KEY or ANTHROPIC_API_KEY environment variables and can be overridden through ai.config.apiKey. For CI, the README says to add the key to your pipeline secrets.

Can Shortest tests run headless in CI?

The README shows a --headless flag and a CI section stating that tests can run in a CI/CD pipeline in headless mode, with the Anthropic API key stored in pipeline secrets. An example GitHub Actions workflow is linked from the README.

Can I add my own assertions to a Shortest test?

Yes. The README documents an .after callback that receives the Playwright page, where you can read values such as localStorage and assert against a database with expect. It also provides beforeAll, beforeEach, afterEach and afterAll lifecycle hooks that receive the page.

Official sources

  1. antiwork/shortest on GitHub
  2. License: MIT
  3. Project website
  4. README
  5. Releases
Add this badge to your README

If you maintain this project, the badge below links readers to this analysis and shows its maintenance status from the daily GitHub snapshot. Paste the markdown into your README; add ?metric=license or ?metric=stars to the image URL for a different field.

Add this badge to your README

markdown
[![Hysen Labs](https://hysenlabs.com/badge/antiwork-shortest.svg)](https://hysenlabs.com/projects/antiwork-shortest)

Star history chart for your README

markdown
[![Star history](https://hysenlabs.com/chart/antiwork-shortest.svg)](https://hysenlabs.com/projects/antiwork-shortest)

Was this useful?