TestSprite CLI: an AI test runner your coding agent can drive from the terminal
Official TestSprite CLI — AI-powered automated testing from your terminal
At a glance
- What is it?
- TestSprite's official CLI wraps a cloud testing platform in a command surface built for coding agents: create a test from a plain-language plan, run it against the live app, pull one failure bundle, fix, rerun. Here is the mechanism, the install path, and where the model stops fitting.
- Who is it for?
- Adopt it if your team already ships with a coding agent and you want that agent to verify its own output against a running app instead of a mock: the setup command wires the skill in, and the create/failure-get/rerun loop is the whole workflow. Skip it if you cannot send a reachable URL and an API key to a hosted service, or if you need deterministic, hand-authored selectors that never change between runs.
- Can I use it commercially?
- Yes. Apache-2.0 is a permissive licence: you can use, modify and sell software built on it, as long as you keep its copyright and licence notices.
- Is it still maintained?
- Yes. The repository received new commits within the last day.
- What is it written in?
- Mainly TypeScript, according to GitHub's language statistics.
Answers come from the project's GitHub data, last synced on September 16, 2026, and from our analysis. They are not legal advice.
DEEP OPEN-SOURCE ANALYSIS
The problem: an agent that writes code faster than anyone can check it
Coding agents produce a working-looking app in minutes. Verifying that app is still a person opening a browser, clicking through a flow, and reading a stack trace. The README frames this as the gap the CLI closes: it "opens your live app, uses it like a real user, and shows your coding agent exactly what broke." That is the whole pitch, and it defines the audience narrowly. This is not a library you import into a test suite you already own. It is a client for a hosted platform, aimed at teams whose primary author of code is an agent and whose primary reader of test output is also an agent. Humans get the same surface from a terminal or CI, but the design decisions (JSON output, exit codes as signals, a single failure bundle instead of a dashboard) are made for a program consuming them, not a person scanning them.
How the verification loop actually moves data
The mechanism is a three-step round trip, and the README's example session is the clearest description of it. First, `testsprite test create` takes a project id, a type (the example uses `frontend`), and a plan file via `--plan-from`, then runs it with `--wait`. The plan is a JSON file describing the test in plain language, so no browser code is written by hand. A failing run exits 1. Second, `testsprite test failure get` pulls what the README calls "ONE self-consistent failure bundle" into a local directory via `--out`. That phrasing is doing real work: the alternative is a dashboard where a human correlates a screenshot, a console log, and a network trace by hand. Third, the agent edits the code and calls `testsprite test rerun`, which exits 0 on success. The test then persists in what the README calls a durable suite, so the rerun is not a one-off. Everything runs against the live product in the cloud, not mocks, which is the architectural claim that separates it from a locally executed Playwright spec.
Installing TestSprite CLI and running your first test
The package requires Node.js 20.19+, 22.13+, or 24+, matching the `engines` field in package.json. Install it globally, or skip the global install and use npx, which the README notes works too.
npm install -g @testsprite/testsprite-cli
testsprite setup`testsprite setup` prompts for your API key, verifies it, and installs the verification-loop skill for the coding agents it detects. Which agents it writes to is not a guess: each of the eight supported targets (`claude`, `cursor`, `cline`, `windsurf`, `antigravity`, `kiro`, `copilot`, `codex`) stores its skill in a different location, so setup combines the agent that invoked it, if that agent identifies itself, with every agent whose config already exists in the project. In a terminal it prints that list before writing anything, so you can narrow it. If nothing identifies an agent, it installs for `claude` and says so, which is the one case where you should pass `--agent <target>` yourself. For CI or onboarding scripts, the README gives a non-interactive form:
TESTSPRITE_API_KEY=sk-... testsprite setup --from-env --yesFrom there the loop is three commands, typed by the agent. The example below is the README's own session, with the plan file it references:
testsprite test create --project proj_8f0f6 --type frontend \
--plan-from ./checkout-flow.plan.json --run --wait --output json
testsprite test failure get test_3a9f21c7 --out ./.testsprite/failure
testsprite test rerun test_3a9f21c7 --wait --output jsonThe first command exits 1 when the run fails, the second writes the failure bundle to `./.testsprite/failure`, and the third exits 0 once the fix holds. Check what setup configured at any time with `testsprite agent status`.
Where the hosted model stops fitting
The tests run in TestSprite's cloud against your live product. That is the source of the value and the source of the constraint. If your app is not reachable from outside your network, or if policy forbids sending a URL and an API key to a third party, this CLI is the wrong tool regardless of how well it works. There is a second, quieter constraint in the CLI's own version negotiation. The backend advertises a minimum supported CLI version; a client below that floor prints a one-line upgrade advisory on stderr, and a sufficiently old client is rejected outright with exit 14 (`CLIENT_TOO_OLD`). That means an agent pinned to an old CLI version can fail for a reason that has nothing to do with the code under test, and the failure surfaces as an exit code rather than a test result. Anyone automating on top of this needs to treat exit 14 as its own branch, not as a test failure. The README does not document rollback behaviour if an upgrade goes wrong, so plan upgrades as a deliberate step rather than a background one.
TestSprite CLI versus writing the Playwright spec yourself
The obvious alternative is a hand-written Playwright suite, which the project's own topics list alongside e2e-testing. The difference is where the test definition lives and who maintains it. A Playwright spec is code: selectors, waits, and assertions are explicit, deterministic, and reviewable in a pull request. It runs wherever you point a browser, including a local container, with no account and no network egress to a vendor. TestSprite's plan file is plain language, and the platform decides how to drive the browser. That trade buys you tests an agent can author and repair without knowing your DOM, and it costs you the ability to read exactly which selector will be clicked. When a flow changes, a Playwright suite fails loudly at a line number; a plan-driven run fails with a bundle you then interpret. Teams with a mature, stable suite and a small number of critical paths have little reason to move. Teams whose agent writes the code and also needs to check it are the ones the design is aimed at.
Licence, maintenance and the cost of staying current
The repository is Apache-2.0, which permits commercial use and modification, and the package is published publicly on npm under `@testsprite/testsprite-cli`. The licence covers the CLI source; it does not describe the hosted platform's terms, and the README points to testsprite.com for the API key, so the service side is a separate question from the code side. On maintenance: the last push was on 2026-09-14, and the most recent release, v0.11.0, was published on 2026-09-12, two days earlier. Nothing in the repository is archived. The upgrade cost is real but bounded, and it is set by the backend rather than by you: because the service advertises a minimum CLI version and rejects clients below it, an old install does not simply keep working. The README documents a `TESTSPRITE_NO_UPDATE_NOTIFIER=1` environment variable to silence the interactive update check, which contacts the npm registry at most once per 24 hours and sends the package name only. Disabling the notice does not disable the floor, so the advisory is a signal worth keeping in CI logs even if you turn the interactive prompt off.
Editorial conclusion
Adopt it if your team already ships with a coding agent and you want that agent to verify its own output against a running app instead of a mock: the setup command wires the skill in, and the create/failure-get/rerun loop is the whole workflow. Skip it if you cannot send a reachable URL and an API key to a hosted service, or if you need deterministic, hand-authored selectors that never change between runs. Before committing, run testsprite agent status to see which of the eight agent targets setup actually wrote to, and confirm your Node version satisfies the engines field, since a CLI below the backend's advertised floor is rejected with exit 14.
Frequently asked questions
What is a CLI test?
In this project it means a test driven from the terminal rather than a browser or dashboard: you create it with testsprite test create, run it with --run --wait, and read the outcome from the exit code and JSON output. The README's example session shows a failing create exiting 1 and a passing rerun exiting 0.
Is TestSprite free to use?
The repository does not state pricing. The CLI itself is Apache-2.0 and published on npm, but it requires an API key obtained at testsprite.com and runs tests through TestSprite's hosted platform, so the code licence and the service terms are separate questions.
What is AI agent testing and how is it performed?
As this project performs it, a coding agent runs testsprite setup to install a verification-loop skill, creates a test from a plain-language JSON plan with test create --plan-from, and pulls a single self-consistent failure bundle with test failure get after a failing run. The agent reads that bundle, fixes the code, and calls test rerun to confirm.
Community notes