Artillery: Load Testing HTTP, WebSocket and Playwright Browsers from One YAML File

What Artillery Is For, and Who It Is Not For

Artillery targets teams that need to answer a performance question about a running system: how many requests per second can this API absorb, does the WebSocket connection hold under concurrent users, does the checkout page still work when a hundred browsers hit it at once. The README frames it as a complete load testing platform, with workload modelling through request chains, multiple steps and transactions, so a scenario can describe a sequence of dependent calls rather than a single endpoint hammered in a loop.

The audience is engineers who are comfortable in JavaScript or TypeScript. The README states that you can use any Node.js module inside a test, which means custom logic, token generation and data setup do not require a separate DSL. The repository ships examples for refresh-auth-token, generating-vu-tokens and script-overrides, which suggests the maintainers expect real authentication flows rather than open endpoints.

It is the wrong tool when the thing you want to measure is not reachable over a network protocol Artillery speaks, or when you need a language binding for a stack that is not Node.js. The README lists HTTP, WebSocket, Socket.io, gRPC and Kinesis; anything outside that list depends on the plugin API, and the plugin API is a real dependency you would own.

How a Scenario Becomes Load: Engines, Phases and the Cloud Runner

The mechanism is a scenario file plus a set of phases. A scenario describes the steps a virtual user takes; phases describe how many virtual users arrive and how fast. The README describes this as powerful workload modelling with request chains, multiple steps and transactions. The repository also carries examples/scenario-weights and examples/multiple-scenario-specs, so a single test run can mix several user behaviours at different proportions rather than assuming one uniform traffic shape.

Protocol support is implemented as engines. The topics list includes grpc, socketio and websocket alongside http, and the README adds Kinesis. Browser load is a separate engine that drives Playwright, and the examples directory has browser-load-testing-playwright, browser-playwright-reuse-authentication and browser-playwright-reuse-typescript. The reuse examples matter: logging in once per virtual user and reusing that session is the difference between a browser test that measures your application and one that measures your login endpoint.

For scale, the README says tests can be distributed on AWS Lambda or AWS Fargate, described as out-of-the-box and free, with no infrastructure to set up or manage. That is the architectural claim worth scrutinising: the load generators are ephemeral cloud workers rather than long-lived machines you provision. The trade-off is that your test now depends on AWS capacity and on the network path from AWS to your target, which is not the same path your users take. For observability, the topics list OpenTelemetry and the README claims 20+ integrations; the examples directory includes prometheus-grafana-dashboards and http-metrics-by-endpoint, which is the more useful of the two because aggregate latency hides which endpoint degraded.

Installing Artillery and Running a First Test

The top-level README does not carry installation instructions; it points to packages/artillery#readme for the details, and the project homepage at artillery.io hosts the docs. Treat the package README as the source of truth for the current install command, since the monorepo layout means the published package is not the repository root.

The repository itself is an npm workspace monorepo. The root package.json declares [email protected] as the package manager and defines build, test and lint scripts that run through turbo, which tells you how the maintainers work on it, not how you install it as a user.

The README gives no example scenario file, and the repository files provided here do not include one either, so the place to start is the examples directory. examples/starter-kit is the obvious first stop, with examples/http-set-custom-header and examples/http-metrics-by-endpoint covering the simpler HTTP cases and examples/browser-load-testing-playwright covering the browser engine. Copy one of those directories and edit it rather than writing a scenario from scratch, because the config and scenario structure is what you would otherwise be guessing at.

For browser load specifically, examples/browser-playwright-reuse-authentication shows the login-and-reuse pattern, which matters because the setup cost per virtual user dominates when each virtual user is a real headless browser. The examples/README.md is the index for the whole set.

Where Artillery Gets Awkward

The licence split is the sharpest constraint. The README states that most of the repository is MPL-2.0, but some Azure-specific modules are under the BSL, and that using Artillery on Azure is fine for evaluation and proof-of-concept while commercial and production usage requires a commercial license. That is not a footnote. If your load generators run on Azure, the free story the README tells about AWS Lambda and Fargate does not carry over, and you need to read LICENSE-BSL.txt rather than assume the MPL covers everything.

Cloud-scale testing also changes what you are measuring. Running virtual users inside AWS Lambda or Fargate puts the generator in a different network position from your real clients, and Lambda has concurrency and duration characteristics that a long-running load generator does not. The README presents this as zero infrastructure to manage, which is true, and the cost is that the generator environment becomes a variable in your results.

Browser load testing is expensive in a way HTTP load testing is not. Each Playwright virtual user is a real headless browser, so the number of concurrent users you can drive per worker is far lower than for HTTP scenarios. The examples for reusing authentication exist precisely because the setup cost per virtual user dominates otherwise. If your question is purely about API throughput, the browser engine adds cost and noise for no benefit.

Finally, the top-level README is thin. Installation, the scenario schema and the CLI flags are all delegated to packages/artillery and the external docs site, so anyone evaluating the project from the repository root alone will not find enough to run a test.

Artillery Against k6 and Locust

The closest alternatives are k6 and Locust, and the difference is mostly about what language your test logic lives in and where the load runs.

k6 scripts are written in JavaScript but execute in a Go runtime, so you get JavaScript syntax without Node.js modules. Artillery's README makes the opposite promise: any Node.js module is available inside a test. If your test needs to sign a request with an in-house library, generate a token with your own SDK, or reuse application code, Artillery removes a translation step that k6 imposes. If you want a single static binary with no Node.js runtime, k6 is the simpler deployment.

Locust takes a third position: scenarios are Python classes, and the tool is built around a distributed master and worker model you run yourself. Artillery's distributed story is serverless on AWS Lambda or Fargate, which the README describes as no infrastructure to set up or manage. That is a genuine operational difference. Locust gives you control over the generator hosts and their network placement; Artillery gives you less to run but less control over where the load originates.

On protocol coverage the three overlap heavily for HTTP and WebSocket. Artillery's distinguishing feature in this comparison is the Playwright engine, which lets a browser-level scenario sit alongside API scenarios in the same project. Neither k6 nor Locust is described in this material, so treat that as a difference in emphasis rather than a claim about their capabilities.

Maintenance, Releases and Licence Cost

The repository is not archived, and the last push was on 2026-09-20. Release cadence is visible in the tags: artillery-2.0.32 on 2026-05-19, artillery-2.0.33 on 2026-06-16, and artillery-2.0.34 on 2026-08-14. That is roughly monthly, which is a reasonable rhythm for a tool you would put in a CI pipeline.

The monorepo is built with turbo and npm workspaces, and the root package.json pins [email protected] while declaring TypeScript ^5.9.3 and Node types ^22.20.1. A pinned package manager version is a small signal that the maintainers care about reproducible builds; it also means contributing requires matching that npm version or the workspace commands may behave differently. The overrides block forces ws ^8.21.0 across engine.io, engine.io-client and socket.io-adapter, which is the kind of transitive pin you see when a dependency has a security or compatibility issue upstream.

Upgrade cost for users is mostly the scenario file format plus any plugins you depend on. The release tags show patch-level versioning within 2.0.x, and the README does not document a deprecation policy or a rollback path for a bad release, so pin the version in CI and read the release notes before moving.

On licensing: MPL-2.0 is file-level copyleft, so modifications to Artillery's own files carry obligations, while your separate scenario files and test code generally do not. The Azure BSL modules are a different regime entirely and the README is explicit that commercial and production usage on Azure needs a commercial license. This is a description of what the README says, not legal advice; if your organisation runs on Azure, get the licence question answered before the proof of concept becomes a dependency.

Editorial conclusion

Adopt Artillery if you already write JavaScript or TypeScript and want HTTP, WebSocket and browser-level load in one scenario file, and if running tests on AWS Lambda or Fargate without managing load generators appeals to you. Do not adopt it if you need the Azure-specific modules for commercial or production work, since those are under BSL and the README states that commercial and production usage requires a commercial license. Before committing, check the packages/artillery README for the current install command and verify whether your target protocols are covered by the built-in engines or need a plugin.

Frequently asked questions

Official sources

1. artilleryio/artillery on GitHub
2. License: MPL-2.0
3. Project website
4. README
5. Releases