BrowserGym: a Gymnasium interface for web agent benchmarks

The problem BrowserGym targets: benchmark fragmentation in web agent research

Web agent research has a comparison problem. MiniWoB, WebArena, VisualWebArena, WorkArena, AssistantBench, WebLINX, OpenApps and TimeWarp each ship their own harness, their own task definitions and their own way of returning a result. A team that wants to report numbers on three of them writes three integrations before writing a single agent.

BrowserGym's answer is to register each benchmark's tasks as Gymnasium environments under a shared prefix. The README shows the shape: browsergym/miniwob.choose-list, browsergym/workarena.servicenow.order-ipad-pro, browsergym/webarena.310, browsergym/visualwebarena.721, browsergym/assistantbench.validation.3. The audience is researchers and engineers evaluating LLM-driven browser agents, not teams automating a checkout flow. The README is explicit on this point: BrowserGym is meant to accelerate web agent research and is not meant to be a consumer product.

How the Gymnasium abstraction is applied to a live browser

The mechanism is a thin adaptation layer over an actual browser session. The core package registers browsergym/openended, and the benchmark packages register their own task suites into the same Gymnasium registry. An agent loop is then the standard one: env.reset() returns an observation and info, env.step(action) returns observation, reward, terminated, truncated and info, and env.close() releases the environment.

The openended environment takes task_kwargs with a start_url and a wait_for_user_message flag. With wait_for_user_message set to True, the environment pauses after each agent message sent to the chat, which is what makes the interactive demo in the README possible. Browser control is delegated to Playwright; the setup instructions install Chromium through playwright install chromium rather than through a system browser.

New benchmarks are added by inheriting AbstractBrowserTask from browsergym.core.task. That single extension point is the architectural bet: if a benchmark can express its tasks as that class, it can be exposed through the same loop as everything else. What the README does not describe is the observation schema, the action space, or how reward is computed per benchmark. Those details live in the individual benchmark packages and their own READMEs, which is where you will spend your reading time.

Installing BrowserGym: package choices and per-benchmark setup

The install surface is deliberately split. The README recommends pip install browsergym for everything, and lists narrower options: browsergym-experiments for experiment utilities plus everything below it, browsergym-core for core functionality with no benchmark beyond the openended task, and one package per suite such as browsergym-miniwob, browsergym-webarena, browsergym-webarena-verified, browsergym-visualwebarena, browsergym-workarena, browsergym-assistantbench and browsergym-timewarp. WebLINX is the odd one out, installed as weblinx-browsergym rather than under the browsergym- prefix.

After the pip step, Playwright needs a browser binary:

playwright install chromium

That is the end of the generic setup. Everything after it is benchmark-specific, and the README points outward for each one: browsergym/miniwob/README.md, browsergym/webarena/README.md, browsergym/webarena_verified/README.md, browsergym/visualwebarena/README.md, browsergym/assistantbench/README.md, the WorkArena repository, the OpenApps docs and the TimeWarp repository. For contributors, the development path is a git clone of ServiceNow/BrowserGym followed by make install. The scripts shown for listing tasks are ordinary Gymnasium registry filters, for example collecting gym.envs.registry.keys() where the id starts with browsergym/miniwob.

The setup burden BrowserGym does not remove

The honest limitation is in the README's own wording: each benchmark comes with its own specific setup that requires following additional steps. BrowserGym unifies the interface, not the environment. WebArena and VisualWebArena are self-hosted site suites, WorkArena targets a ServiceNow instance, and MiniWoB has its own README. If your goal is to run one benchmark, you will still read that benchmark's instructions and provision whatever it needs. BrowserGym saves you the agent-side integration, not the infrastructure-side one.

A second constraint is the API surface itself. Because tasks are exposed as Gymnasium environments, the step signature returns terminated and truncated alongside reward. For open-ended web tasks, where success is often judged by a separate evaluator or by a human, that reward signal is not necessarily dense or even present at every step. The README's openended example does not show how reward is produced, and it should not be assumed to be a shaped learning signal without checking the benchmark in question.

The third is the packaging warning. Installing browsergym pulls every benchmark listed above, each with its own dependencies and setup requirements. For a single-benchmark evaluation, one of the narrower packages is the more predictable choice, and the README already offers them.

BrowserGym versus a plain Playwright script

The obvious alternative for browser automation is Playwright directly, which BrowserGym itself uses underneath. The difference in approach is what each one optimizes for. A Playwright script drives a browser toward an outcome you already know how to reach: click this, fill that, assert the confirmation page. BrowserGym inverts that. It hands the browser to an agent, defines a task with a start state and a termination condition, and returns a reward you can aggregate across a suite.

That inversion is why BrowserGym's abstractions look odd for production work. The openended environment waits for user messages, the benchmark environments carry evaluation logic, and the extension point is a task class rather than a page object. If your problem is a deterministic workflow, Playwright alone is less machinery. If your problem is measuring whether an agent can complete 300 varied web tasks, the Gymnasium loop is the part you would otherwise write yourself, badly, once per benchmark.

AgentLab, also from ServiceNow, sits above BrowserGym for implementing, testing and evaluating agents across these benchmarks. The README describes it as the companion framework, which makes BrowserGym the environment layer rather than the experiment layer.

Release cadence, licence status and maintenance signals

The release history shows a steady line: v0.14.2 in August 2025, a development tag v0.14.3.dev4 in January 2026, then v0.14.3 on 2026-01-20. The repository is not archived, and the last push recorded is 2026-07-17. The versioning is pre-1.0, which is the meaningful signal here: minor releases can carry interface changes, and pinning a version is a reasonable default for anything you need to reproduce.

The licence field is reported as NOASSERTION, and the README's own badge links are inconsistent, one pointing at opensource.org/licenses/MIT and the badge URL itself referencing Apache-2.0. That ambiguity matters if you plan to redistribute BrowserGym or build on it commercially. The package metadata on PyPI is the place to resolve it; this article cannot settle it, and nothing here should be read as legal advice. Verify the licence from the package metadata before you depend on it.

Maintenance cost is mostly external. BrowserGym's own code changes at a normal pace, but each benchmark it wraps has its own upstream: WebArena, WorkArena, OpenApps, TimeWarp, WebLINX. When one of those changes its tasks or setup, the corresponding browsergym-* package has to follow. Budget for that drift rather than treating the benchmark list as fixed.

Editorial conclusion

Adopt BrowserGym if you are building or evaluating a web agent and want one reset/step loop across several published benchmarks, and you accept that each benchmark needs its own setup per its README. Do not adopt it as a browser automation library for production workflows: the README states it is not a consumer product and to use it with caution. Before committing, verify two things: which browsergym-* packages you actually need, since the meta-package pulls everything, and whether the specific benchmark you care about is reproducible in your environment, because MiniWoB, WebArena, WorkArena and the rest each document additional steps that BrowserGym does not perform for you.

Official sources

1. Issues
2. README
3. Releases
4. ServiceNow/BrowserGym on GitHub