AgentLab: A Research Harness for Web Agents, Not a Product

The problem AgentLab addresses: comparing web agents on shared ground

Web agent papers are hard to compare. Each one tends to ship its own harness, its own task set, and its own scoring script, so a result reported on WebArena in one codebase cannot be rerun in another without reimplementation. AgentLab positions itself as the layer that removes that reimplementation. It is described as a framework for developing and evaluating agents on the benchmarks supported by BrowserGym, and the README calls it the preferred way for running benchmarks like WebArena. The target user is a researcher or research engineer who needs to run an agent configuration against a fixed task set, repeat the run, and compare the numbers to someone else's run. It is explicitly not aimed at end users: the README carries a warning that AgentLab is meant to accelerate web agent research and is not a consumer product, with the instruction to use with caution. That framing matters when you evaluate the setup steps below, because several of them assume you can host a browser environment and supply your own model API keys.

How the pieces fit: BrowserGym tasks, agent args, Ray, study directories

The architecture visible in the README separates three concerns. Task environments come from BrowserGym, a sibling ServiceNow project that supplies the benchmark integrations; AgentLab does not reimplement WebArena or WorkArena, it consumes them. Agents are Python objects that extend an AgentArgs class, and the README stresses that the object must be imported from a module reachable on PYTHONPATH so it can be unpickled correctly, which tells you the agent configuration is serialized and shipped to worker processes rather than constructed inside each worker. Execution is parallelized with Ray, which is listed as a headline feature for large scale experiments. The unit of work is a study: make_study takes a benchmark name, a list of agent configurations, and a comment, then study.run(n_jobs=5) dispatches the work. Results land under a root directory controlled by AGENTLAB_EXP_ROOT, defaulting to $HOME/agentlab_results. Because a study is a directory rather than an in-memory run, it can be reloaded later with Study.load and inspected with find_incomplete, which is the mechanism behind the relaunch workflow. Model access is abstracted behind what the README calls a unified LLM API covering OpenRouter, OpenAI, Azure, and self-hosted models through TGI, so swapping providers is a configuration change rather than a code change.

Getting it running: the actual commands and environment variables

Installation is a pip install of the agentlab package, plus a separate playwright install step for the browser binaries. Python 3.11 or 3.12 is required; the README does not mention 3.13. Environment variables drive the rest. AGENTLAB_EXP_ROOT sets where experiment results are written and defaults to $HOME/agentlab_results. OPENAI_API_KEY is needed for OpenAI models, OPENROUTER_API_KEY for OpenRouter, and AZURE_OPENAI_API_KEY plus AZURE_OPENAI_ENDPOINT for Azure. Before any of that is useful you have to prepare the benchmark itself, following the per-benchmark setup links in the supported benchmarks table; those setups are not described in the AgentLab README and several of them involve hosting Docker containers. The minimal experiment script imports a predefined configuration such as AGENT_4o_MINI from agentlab.agents.generic_agent, calls make_study with a benchmark string like miniwob, webarena, or workarena_l1, and calls run with a job count. Resuming a partial run means constructing a Study from a path on disk, calling find_incomplete with include_errors set to true, and calling run again. There is also a single-task path: agentlab-assistant --start_url https://www.google.com launches an interactive assistant, and --agent_config="module.path.to.your.AgentArgs" points it at your own agent instead of the default. The README notes that this assistant runs at your own cost and risk.

The benchmark table is the real feature list, and part of it is unfinished

The supported benchmarks table is where AgentLab's scope becomes concrete, and it is also where the caveats live. WebArena and WebArena-Verified both list 812 task templates with a 30 step cap and multi-tab support, self hosted via Docker. WorkArena is split across three levels with 33, 341, and 341 templates and differing step caps of 30 and 50, served from a demo instance rather than a local container. WebLinx is the outlier: 31,586 task templates but a maximum of one step, which makes it a different kind of evaluation from the multi-step browsing benchmarks. VisualWebArena lists 910 templates, AssistantBench 214 against the live web, MiniWoB 125 against self-hosted static files, and OSWorld 369. GAIA and Mind2Web-live are marked soon, with no task counts or setup links at all, and the BrowserGym Leaderboard column reads soon for every single row. That last detail is worth pausing on. The README advertises a unified leaderboard as a feature, but the table that would feed it shows no benchmark as currently listed. Treat leaderboard participation as something to confirm against the live Hugging Face space rather than something the README demonstrates.

Where AgentLab is the wrong tool

The most obvious limitation is the setup surface. AgentLab does not ship the benchmarks; it delegates to BrowserGym setup instructions that live in other repositories, and WebArena, VisualWebArena, and WebArena-Verified all expect you to host Docker containers yourself. If your goal is to evaluate an agent on a hosted task suite without operating browser infrastructure, this framework adds work rather than removing it. The second limitation is the state model. Because agent configurations are pickled and unpickled across Ray workers, the README's warning that the object must be importable from a module on PYTHONPATH is not a stylistic note, it is a constraint that will bite anyone who defines an agent class inside a notebook or a script that is not on the path. Expect opaque unpickling failures when you move a working local agent into a parallel study. Third, the README itself sets the boundary: it is not a consumer product, and the assistant command is offered with an explicit at your own cost and risk caveat. Anyone looking for a supported browsing agent should stop reading here. Finally, the material does not describe a hosted evaluation service, a scheduler, or a result database beyond the filesystem layout under AGENTLAB_EXP_ROOT, so teams expecting managed infrastructure will be disappointed.

How it differs from rolling your own Playwright loop

The realistic alternative is not another named framework; it is a hand-written Playwright or Selenium script per benchmark, which is what many agent papers do today. The difference in approach is structural. A hand-written loop gives you total control over the browser session and the scoring logic, and it costs you a reimplementation every time you add a benchmark or want to compare against a published number. AgentLab inverts that: you accept BrowserGym's task definitions and its step caps, and in exchange you get a study abstraction that records results to disk, a Ray path to run many configurations at once, and a resume mechanism via find_incomplete for runs that die partway. The trade is flexibility for comparability. If your research question requires instrumentation that BrowserGym does not expose, the hand-written loop is the better choice, and no amount of framework convenience compensates. If your question is how configuration A compares to configuration B on WebArena under a 30 step budget, AgentLab is doing exactly the work you would otherwise redo.

Maintenance cost, licensing, and what to check before you commit

Maintenance here has two layers. The package layer is active: releases v0.4.0, v0.4.1, and v0.4.2 landed between February 2025 and January 2026, with the repository's last push recorded in July 2026, so you are not adopting an abandoned codebase. The environment layer is the expensive one. Browser binaries come from playwright install and need periodic refresh, benchmark containers need to be rebuilt when BrowserGym changes, and the Python requirement is pinned to 3.11 or 3.12, which means upgrading your interpreter past 3.12 is a decision you make against this framework's stated support. On licensing, the material is inconsistent and you should resolve it yourself rather than assume. The repository metadata reports NOASSERTION, while the README's PyPI licence badge links to the Apache 2.0 text. Those are not the same statement, and the difference matters if you plan to redistribute a modified version or bundle it into a commercial product. Check the LICENSE file in the repository and the PyPI metadata directly before relying on either. Nothing here is legal advice; it is a note that the two sources disagree.

Editorial conclusion

Adopt AgentLab if you are already committed to BrowserGym benchmarks and need to run many agent configurations in parallel with results you can reload and resume. Do not adopt it if you want a hosted evaluation service or a consumer-facing browsing assistant; the README's own warning says it is not a consumer product. Before committing, verify three things yourself: that the specific benchmark you care about is not still marked soon in the supported benchmarks table, that the licence identifier resolves (the repository reports NOASSERTION while the PyPI badge links to Apache 2.0), and that your Python version is 3.11 or 3.12, which is the range the setup section names.

Official sources

1. Issues
2. README
3. Releases
4. ServiceNow/AgentLab on GitHub