Windows Agent Arena: a Windows 11 VM harness for desktop agent benchmarks

The problem WAA was built to remove: agent demos that cannot be compared

Desktop agent work has a measurement problem. A demo video shows one agent clicking through one application on one machine, and nothing in that video tells you whether a second agent would have done better on the same task, or whether the first agent would repeat its result tomorrow. Windows Agent Arena addresses that by fixing the environment instead of the agent. The README describes it as "a scalable Windows AI agent platform for testing and benchmarking multi-modal, desktop AI agents", and the operative word is reproducible: every run happens inside the same prepared Windows 11 image, against the same task definitions, with the same command channel into the VM. The intended audience is researchers and developers who need a number they can put next to someone else's number. If you are building a product on top of a computer-use model and only need it to work on your own machine, this is more machinery than the job requires.

What actually runs inside the VM: a Python server on the guest, a client on the host

The architecture visible in the material is a two-sided split. On the guest side, a prepared Windows 11 snapshot hosts a Python server that "receives and executes agent commands", per the README. On the host side, the client scripts in src/ drive that server, feed the model observations, and record outcomes. The agent never touches the desktop directly; it emits commands that the in-VM server performs, which is what makes the loop observable and repeatable. Multi-modality enters through the observation path rather than the action path: the released Navi agent pairs with OmniParser, Microsoft's open-sourced "screen understanding model", and the top-performing configuration in the paper is invoked with --som-origin mixed-omni. Two architecture diagrams are linked from the README, one for local and one for cloud deployment, which implies the same client/server contract holds whether the VM sits on your workstation or on Azure ML. The README does not spell out the wire format between client and server, so treat that as something to read in src/win-arena-container rather than something documented at the top level.

Getting it running locally: conda, config.json, two Docker images and a renamed ISO

The local path has four gates. First, environment: Python 3.9, with conda recommended (conda create -n winarena python=3.9), then pip install -r requirements.txt from the cloned repository. A Docker daemon must be running, and on Windows the README points at Docker with WSL 2. Second, credentials: a config.json at the project root holding OPENAI_API_KEY, or AZURE_API_KEY plus AZURE_ENDPOINT for an Azure OpenAI endpoint. Third, images: docker pull windowsarena/winarena-base:latest for the base layer that carries packages and models, then cd scripts && ./build-container-image.sh to produce windowsarena/winarena:latest from the current src/ tree. The script accepts --build-base-image true when Dockerfile-WinArena-Base has changed, and --help lists the rest. Fourth, the guest: download the Windows 11 Enterprise Evaluation ISO (90-day trial, English, United States, roughly 6 GB), rename it to setup.iso, and place it at src/win-arena-container/vm/image. The README then describes an automatic setup that produces a WAA snapshot, also called a golden image, around 30 GB, containing the applications the benchmark needs. That snapshot is the artifact you will rebuild whenever the task set or guest configuration moves.

The hard difficulty flag changes the task, not the score

One design decision deserves attention because it changes what a benchmark number means. The default is diff_lvl="normal" in src/win-arena-container/start_client.sh. Switching it to diff_lvl="hard" makes agents "initialize/set up the task themselves (e.g., finding and opening the right program/application for the task) rather than have the task set up for them by the task config", per the release note of 2024-11-10. In other words, the normal mode measures whether an agent can complete a task given a prepared desktop, and the hard mode measures whether it can also reach the starting line. Those are different capabilities, and a score from one mode is not comparable to a score from the other. Anyone publishing results should state which mode produced them, and anyone comparing against a third party's numbers should check the same thing before drawing conclusions.

Where the harness fights you: setup weight, API assumptions and a narrow OS target

The costs are concrete. You need a Docker daemon, an OpenAI or Azure OpenAI key, Python 3.9, a 6 GB ISO download, and about 30 GB for the snapshot before a single task runs. The snapshot is not a one-time cost either: the README frames it as something you prepare "before running the arena", and a golden image that must contain the benchmark's applications will need rebuilding as those applications or their configuration change. The model dependency is also narrower than the platform name suggests. The quickstart assumes an OpenAI or Azure OpenAI endpoint, and the released agent code targets OmniParser for screen understanding, so swapping in a local vision model is not a documented path. Finally, the target is Windows 11 specifically. If your agents operate on macOS or a Linux desktop, the VM, the in-guest server and the task configs are all aimed at the wrong operating system, and no amount of host-side work fixes that. The last push to the repository is dated 2026-04-13 while the most recent tagged release is v0.0.4 from 2024-09-28, so the tagged releases and the main branch have diverged for a long stretch; pin a commit rather than a tag if you need to reproduce a specific run.

How it differs from OSWorld and other desktop agent benchmarks

The obvious comparison is OSWorld, which takes the opposite bet on scope: it spans multiple operating systems rather than committing to Windows. That breadth buys cross-platform comparisons and costs fidelity to any single OS. WAA narrows the target and spends the saved effort on Windows-specific realism, a prepared Windows 11 guest with the applications the tasks need, and an in-VM Python server that executes agent commands. The deployment story differs too: WAA ships Azure ML scripts for running many agents in parallel, which the README frames as producing benchmark results for hundreds of tasks "in minutes, not days". If your research question is about Windows applications specifically, or about how an agent behaves when the desktop is a real Windows 11 install rather than a simulated one, the narrower harness is the better fit. If your question is whether a policy generalises across operating systems, a multi-OS benchmark answers it more directly, and you would be paying WAA's setup cost for a property it was not designed to measure.

Licence, maintenance and what the release cadence tells you

Windows Agent Arena is MIT licensed, which permits commercial and modified use provided the copyright notice and permission notice are retained; this is a summary of the licence identifier, not legal advice, and you should read the LICENSE file for the binding terms. The maintenance picture from the supplied metadata is mixed. The repository is not archived and shows a push in April 2026, but the newest tagged release is v0.0.4 from September 2024, and the three releases listed (v0.0.2, v0.0.3, v0.0.4) all landed within a few weeks of each other in 2024. For a benchmark, that pattern matters more than for a library: the value of a benchmark comes from the task set staying fixed long enough for results to be comparable, so a slow release cadence is not automatically a problem. What you should not do is assume the tag and the branch behave identically. Budget for the golden-image rebuild as a recurring cost tied to guest changes, and read src/win-arena-container/start_client.sh and the config.json keys directly rather than relying on the README's summary, since the README itself defers to the linked architecture diagrams for the component detail.

Editorial conclusion

Adopt Windows Agent Arena if you are evaluating a computer-use agent and need a Windows-specific, repeatable task set rather than ad hoc screen recordings. Do not adopt it if you want a general agent framework, if you cannot supply a Docker daemon plus a 6 GB Windows 11 Enterprise Evaluation ISO and roughly 30 GB of snapshot storage, or if your agents target macOS or Linux desktops. Before committing, verify the pinned Python 3.9 requirement against your environment, confirm whether your OpenAI or Azure OpenAI endpoint accepts the model names the client scripts send, and check the diff_lvl setting in src/win-arena-container/start_client.sh, because normal and hard change what the agent is expected to set up itself.

Official sources

1. License: MIT
2. microsoft/WindowsAgentArena on GitHub
3. Project website
4. README
5. Releases