OSWorld: A Real-Desktop Benchmark for Multimodal Agents
[NeurIPS 2024] OSWorld: Benchmarking Multimodal Agents for Open-Ended Tasks in Real Computer Environments
At a glance
- What is it?
- OSWorld from xlang-ai evaluates AI agents on open-ended tasks in real Ubuntu, Windows, and macOS environments. This review covers its architecture, setup paths, and the trade-offs you should weigh before adopting it.
- Who is it for?
- Adopt OSWorld if you need to benchmark a multimodal agent against realistic, open-ended desktop tasks where state changes are the success metric. It is a strong fit for academic researchers and industrial teams with access to VMware, VirtualBox, or KVM-capable servers.
- Can I use it commercially?
- Yes. Apache-2.0 is a permissive licence: you can use, modify and sell software built on it, as long as you keep its copyright and licence notices.
- Is it still maintained?
- Yes. The repository last received commits 1 day ago.
- What is it written in?
- Mainly Python, according to GitHub's language statistics.
Answers come from the project's GitHub data, last synced on September 15, 2026, and from our analysis. They are not legal advice.
DEEP OPEN-SOURCE ANALYSIS
Why OSWorld exists and who needs it
Most agent benchmarks run inside a controlled grid or a simulated browser where the environment resets after every episode. OSWorld takes a different route: it places an agent in a real operating system, either Ubuntu, Windows, or macOS, and asks it to complete tasks that require typing, clicking, dragging, and reading the screen. The agent sees pixels and receives keyboard and mouse actions, which is closer to how a human operates a computer. The intended audience is researchers building multimodal large language models or vision-language models that must act on a GUI, plus engineers who want a repeatable stress test for an automation product. The benchmark focuses on open-ended tasks, meaning the agent cannot rely on a fixed action space or a known set of success states.
The core mechanism: state-based scoring on a real OS
OSWorld does not just check whether the agent's final answer matches a string. According to the repository, the environment sets up an initial state inside a virtual machine, then the agent performs actions through the desktop. After the agent finishes, the evaluator examines the resulting system state, such as whether a file was created with the correct content, a spreadsheet formula was entered, or a setting was toggled. This state-based evaluation is more robust than simple text matching because it catches partial successes and unexpected side effects. The benchmark includes a set of tasks defined in the evaluation_examples folder, each with a description, an initial state, and a success condition. The agent operates through a DesktopEnv class, which abstracts the underlying VM provider. The design forces the model to handle real-world variability: window focus, dialog boxes, and system notifications are not hidden from the agent.
Installation and the VMware path
The README gives a concrete setup for a bare-metal machine. You clone the repository, install requirements.txt, and optionally create a Conda environment with Python 3.10 or higher. Then you must install VMware Workstation Pro, or VMware Fusion on Apple Silicon, and configure the vmrun command. A quick verification is running vmrun -T ws list, which should show running virtual machines. The repository also offers a lighter installation: pip install desktop-env installs the environment package without the benchmark tasks. That separation suggests you could build your own evaluation suite on top of the environment if the default tasks do not fit. The setup script downloads virtual machines automatically, which saves time but assumes you have enough disk space and a hypervisor that works.
Docker, Modal, and the KVM requirement
For servers or cloud instances that are already virtualized, the README points to a Docker route. The key prerequisite is KVM support; you check it with egrep -c '(vmx|svm)' /proc/cpuinfo. If the count is zero, the performance will suffer. The Docker provider requires you to pass provider_name=docker and os_type=Ubuntu or Windows when initializing DesktopEnv. There is also a Modal provider that uses VM Sandboxes, with commands like modal setup and python -m desktop_env.providers.modal.setup --os Ubuntu. The README warns that abnormal interruptions can leave residual Docker containers, and gives a cleanup command: docker stop $(docker ps -q) && docker rm $(docker ps -a -q). This is a real operational concern; a benchmark run that crashes mid-way can clutter your host.
The OSWorld-Verified update and the scoring shift
A July 2025 update introduced OSWorld-Verified, which fixed community-reported issues and added AWS support to cut evaluation time to under an hour through parallelization. The update also changed the benchmark signals, meaning model results from older versions are not directly comparable to the new ones. The README asks users to compare against the new results on the official website. This is a critical point for anyone who has already published results or is tracking progress over time. The scoring methodology changed enough that you cannot simply carry forward a baseline number. If you adopt OSWorld now, you must use the latest version and re-run any historical comparisons. The repository also notes that VMware Fusion is recommended for Apple Silicon, since KVM is not supported on macOS hosts, which limits the Docker path on a Mac.
Where OSWorld is the wrong tool
OSWorld is a benchmark, not a framework for building agents. It does not provide a model, a training loop, or a reward function that you can tune. Its strength is evaluation, so if you need to train an agent or prototype a quick interaction, this repository will feel heavy. The environment requires a hypervisor, which is a significant dependency. A team without access to virtualization, or one that only has a shared CI server without KVM, will struggle to run even a single episode. Also, the benchmark's open-ended nature means the agent must handle real UI latency and non-deterministic behavior, which can make debugging your agent harder than in a synthetic grid. If your task is strictly a text-based API call or a command-line interaction, OSWorld is overkill; a simpler environment would be more appropriate.
Alternatives and the difference in approach
A common alternative is a simulated desktop environment like MiniWoB or a browser-based benchmark such as WebArena. MiniWoB runs inside a browser and provides a simplified, deterministic web UI with a fixed set of tasks. The difference is that MiniWoB abstracts away the operating system and most of the visual complexity, which makes it faster to run and easier to debug, but less representative of real desktop software. WebArena also focuses on web interactions but on live, realistic sites, which is closer to OSWorld in spirit yet still confined to a browser. OSWorld's use of actual virtual machines with real applications like LibreOffice or GIMP is what sets it apart. You trade setup complexity and resource usage for ecological validity. If you need a quick iteration loop, the simulated options win; if you need to prove an agent can operate a real desktop, OSWorld is the more demanding test.
Maintenance, license, and upgrade cost
The project is under Apache-2.0, which permits commercial use and modification with attribution. The repository is not archived and had a release v0.1.16 in June 2024, with the main branch updated as late as August 2026 according to the metadata. The README shows active development, including the OSWorld-Verified update and new provider support. The upgrade cost is real: the scoring changes mean you must update your evaluation harness when a new version lands. Also, the environment relies on external hypervisors, so you must track VMware or VirtualBox compatibility with your host OS. The desktop-env package on PyPI suggests a modular approach, but you still need to manage VM images and snapshots. The documentation mentions a cache file for pre-downloaded init states, which can reduce setup time, but that is another external dependency to maintain.
Editorial conclusion
Adopt OSWorld if you need to benchmark a multimodal agent against realistic, open-ended desktop tasks where state changes are the success metric. It is a strong fit for academic researchers and industrial teams with access to VMware, VirtualBox, or KVM-capable servers. Skip it if you lack the hardware or time to manage virtual machines and snapshots; a simpler simulated environment would be more practical. Before committing, verify that your host supports KVM or that you can install VMware Workstation Pro, confirm your Python version is at least 3.10, and read the OSWorld-Verified report to understand the recent benchmark scoring changes.
Community notes