OSWorld-V2: A Long-Horizon Benchmark for Computer Use Agents
OSWorld 2.0: Benchmarking Computer Use Agents on Long-Horizon Real-World Tasks
At a glance
- What is it?
- OSWorld 2.0 is a Python benchmark for evaluating computer use agents on long-horizon, real-world desktop tasks, with release-pinned task files, assets and mocked websites. The core judgement: adopt it if you need a reproducible target for agent evaluation, but plan for version-locked infrastructure and a gated asset download.
- Who is it for?
- Adopt OSWorld-V2 if you are evaluating computer use agents on long-horizon desktop tasks and need a release-pinned, reproducible target; do not adopt it if you want a lightweight unit-test suite or a benchmark you can run from a single pip install, because the task assets are gated and the mocked websites must match the release.
- Can I use it commercially?
- Yes. Apache-2.0 is a permissive licence: you can use, modify and sell software built on it, as long as you keep its copyright and licence notices.
- Is it still maintained?
- Yes. The repository last received commits 2 days ago.
- What is it written in?
- Mainly Python, according to GitHub's language statistics.
Answers come from the project's GitHub data, last synced on September 15, 2026, and from our analysis. They are not legal advice.
DEEP OPEN-SOURCE ANALYSIS
What OSWorld-V2 Is For and Who Should Care
OSWorld 2.0 is a benchmark for computer use agents, meaning systems that operate a desktop through a GUI rather than through an API. The repository describes it as benchmarking computer use agents on long-horizon real-world tasks, which places it in a different category from short-horizon GUI benchmarks where a single click or a two-step form fill is the unit of evaluation. The target audience is research and evaluation teams that need a fixed target against which agent versions can be compared. The paper, environment and benchmark were released on 2026-06-26, and the project page and trajectory viewer are separate hosted services. If your work involves training, comparing or reporting on agents that complete multi-step desktop workflows, this is the kind of harness the repository is built to provide. If you only need to test a small set of scripted UI interactions, the full provider and asset machinery here is more than the task requires.
Release Pinning as the Central Design Constraint
The most consequential thing about OSWorld-V2 is not the task format but the release discipline. The README states that for reproducible evaluation you must use the code, the Hugging Face Python task files, the Hugging Face task assets and the mocked website deployment from the same release. Four components move together: the repository tag, the task files at xlangai/osworld_v2_tasks, the assets at xlangai/osworld_v2_assets_gated, and the website deployment at Task-Web/OSWorld-web. The current recommended release is osworld-v2-2026.08.08. The README explicitly warns against mixing releases or replacing a pinned tag with main or latest, and notes that main is a development branch. This is a deliberate trade-off: it buys comparability at the cost of setup friction. A benchmark whose score depends on which website build was live is not a benchmark, so the pinning is justified, but it means every evaluation run is an infrastructure exercise before it is a measurement exercise. The 2026.08.08 release reuses the provider VM images from 2026.06.24, which the README says are unchanged between the two releases. That is the one place where the version matrix does not force a full re-provision.
The Mechanism: Tasks, Assets, Mocked Sites and Provider VMs
The architecture visible in the repository splits into four moving parts. First, Python task files downloaded from the xlangai/osworld_v2_tasks dataset define what each task is. Second, task assets from xlangai/osworld_v2_assets_gated provide the files and resources those tasks operate on. Third, mocked websites deployed from Task-Web/OSWorld-web supply the web surfaces the agent interacts with, either hosted by the team under a suffix such as site.hku.icu or self-hosted. Fourth, environment providers supply the actual desktop, with Docker listed for Linux in the provider setup documentation. The data flow is therefore: the harness loads a task definition, points the agent at a provider-backed desktop, and the agent acts against both the local environment and the mocked websites. The trajectory viewer and the downloadable trajectory dataset on Hugging Face suggest results are captured as full interaction traces rather than single scores, which is consistent with long-horizon evaluation where the path matters as much as the outcome. The task assets are gated, meaning access is controlled on the Hugging Face side, which is a real logistical step rather than a formality.
Getting a Release Running: Commands and Environment Variables
The README gives a concrete migration path from osworld-v2-2026.06.24 to osworld-v2-2026.08.08. Task files come down with uv run scripts/tools/download_osworld_v2_tasks.py --benchmark-release osworld-v2-2026.08.08. Assets come down with uv run scripts/tools/download_osworld_v2_assets.py --benchmark-release osworld-v2-2026.08.08 --target-dir cache/osworld_v2_assets --clean, after which you export OSWORLD_FILE_BASE_URL="$(pwd)/cache/osworld_v2_assets". The checkout itself moves to the tag xlang-ai/OSWorld-V2@v2026.08.08, and self-hosted websites move to Task-Web/OSWorld-web@v2026.08.08. When using the team-hosted websites, the README shows export WEBSITE_HOST_SUFFIX="site.hku.icu". The host environment needs uv sync --frozen, or uv pip install "imageio-ffmpeg>=0.6.0" if you are updating an existing environment, because the dependency files changed and now require imageio-ffmpeg 0.6.0 or later. Package setup requires Python 3.12 or above through pyproject.toml, and uv sync --extra full pulls heavier agent or OCR stacks, which the README notes are only for v1 tasks. There is also a setup-osworld skill intended to be driven by an agent with a single prompt, but the README's sample prompt asks the agent to confirm provider choices and pause before any cloud spend, DNS, SSH or secret step. Treat that skill as a bootstrap, not as an unattended installer.
Where This Benchmark Fails You
The failure modes are mostly operational. The README records that the team no longer hosts the osworld-v2-2026.06.24 websites at web.hku.icu, so anyone evaluating that release must self-host Task-Web/OSWorld-web@v2026.06.24. That is a concrete example of hosted infrastructure being withdrawn, and it should shape how much you depend on the team-hosted suffix. The gated asset download adds an access-control step that can block a fresh environment before any agent runs. The provider images are pinned by release, and while 2026.08.08 reuses the 2026.06.24 images, older or newer combinations are not covered by that guarantee. The README also warns that the benchmark release and the development branch are different things, so a checkout tracking main will not reproduce a published score. Finally, this is the wrong tool when the question is whether an agent can complete a single well-defined UI action. The long-horizon framing, the asset download, the mocked website deployment and the provider VM provisioning are all costs that only pay off when the task genuinely spans many steps.
OSWorld 1.0 and the Migration Path
The most direct alternative is OSWorld 1.0 at xlang-ai/OSWorld. The difference in approach is not cosmetic: OSWorld 2.0 is organised around release-pinned components, with task files, assets, mocked websites and code all moving as a set, and the README points to docs/MIGRATING_FROM_OSWORLD_V1.md for a checklist covering dependency setup, task conversion, provider reuse, AWS notes, mocked websites, GitLab, agent migration and result comparability. That checklist exists because the two versions are not drop-in compatible, and the README notes that the optional full dependency set is only for v1 tasks, which suggests v1 tasks carry heavier agent or OCR requirements. If you have existing v1 results, the migration is a project rather than a version bump, and the README's emphasis on result comparability implies that scores across the two versions should not be treated as interchangeable. For teams starting fresh, the release-aware download scripts and the pinned manifest in 2026.08.08 are the cleaner starting point.
Maintenance, Licensing and Upgrade Cost
The repository is not archived, and the last push was on 2026-09-11, two days before this review. Releases are dated v2026.08.08 and v2026.06.24, and the README's update log shows website hosting changes as recently as 2026-09-08. That cadence matters for planning: the benchmark releases are dated snapshots, so an evaluation quoted without a release tag is not comparable to one with a tag. Upgrading between releases means re-downloading task files and assets, re-pointing the code checkout, updating the website deployment or suffix, and re-syncing host dependencies, as the 0808 migration instructions show. The provider VM images are the one component that did not change between 0624 and 0808, so image provisioning is not automatically part of every upgrade. The licence is Apache-2.0, which permits commercial and academic use and modification, and the repository links the standard licence text. Note that the licence covers the code; the task assets are distributed separately through a gated Hugging Face dataset, and the mocked websites live in a different repository. Those separate distributions may carry their own terms, and the README does not state what they are. If asset or website licensing matters to your deployment, that is a question for the dataset and website repositories rather than the Apache-2.0 grant here.
Editorial conclusion
Adopt OSWorld-V2 if you are evaluating computer use agents on long-horizon desktop tasks and need a release-pinned, reproducible target; do not adopt it if you want a lightweight unit-test suite or a benchmark you can run from a single pip install, because the task assets are gated and the mocked websites must match the release. Before committing, verify that the 2026.08.08 task files, assets, code checkout and website suffix all point at the same release tag, and confirm you can self-host Task-Web/OSWorld-web if the team-hosted suffix is unavailable.
Community notes