OSWorld-V2: a release-pinned benchmark for computer use agents, and what each upgrade makes you redo
OSWorld 2.0: Benchmarking Computer Use Agents on Long-Horizon Real-World Tasks
At a glance
- What is it?
- OSWorld-V2 from xlang-ai benchmarks computer use agents on long-horizon real-world tasks. The work is in keeping code, task files, assets, mocked websites and VM images on the same release tag.
- Who is it for?
- OSWorld-V2 is for researchers and teams who can run Linux-oriented evaluation infrastructure, self-host mocked websites and keep every component on one release tag. It is a poor fit for anyone who wants a quick leaderboard check or a single-command demo.
- Can I use it commercially?
- Yes. Apache-2.0 is a permissive licence: you can use, modify and sell software built on it, as long as you keep its copyright and licence notices.
- Is it still maintained?
- Yes. The repository last received commits 2 days ago.
- What is it written in?
- Mainly Python, according to GitHub's language statistics.
Answers come from the project's GitHub data, last synced on November 15, 2026, and from our analysis. They are not legal advice.
Star history
- xlang-ai/OSWorld-V2421 ★ · +118
| Repository | 2026-09-13 | 2026-11-15 |
|---|---|---|
| xlang-ai/OSWorld-V2 | 303 | 421 |
118 Stars in the last 63 days, measured from the daily GitHub snapshot. Compare with other projects
Editorial analysis
Three releases are active, and only osworld-v2.1 is the recommended one
The version table in the repository lists three benchmark releases, all with status Active: osworld-v2.1, osworld-v2-2026.08.08 and osworld-v2-2026.06.24. Only osworld-v2.1 carries the label recommended. It was released on 2026-09-16 as a bug-fix release with updated agents, evaluation workflows, and pinned task, asset, website and provider image references.
The main branch is a different thing. A warning in the project's instructions calls it the development version and tells you to use a supported release for reproducible benchmark evaluation. In practice that means you pick a tag, not a branch. Releases also appear on the GitHub releases page as osworld-v2.1, v2026.08.08 and v2026.06.24, so the tag names do not all follow one pattern, and you should copy the exact name from benchmark_releases before typing it into a command.
Upgrading to 2.1 means a detached checkout of a tag, not a pull
Moving from OSWorld 2.0 to 2.1 starts with saving local changes, then checking out the release tag in a detached state:
git fetch origin tag osworld-v2.1
git switch --detach osworld-v2.1
uv sync --frozenBecause the checkout is detached, any edits you commit there are not on a branch. If you maintain local agent code, keep it on your own branch and rebase onto the tag, otherwise the next tag switch can leave your commits hard to find. The `uv sync --frozen` step installs from the lock file, so it will not resolve newer versions on its own. The earlier 2026.08.08 upgrade changed the uv dependency files to require `imageio-ffmpeg>=0.6.0`, which is why an existing host environment from the 0624 release needs the sync before it will run.
Task files and assets come from two Hugging Face repositories, one of them gated
Task files are published as a Hugging Face dataset, xlangai/osworld_v2_tasks, and the 2.1 copy is fetched with `uv run scripts/tools/download_osworld_v2_tasks.py --benchmark-release osworld-v2.1`. The complete assets live in a separate gated dataset, xlangai/osworld_v2_assets_gated, so you need access approved before the download will work. The download goes into a fresh directory:
uv run scripts/tools/download_osworld_v2_assets.py \
--benchmark-release osworld-v2.1 \
--target-dir cache/osworld_v2_assets_v2.1
export OSWORLD_FILE_BASE_URL="$(pwd)/cache/osworld_v2_assets_v2.1"
export OSWORLD_BENCHMARK_RELEASE="osworld-v2.1"The fresh directory matters. Pointing OSWORLD_FILE_BASE_URL at an older assets folder mixes releases. The public asset set, xlangai/osworld_v2_assets, is pinned by the code and needs no separate download, but a custom mirror of it must be moved to the same tag.
Tasks run against mocked websites that you may have to host yourself
Many tasks use mocked websites from a separate repository, Task-Web/OSWorld-web. For 2.1 you update that checkout to the osworld-v2.1 tag, run `git submodule sync --recursive` and `git submodule update --init --recursive`, redeploy it, and set `export WEBSITE_HOST_SUFFIX="<your-v2.1-host-suffix>"` on the evaluation host.
Hosting differs by release. The osworld-v2-2026.08.08 release uses the team-hosted suffix site.hku.icu. Since 2026-09-08 the team no longer hosts the osworld-v2-2026.06.24 websites at web.hku.icu, so that release works only if you self-host v2026.06.24 of the website repository. Anyone who needs to reproduce an old result therefore takes on a deployment task before the first run, and a wrong suffix will break the tasks that depend on those sites.
Docker keeps an old VM around unless you move docker_vm_data
Provider images are pinned per release. For Docker, 2.1 uses xlangai/[email protected] and a pinned runtime digest. The instructions say to stop the old VM and move the existing `docker_vm_data` to a backup location before the first run, so that the matching image is downloaded. If you skip that, the old VM data stays in place and the run may start from the wrong image.
For AWS the guidance is to reuse ami-01017272139e01feb in us-east-1, with a 1920x1080 screen, and no new AMI is needed. The provider images were unchanged between the 0624 and 0808 releases, so the image work only applies when you cross into 2.1. The consequence is that a Docker user upgrading carries a disk cleanup step, while an AWS user mostly changes the asset path, website suffix and credentials, then runs a small evaluation as a smoke check.
The dependency list pins old library versions and targets Python 3.12 on Linux
pyproject.toml names the package osworld at version 0.1.0 and requires Python 3.12 or newer. Its classifiers list Operating System :: POSIX :: Linux and Development Status :: 4 - Beta. The dependency list is wide: client libraries for openai, anthropic, google-genai, groq, together and zhipuai sit next to boto3, playwright, librosa, pymupdf and pytesseract.
Several pins are old, such as gymnasium~=0.28.1, requests~=2.31.0, matplotlib~=3.7.4, flask~=3.0.0 and opencv-python~=4.8.1.78. Installing into an environment you already use for other projects risks version conflicts, so give the benchmark its own environment. requirements.txt is only a shim containing `-e .`, with pyproject.toml as the source of truth, and uv.lock is what `uv sync --frozen` consumes.
A score only means something next to its release tag
Each release pins its own task files, assets, mocked websites and provider images, and the 2.1 release also changed agents and evaluation workflows. So an OSWorld-V2 number from one tag is not the same measurement as a number from another. When you publish results, name the release, the asset directory and the website version you used.
The repository gives you supporting material: a paper, a project site at osworld-v2.xlang.ai, a Trajectory Viewer, and a Trajectory Download dataset on Hugging Face. The code is organised around main.py, run.py, desktop_env, mm_agents, hybrid_agents and evaluation_examples. The paper is also checked in as a PDF. The repository is Apache-2.0 licensed, and the last push was on 2026-10-23. If you only want to inspect how existing agents behaved, the trajectory downloads let you read recorded runs without rebuilding the whole environment, which saves the website and image setup described above.
Editorial conclusion
OSWorld-V2 is for researchers and teams who can run Linux-oriented evaluation infrastructure, self-host mocked websites and keep every component on one release tag. It is a poor fit for anyone who wants a quick leaderboard check or a single-command demo. Before starting, decide which release you will report against, because tasks, assets, websites and images are all pinned to it.
Frequently asked questions
What is OS World?
OSWorld-V2 in this repository is described as a benchmark for computer use agents on long-horizon real-world tasks. The repo holds the environment, task loading, agents and evaluation scripts.
What is OSWorld verified?
The OSWorld-V2 repository lists three benchmark releases, osworld-v2.1, osworld-v2-2026.08.08 and osworld-v2-2026.06.24. Check the project website to see how any other OSWorld variant relates to them.
What is OSWorld SFT?
The OSWorld-V2 repository is a benchmark for evaluating computer use agents, with run.py and evaluation scripts. Fine-tuning data for the benchmark is not something the repository's release list describes.
Official sources
Add this badge to your README
If you maintain this project, the badge below links readers to this analysis and shows its maintenance status from the daily GitHub snapshot. Paste the markdown into your README; add ?metric=license or ?metric=stars to the image URL for a different field.
[](https://hysenlabs.com/projects/xlang-ai-osworld-v2)Star history chart for your README
[](https://hysenlabs.com/projects/xlang-ai-osworld-v2)Was this useful?