OSWorld: three names, three versions and three dependency lists, over a benchmark that was revised under published results
[NeurIPS 2024] OSWorld: Benchmarking Multimodal Agents for Open-Ended Tasks in Real Computer Environments
At a glance
- What is it?
- A benchmark for multimodal agents working in real desktop environments, published as a paper, an environment package and a task set, with four ways to provide the machines. Its project metadata, its packaging script and its repository name do not agree, its Python floor differs between three files, and its own update log asks readers to disregard previously published scores.
- Who is it for?
- OSWorld fits a research group evaluating an agent on real desktop applications rather than a simulated environment. Four things to know before you run or cite it.
- Can I use it commercially?
- Yes. Apache-2.0 is a permissive licence: you can use, modify and sell software built on it, as long as you keep its copyright and licence notices.
- Is it still maintained?
- Yes. The repository last received commits 1 day ago.
- What is it written in?
- Mainly Python, according to GitHub's language statistics.
Answers come from the project's GitHub data, last synced on November 15, 2026, and from our analysis. They are not legal advice.
Star history
- xlang-ai/OSWorld3,262 ★ · +127
| Repository | 2026-09-10 | 2026-11-15 |
|---|---|---|
| xlang-ai/OSWorld | 3.1k | 3.3k |
127 Stars in the last 66 days, measured from the daily GitHub snapshot. Compare with other projects
Editorial analysis
Three names and three versions for one project
Follow the packaging and you will find three identities. The repository is one project. The project metadata in the repository declares a second name and a third version, and its description field is still the scaffolding placeholder text rather than a real description. The setup script declares a third name entirely, at a third version, and its project address points at a different repository than the one you are reading. That is not a mistake in one file: the README tells you that you can install the environment from a package index, with no benchmark tasks included, which is the same distribution the setup script publishes. So a reader who installs from the index, a reader who clones and installs, and a reader who reads the metadata are looking at three different names and three different version numbers, and the release tags sit at a fourth.
The benchmark was revised under published results and the readme says so
The updates section is the most important paragraph in the README for anyone citing a number. An entry dated in mid 2025 announces a verified revision of the benchmark, saying they made major updates, fixed several issues reported by the community, added more support for one cloud platform with parallelisation that can reduce evaluation time to within an hour, and made the benchmark signals more effective. It then says new model results were run on the latest version and published on the project site, and asks you to compare your results against the new benchmark results when running the latest version. That is the project admitting that earlier numbers were measured against a different instrument. The earlier entries in the same list show the usual shape of a research artefact: a paper released alongside the environment, then a refactor of the virtual machine integration, then a hosted provider added, then the revision.
Three Python floors across three files
The language requirement is stated three times and it is not the same number twice. The installation section says to ensure the version of Python is at least 3.10 and recommends a recent version of a common environment manager, with a commented example creating a 3.10 environment. The setup script declares 3.10 or newer as its requirement. The project metadata declares 3.12 or newer. The repository also carries a tool version file and a directory environment file, so the intended interpreter is meant to be pinned somewhere too. This matters more than the usual version drift because the newest floor excludes interpreters that the documentation's own commented example creates, so the two instructions cannot both be followed. It also means an installation that resolves from the metadata will refuse a machine that satisfies the documentation.
Three dependency lists, and the modern one is the loosest
Dependencies are declared in three places: the project metadata, the setup script's requirement list, and a requirements file at the repository root. They do not agree. The metadata leaves many entries unpinned where the other two put compatible-release pins on the same packages, and the requirements file contains packages the metadata does not list at all, including an agent framework, an input library, three cloud provider toolkits, a JSON repair library, two web scraping or search helpers and an experimental model client. It also lists one headless build of an image library twice. The requirements file is what the installation section tells you to install, so the pinned set is the one you get by following the README, and the loose set is the one a modern installer would read. Both are shipped, and the reproducibility of an evaluation run depends on which one was used.
Installing from source downloads browser binaries
The setup script does something unusual for a Python package: it replaces the standard install command with a subclass that, after the ordinary installation finishes, shells out to install the browser binaries the automation library needs. It wraps that call so a failure prints a message telling you to run the step manually rather than aborting. That is a reasonable convenience, and it is also an install that reaches the network and downloads a browser into your machine's cache as a side effect of adding a dependency. For a benchmark whose whole point is measuring an agent operating a real browser, the browser is not incidental, so this is defensible. It does mean that installing the environment package on a machine you wanted to keep clean will leave browser binaries behind.
Four providers, two of which put the desktop on someone else's machine
The installation section documents four ways to provide the machines, and the split matters. On bare metal you install a desktop virtualisation product, configure its command line tool, verify it by listing running machines, and the project's setup script downloads the virtual machines and configures the environment for you. On a server the recommendation is a container, and you are told to check hardware virtualisation support first with a processor flag count, The support check is one command:
egrep -c '(vmx|svm)' /proc/cpuinfowith a note that desktop systems generally do not offer it and that you should use the desktop virtualisation product on those. The other two are hosted: one runs the machines in a serverless sandbox platform after you stage a disk image into it, and the other runs inside a sandbox where the sandbox itself is the desktop, so it needs no local virtualisation and no image flow at all. Both of those hand the evaluated desktop to a third party.
The documentation lives on one account and the cache on a drive
The link row at the top of the README is a map of where this project actually lives. The website is on the project's own pages domain, the paper is on an archive, the task data is a directory in this repository, and there is a hosted data viewer. The documentation, though, is on a personal account's pages site rather than the project's domain, so the docs for the benchmark live on somebody's individual profile. The virtual machine images are not linked from the repository at all; a later update added a link to a file on a consumer drive hosting pre-downloaded files for environment setup. That is a distribution chain with three different hosts, and it is worth knowing that a benchmark's environment is not obtainable from the artefact itself.
Both OpenCV builds, three OCR stacks and four PDF libraries in one manifest
The dependency list is where a benchmark's task coverage becomes visible, and it is long enough to show its priorities by weight. There are three optical character recognition stacks side by side, four separate PDF libraries, four office document libraries, two browser automation libraries, a screen input library, a window management library, an audio library, an experiment tracking service, a drive downloader, and clients for at least six model providers on three continents. Two entries are worth naming individually because they conflict: the manifest requires both a full build of an image library and its headless build, which install the same module differently. The root requirements file repeats one of them twice. Neither is fatal, and both are the kind of thing that turns a fresh install into an afternoon of dependency resolution.
Editorial conclusion
OSWorld fits a research group evaluating an agent on real desktop applications rather than a simulated environment. Four things to know before you run or cite it. The benchmark was revised in mid 2025 and the update log explicitly asks you to compare against the new numbers, so any earlier score is not comparable. The project names and versions disagree across three files, so read carefully which artefact you are installing before quoting a version. There are three dependency lists and the modern one is the loosest, which makes a fresh install of the environment different from a pinned one. And two of the four machine providers put the desktop on someone else's infrastructure, so the evaluation environment is not entirely yours.
Frequently asked questions
What is OSWorld?
A benchmark for multimodal agents on open-ended tasks in real computer environments, published as a paper, an environment package and a task set, with a hosted data viewer. Runs launch real virtual machines and score tasks across desktop applications, and the results are published on the project site.
What is the difference between OSWorld and OSWorld Verified?
Verified is a later revision of the benchmark. The updates section says it made major changes, fixed community-reported issues, added more support for one cloud platform with parallelisation that can cut evaluation time to within an hour, and made the benchmark signals more effective, and it asks readers to compare results only against the new numbers.
How do I run OSWorld?
Four documented providers. On bare metal it uses a desktop virtualisation product and the setup script downloads the machines for you. On a server it runs in a container after checking processor virtualisation support. Two hosted providers put the machines on someone else's infrastructure: one stages a disk image into a serverless sandbox platform, the other uses a sandbox as the desktop itself so no host virtualisation is needed.
What are the Python version requirements for OSWorld?
The installation section says 3.10 or newer, the setup script declares 3.10 or newer, and the project metadata declares 3.12 or newer. The repository also carries a tool version file and a directory environment file, so the intended interpreter is meant to be pinned elsewhere too.
What is the desktop-env package?
A separately published distribution for setting up and evaluating desktop automation tasks, at version 1.0.1, whose project address points at a different repository than the one hosting the benchmark. The README says it can be installed from the package index if you want the environment without the benchmark tasks.
What happens when OSWorld is installed from source?
The setup script replaces the standard install command with one that, after the ordinary install, shells out to install the browser binaries the automation library needs, and prints a message telling you to run that step manually if it fails. So an install reaches the network and leaves browser binaries in a local cache.
Official sources
Add this badge to your README
If you maintain this project, the badge below links readers to this analysis and shows its maintenance status from the daily GitHub snapshot. Paste the markdown into your README; add ?metric=license or ?metric=stars to the image URL for a different field.
[](https://hysenlabs.com/projects/xlang-ai-osworld)Star history chart for your README
[](https://hysenlabs.com/projects/xlang-ai-osworld)Was this useful?