Agent S: a computer-use agent that drives a real desktop, and what the Apache-2.0 code actually gives you

The problem Agent S addresses: software that only exposes a GUI

Most automation assumes an interface you can call. A REST endpoint, a CLI, a library. A large amount of desktop software does not offer one. Legacy internal tools, vendor installers, settings panes, and web applications that render everything client side leave you with the same surface a person uses: pixels in, clicks and keystrokes out. Agent S targets exactly that gap. The README describes it as an open source computer use agent framework that takes a natural-language task, looks at the screen, and completes it by clicking, typing and scrolling, with the explicit claim of no API integration and no per-app scripting. The intended audience is stated in the same paragraph: people researching OS agents, people automating their own desktop, and contributors to open source computer use. That is a research framing, not a product framing, and it matters when you decide whether to build on it.

How the loop works: screenshot in, action out, memory in between

The repository topics are the clearest description of the architecture available without reading the source: agent-computer-interface, grounding, gui-agents, in-context-reinforcement-learning, memory, mllm, planning, retrieval-augmented-generation. Read together, they describe a cycle rather than a pipeline. The agent captures the screen, a multimodal model proposes an action, the action is executed against the real desktop, and the resulting screen state feeds the next step. Grounding is the subproblem of turning a model's intent into a coordinate or an element reference that actually lands on the right control, and it is listed separately from planning because the two fail independently. Memory and retrieval-augmented-generation appear as their own topics, which suggests the framework carries state across steps rather than treating each screenshot as an isolated question. The README also names in-context reinforcement learning, which in this context points to learning from prior trajectories inside the prompt rather than updating model weights. Nothing in the supplied material documents the concrete storage format for that memory, how trajectories are selected for retrieval, or how grounding is implemented. Treat those as things to read in the source before you depend on them.

Installing gui-agents and the shape of the setup

The library is published on PyPI under the name gui-agents, not agent-s. The README's own update log ties version numbers to that package: the 2025/08/01 entry for Agent S2.5 is annotated as gui-agents v0.2.5, and the 2024/12/05 entry for v0.1.0 is described as the first release allowing Agent-S to be used for Mac, OSWorld and WindowsAgentArena with ease. So the install path is a pip install of gui-agents, and the releases listed on the repository run v0.3.0, v0.3.1 and v0.3.2, with v0.3.2 dated 2025-12-16. The README advertises macOS, Windows and Linux via badges, and the update log records Linux and Windows support arriving in v0.1.2. Model access is described as working with models from OpenAI, Anthropic and open-weight providers, which means you supply credentials and a model identifier rather than receiving a bundled model. The README does not, in the material provided, spell out the exact Python version, the extra names, or the environment variable keys for those providers. Those details live in the Installation and Setup section of the repository README, which is the first place to look, and the example scripts are the second, because the provider and model string are configured there.

The benchmark claims and what they do not tell you

The headline numbers are specific and attributable. Agent S3 is described as the first computer use agent to surpass human performance on OSWorld, at 72.60 percent, announced 2025/12/15. The 2025/10/02 entry records the initial Agent S3 release at 69.9 percent on OSWorld, with generalizability reported on WindowsAgentArena and AndroidWorld. The linked paper is arXiv 2510.02250, titled Scaling Agents for Computer Use, accepted to TMLR 2026. Separately, the README reports that Sai, Simular's hosted agent built on the ideas in this repo, reached 73 percent on OSWorld 2.0, ahead of GPT-5.6 Sol at 62.57 percent as reported by OpenAI, at lower cost. Two cautions follow from that. First, the 73 percent figure belongs to Sai, a hosted product, not to the open source repository, and the README says so plainly. Second, benchmark success rates are aggregate over a task suite; they say nothing about whether your particular application, locale or window manager is handled. The material gives no per-task breakdown, no variance across runs, and no cost figure for running the framework yourself.

Where this approach breaks down

The failure modes are structural, not incidental. A screenshot-driven agent is a probabilistic controller sitting on top of a deterministic machine. If a dialog appears at step four that the plan did not anticipate, the next screenshot changes and the trajectory can drift; nothing in the framework description guarantees recovery. Grounding failures are the sharpest version of this: clicking four pixels off a button is indistinguishable from clicking the wrong button, and the agent may not notice. Cost is the second constraint. Every step requires a multimodal model call with an image attached, so a fifty-step task is fifty billed inference requests, and the README's cost comparison with Sai is a hosted-service claim rather than a figure for self-hosting. Third, the framework requires an interactive desktop session. That makes headless CI awkward and means the machine being driven is occupied while the agent works. Finally, this is the wrong tool when an API exists. If a service offers an endpoint, calling it is faster, cheaper, testable and does not depend on a model's judgment about where a button is. Agent S is a fallback for the cases where that option is absent, and the README's own framing as a research framework is consistent with that.

Playwright and pyautogui: the same desktop, a different contract

The honest alternative for most readers is not another agent framework but ordinary automation. Playwright drives a browser through a documented protocol and addresses elements by selector, role or text, so a test either finds the element or fails loudly at the line that broke. pyautogui moves the real cursor and sends real keystrokes, and it is deterministic: the same script produces the same sequence every time. The difference in approach is where the intelligence sits. Agent S asks a model to decide what to do next from a screenshot, which generalises to unfamiliar interfaces and is why it can handle an application nobody wrote a script for. Playwright and pyautogui encode the decision once, in code, which is why they are reproducible, reviewable in a pull request, and free to run. The trade is explicit: you pay in brittleness when the UI changes, and you save on inference cost and nondeterminism. A reasonable pattern is to keep Agent S for the exploratory or one-off case and hand the resulting sequence to a script once you know the steps, though the repository material does not describe any trajectory-export feature that would automate that handoff.

Maintenance, version churn and the licence

The release cadence visible in the material is uneven. v0.3.0 and v0.3.1 landed a day apart in early October 2025, then v0.3.2 arrived on 2025-12-16, roughly two and a half months later. The update log shows the project has renamed and reshaped itself repeatedly: Agent S, then S2, S2.5 and S3, with the package staying gui-agents throughout and version numbers drifting between the two naming schemes. That history is the practical maintenance cost. A tutorial written against v0.2.x may not match v0.3.2, and the README's own annotation that S2.5 equals gui-agents v0.2.5 is the kind of mapping you have to reconstruct by hand. Pin the version. Read the release notes for the version you pin. On licensing, the repository is Apache-2.0, which permits commercial use and modification and includes an explicit patent grant; it also requires that you preserve the licence and notice files and state significant changes. That is the general shape of the terms, not legal advice, and if you are redistributing the framework inside a product you should read the LICENSE file in the repository and the notices for any bundled dependencies. The README does not list third-party licence obligations, so that check is on you.

Editorial conclusion

Adopt Agent S if you are researching OS-level agents, reproducing OSWorld or WindowsAgentArena numbers, or need a GUI fallback for applications that expose no API, and you accept that every run costs model tokens and a live desktop session. Do not adopt it for high-volume, deterministic automation of a stable web form; Playwright or pyautogui will be cheaper and reproducible there. Before committing, verify three things against the current README: which Python version and extra the install requires, which provider and model string the example scripts expect, and whether the version you pin matches the Agent S3 code or an older Agent S2 release, since the API changed across v0.2.x and v0.3.x.

Official sources

1. License: Apache-2.0
2. Project website
3. README
4. Releases
5. simular-ai/Agent-S on GitHub