Agent S and the gap between its benchmark claims and its pip package

A GUI agent with no API integration and no per-app scripting

The core claim is a specific interaction model. Agent S takes a natural-language task, looks at the screen, and completes the task by clicking, typing, and scrolling in ordinary desktop and web applications. That screen-level approach is the real difference between it and a tool that wraps one application's endpoints.

That design has a matching boundary. Because the screen is the agent's only interface, the target is an environment where a desktop exists: it runs on macOS, Windows, and Linux. A headless server with no display, a scheduled cron job, or a workflow that would be cleaner as one HTTP call sits outside what running it on your own computer can do; the README's no-setup option instead runs Sai on a cloud computer through the Simular API.

The package is gui-agents and the CLI reaches into the s3 module

What you install is not named Agent S. setup.py declares name="gui-agents" at version 0.3.2, authored by Simular AI with the contact [email protected], and gives the description as a library for creating general purpose GUI agents using multimodal LLMs. The repository, the distribution, and the import path are three separate strings: Agent-S, gui-agents, and gui_agents.

The entry point shows which generation the command follows, through a console_scripts mapping of agent_s=gui_agents.s3.cli_app:main. So the executable is named agent_s while the module it dispatches into is the s3 subpackage, and the tags v0.3.0, v0.3.1, and v0.3.2 track that distribution rather than the repository. Look for gui-agents when you install and gui_agents when you import.

python_requires stops at 3.12 while the classifiers stop at 3.9

Version support is declared twice, and the two declarations do not line up. python_requires is written as >=3.9, <=3.12, while the classifier list names Programming Language :: Python :: 3 and Programming Language :: Python :: 3.9 and stops there.

The effect is a hard ceiling at 3.12 and a floor at 3.9, neither of which the classifiers convey. On a machine running 3.13 the install is refused by the constraint even though the project would still file that interpreter under Python 3, and on 3.9 it installs cleanly even though 3.9 is the only point release anyone recorded. If your environment pins a newer interpreter, this package does not become a candidate until that range widens.

Conditional pyobjc, pywinauto and pywin32 make three different installs

Three dependencies are gated on the operating system. install_requires carries pyobjc with the marker platform_system == "Darwin", plus pywinauto and pywin32 each marked platform_system == "Windows" and annotated with an inline comment reading Only for Windows. requirements.txt repeats all three markers under a Platform-specific dependencies heading, and additionally lists black.

So the dependency set is one list or three depending on where you resolve it, and a lockfile captured on Linux will not reproduce the same tree on a Mac or a Windows machine, because the Darwin and Windows packages never enter the Linux resolution at all. It also means the pyobjc and pywin32 code paths can only be exercised on their own platforms, and the install metadata offers a Linux user no way to check them.

paddleocr and paddlepaddle are required, not extras

The OCR stack is not optional. install_requires includes paddleocr, paddlepaddle, and pytesseract, so two separate OCR engines plus the PaddlePaddle runtime arrive with a plain install. The optional set is minimal, containing only black under dev, which an inline comment identifies as a code formatter for linting.

The consequence is install weight paid whether or not a given run ever reads text off a screen. If your tasks are pure pointer work you still pull the Paddle runtime, and there is no extras_require entry that lets you decline it. Sitting alongside those are fastapi, uvicorn, and websockets, along with both selenium and pyautogui, which implies a server surface and two separate automation paths without saying which one handles which platform.

Two benchmark numbers from two benchmarks, neither shipped in the package

The headline result and the released package are about different things. The front page leads with Sai, described as Simular's production computer use agent, at 73% on OSWorld 2.0, announced on 2026/08/28 as beating GPT and Opus at lower cost. Separately, the update log records Agent S3 at 72.60% on OSWorld in December 2025, and Agent S4 at 21.3% on OSWorld V2 with Qwen3.8-27B in October 2026.

Those are separate benchmarks: the 73% belongs to the hosted Sai, while the README's version table says Agent S3, the 72.60% result, is the agent in gui-agents today. The newest tag is v0.3.2 from 2025-12-16, one day after the 72.60% entry, and the last push to main was 2026-10-30.

Cloud is the offered off-ramp, and the tree is built around benchmarks

There is an explicit exit from setup: the Quick Start's no-setup option runs Sai, Simular's production agent, on a cloud computer through the Simular API, while running on your own computer points to the Agent S3 README. The project draws that distinction itself rather than leaving it implied.

The tree shows what the framework side holds. Alongside gui_agents/, requirements.txt and setup.py, there is agent_s4/ (a separate distribution per setup.py), integrations/, models.md, evaluation_sets/, tests/, and images/, plus osworld_setup/, WAA_setup.md and scienceboard_setup/ named for benchmark environments. A large share of the repository is therefore harness for reproducing published numbers rather than library code, which tells you what to expect when you open an issue: benchmark reproduction is the visible work, and the shipped surface is a single console entry point.

Editorial conclusion

Agent S fits researchers and teams automating a real desktop, provided your interpreter is between Python 3.9 and 3.12 and you accept that paddlepaddle arrives with it. It does not fit headless servers or problems an API would solve, because the whole design is clicks and keystrokes on a visible screen. Before installing, check python_requires, and note that the 73% OSWorld 2.0 figure belongs to the hosted Sai rather than to gui-agents 0.3.2.

Frequently asked questions

Official sources

1. License: Apache-2.0
2. Project website
3. README
4. Releases
5. simular-ai/Agent-S on GitHub