Windows-Use: a GUI agent that reads the accessibility tree instead of pixels

The problem: Windows automation that survives a UI redesign

Most Windows automation is written against coordinates, window titles, or control IDs. All three break. A button moves twenty pixels and the script clicks the background. A vendor renames a dialog and the window-title match fails silently. Windows-Use takes a different position: instead of hardcoding where things are, it reads the Windows UI Automation API at run time to discover what is currently on screen, then asks an LLM to choose the next action. The README states the agent "reads the screen via the Windows UI Automation API and uses any LLM to decide what to click, type, scroll, or run", and adds that no computer vision model is required. That last clause is the design commitment worth noting. Many GUI agents work from screenshots and a vision model; this one defaults to the accessibility tree, with screenshots available as an opt-in through use_vision. The intended user is a Python developer who needs to drive a Windows application that has no usable API, on a machine where an LLM call is acceptable latency.

How the agent decides what to do: accessibility tree, then LLM, then tool call

The data flow visible in the README is a loop. The agent inspects the current UI state (accessibility tree by default, screenshots if use_vision is enabled), sends that state plus the task and the conversation history to the configured LLM, receives a tool call, executes it, and repeats. The Agent constructor exposes the knobs for that loop directly: mode="normal" keeps full context while mode="flash" is described as lightweight and faster, use_accessibility=True turns the accessibility tree on, and use_annotation=False controls whether UI elements are labelled on screenshots. Two stop conditions are configurable: max_steps caps the number of steps before giving up, and max_consecutive_failures aborts after N consecutive tool failures. The default in the configuration example is max_steps=25 and max_consecutive_failures=3, while the CLI documents a different default of 200 for --max-steps. That discrepancy between the library default and the CLI default is worth knowing before you tune anything, because a task that terminates early under one entry point may run much longer under the other. The tools themselves are not user-selected: the README says the agent "has access to these tools automatically, no configuration needed", listing click_tool, type_tool, scroll_tool, move_tool, and further tools for PowerShell, filesystem access, browser scraping via the accessibility tree, and virtual desktop management.

Installation and the smallest working invocation

The prerequisites are Python 3.10 or newer and Windows 7 through 11. Installation is a single pip command, with uv given as an alternative:

pip install windows-use uv add windows-use

The README's quick-start examples follow the same three-line shape across providers. For Anthropic:

from windows_use.providers.anthropic import ChatAnthropic from windows_use import Agent, Browser

llm = ChatAnthropic(model="claude-sonnet-4-5") agent = Agent(llm=llm, browser=Browser.EDGE) agent.invoke(task="Open Notepad and write a short poem about Windows")

The browser argument takes Browser.EDGE, Browser.CHROME, or Browser.FIREFOX, and matters because one of the tools scrapes web pages through the browser's accessibility tree. An async path exists through ainvoke, which returns an object whose content attribute holds the result. The provider list is broad: Anthropic, OpenAI, Google, Groq, Ollama, Mistral, Cerebras, DeepSeek, Azure OpenAI, Open Router, LiteLLM, NVIDIA, and vLLM, each with its own import path under windows_use.providers. The Ollama example is the one that runs without a hosted API key, and it sets use_vision=False, which is consistent with the accessibility-tree-first design. The README also recommends claude-haiku-4-*, claude-sonnet-4-* or claude-opus-4-* models for best results, which is a hint about where the author's own testing concentrated.

The CLI and what the session commands actually change

Running windows-use with no arguments starts an interactive session. The documented options are --model or -m for the model, --provider or -p for the provider, --max-steps for the per-task step ceiling (documented default 200), and --debug or -d for debug logging. Inside a session, backslash commands switch configuration without restarting: \llm changes provider or model, \key changes the API key, \speech configures speech-to-text and text-to-speech, \voice records voice input, \clear clears the screen, and \quit exits. The presence of \key is the practically important one. It means credentials can be entered at the prompt rather than baked into a script, which is a meaningfully different operational posture from the library examples, where the API key is presumably read from the environment by the provider class. The README does not spell out the environment variable names for any provider, so that detail has to come from the provider packages themselves. Voice input and output are listed as capabilities, with the \speech and \voice commands as the configuration surface, but the README gives no example of a speech configuration and no list of supported engines.

Where this breaks: ambiguity, cost, and applications with no accessible tree

The central limitation is the one the architecture implies. An LLM choosing each action means the same task can take different paths on different runs, and a task that succeeds once is not guaranteed to succeed the next time. For anything with a compliance or reproducibility requirement, that is disqualifying, and no configuration flag in the README changes it. The second limitation is the accessibility tree itself. Applications that paint their own UI without exposing UI Automation elements, which includes some games, some Electron apps, and some remote-desktop surfaces, give the agent little to work with. use_vision=True is the escape hatch, but it inverts the project's main advantage: you are back to screenshot interpretation, with the associated token cost and latency. Third, the step budget is a real constraint rather than a formality. With max_steps=25 in the configuration example, a multi-window task will hit the ceiling, and the failure mode is a truncated run rather than a clean error. Fourth, every step is an LLM round trip, so a long task costs both wall-clock time and API spend proportional to the number of steps. Fifth, the platform lock is absolute: this is Windows 7 through 11 only, and the accessibility-tree approach has no direct equivalent on macOS or Linux. Finally, the README marks the file, memory and multi-select tools as experimental behind experimental=False, so the persistent-memory capability advertised in the feature list is not on by default.

The alternative: PyAutoGUI and scripted automation

The obvious comparison is PyAutoGUI, which also drives the mouse and keyboard on Windows but does so through explicit coordinates and explicit calls. The difference is where the intelligence sits. With PyAutoGUI, you write click(100, 200) and the script does exactly that, every time, in milliseconds, with no API key and no model. With Windows-Use, you write a task in English and the agent decides what click(100, 200) should be on this particular run. PyAutoGUI is deterministic, free per action, and portable across operating systems; Windows-Use tolerates UI changes that would break a coordinate script, and handles tasks whose steps you cannot enumerate in advance. Neither is strictly better. If your target application has stable layout and you can describe the sequence, PyAutoGUI is the cheaper and more predictable choice, and Windows-Use adds a model dependency you do not need. Windows-Use earns its place when the sequence is genuinely unknown at authoring time, or when the UI shifts often enough that maintaining a coordinate script costs more than the LLM calls. A second alternative worth naming is the vendor's own automation interface, where one exists: the Windows-Use README itself lists running PowerShell commands as one of the agent's tools, which is an admission that for anything PowerShell can do directly, going through the GUI is the long way around.

Maintenance, licensing, and what to check before you depend on it

The licence is MIT, which permits commercial and closed-source use and requires only that the copyright notice and permission notice be preserved. That is a permissive baseline, and nothing in the README suggests additional terms, but licence questions about your own distribution are for your own counsel, not for this review. On maintenance: the repository is not archived, the last push recorded is 2026-07-07, and the three most recent releases (v0.7.9, v0.8.0, v0.8.1) all landed on 2026-04-30. Three releases on one day suggests a batch of fixes rather than a steady cadence, and the gap between that batch and the last push is roughly two months. There is no published roadmap in the material, so upgrade cost is hard to project. The practical exposure is provider drift: thirteen provider integrations means thirteen SDKs that can change their model names and authentication, and the README's own model recommendations (claude-haiku-4-*, claude-sonnet-4-*, claude-opus-4-*) will age as those model lines move. Pinning the package version and the provider SDK version together is the conservative move. The absence of a homepage means the README and the repository are the documentation, and the README is where you will find the tool list, the configuration keys, and the CLI options, all of which are reproduced above.

Editorial conclusion

Adopt Windows-Use if you are automating Windows-only desktop work on a machine you control, and you are willing to accept an LLM in the loop deciding each click. Do not adopt it if the task must be deterministic, auditable, or portable to macOS or Linux; for that, a scripted tool such as PyAutoGUI is the better starting point. Before committing, verify three things against your own environment: whether your target application exposes a usable accessibility tree, whether the default max_steps=25 budget is enough for your task, and whether the provider you intend to use is one of the thirteen listed in the README.

Official sources

1. Issues
2. Jeomon/Windows-Use on GitHub
3. License: MIT
4. README
5. Releases