AskUI Python SDK: vision-first desktop and mobile automation with a ComputerAgent

The fragility problem AskUI Python SDK targets

Selector-based automation assumes the application exposes a stable tree of identifiers. Desktop software often does not. A Qt dialog, a Java Swing tool, a Citrix session or a physical HMI panel may expose nothing an XPath or CSS query can reach, and even when it does, a label rename or a layout shift breaks the script. The README states the project's premise directly: traditional UI automation is fragile because every button move, label change or layout shift breaks scripts, and AskUI Agents address this by combining vision-based automation with AI-powered agents.

The intended audience follows from that. The repository topics list agents, computer-vision, cua, llms, rpa and test-automation, and the README frames the use cases as automating desktop apps, testing mobile applications and building RPA workflows. This is a tool for people who already accept that an element must be found on screen, either by what it looks like or by a natural language description, rather than by an attribute in a document object model. If your target is a web page rendered in a browser you control, the vision layer is overhead you do not need.

Two modes: single-step commands and agentic instructions

The SDK exposes one object, ComputerAgent, used as a context manager. Inside the block you have two styles. The agentic style hands a multi-step goal to the model and lets it plan: the README's example opens a browser, navigates to GitHub, searches and stars a repository in one act call. The single-step style is closer to a classic automation API: act to locate or perform an action, type to enter text, click to press a named element.

The same object also reads the screen. get takes a question in natural language and returns an answer extracted from what is displayed, which the README demonstrates by asking for a current account balance and printing the result. That mixing of action and extraction in one loop is the part worth noticing: you can locate a login form with act, fill it with type, advance with click, and then ask get for a value on the resulting page without leaving the agent context.

Underneath, the package depends on askui-agent-os, grpcio and protobuf, which suggests a client process talking to a separate runtime over gRPC rather than a pure in-process library. The pyproject.toml also lists fastapi, fastmcp and gradio-client, and the README documents extending agents with custom tools through the Model Context Protocol. The practical consequence is that a working install involves more moving parts than a single Python package, and the extras table exists precisely because the default install is kept small.

Installing askui and running a first agent

The README gives a single install command and a Python floor. The package name is askui, and the supported range is Python 3.10 up to but not including 3.14, because imagehash depends on scipy and the pyproject.toml comment notes that constraint.

pip install askui

The default install carries the core dependencies. Extras are added only when needed: office-document for reading or converting Excel and Word files through MarkItDown, bedrock for running Anthropic Claude through AWS Bedrock, vertex for Claude on Google Vertex AI, and otel for OpenTelemetry export. The README warns that askui[all] pulls in every optional extra in one command and produces a larger install, and suggests picking individual extras when you know what you need.

pip install askui[all]

Before the first run you need a model provider. The README's quickest path is AskUI's own hosted models: sign up at hub.askui.com to activate a free trial without a credit card and to obtain a workspace ID and access token. Those go into two environment variables. On Linux and macOS the README shows export statements; on Windows PowerShell it shows the $env: form.

export ASKUI_WORKSPACE_ID=<your-workspace-id-here>
export ASKUI_TOKEN=<your-token-here>

With credentials in place, the smallest working script constructs a ComputerAgent and issues an instruction. The README says to run it with python <file path>, for example python test.py.

from askui import ComputerAgent

with ComputerAgent() as agent:
    agent.act(
        "Open a browser, navigate to GitHub, search for 'askui vision-agent', "
        "and star the repository"
    )
    balance = agent.get("What is my current account balance?")
    print(f"Balance: {balance}")

What you should see is the agent taking control of the desktop and performing the steps, with the extracted value printed at the end. If nothing happens, the two environment variables are the first thing to check, since the README ties the hosted provider path to them.

Extending the agent with custom tools over MCP

Tools are passed per instruction rather than registered globally. The README's example imports ComputerSaveScreenshotTool with a base_dir and PrintToConsoleTool, then supplies them in the tools list of a single act call. The agent can then use those capabilities while executing that instruction.

from askui import ComputerAgent
from askui.tools.store.computer import ComputerSaveScreenshotTool
from askui.tools.store.universal import PrintToConsoleTool

with ComputerAgent() as agent:
    agent.act(
        "Take a screenshot of the current screen and save it, then confirm",
        tools=[
            ComputerSaveScreenshotTool(base_dir="./screenshots"),
            PrintToConsoleTool()
        ]
    )

Passing tools per call is a design choice with a cost. It keeps the capability surface of any single instruction explicit, which is good for auditing what an agent was allowed to do, but it also means a long workflow repeats the same tool list at every step unless you wrap the calls yourself. The pyproject.toml pins anyio to 4.10.0 with a comment explaining that listing MCP tools through fastmcp inside the runner fails otherwise, so this integration is sensitive to dependency resolution.

Where the model dependency becomes the limitation

Every act and get call that cannot be answered from cache is a model call. The README lists caching as a feature meant to save expensive calls to AI APIs by using them only when really necessary, which is an admission that the uncached path costs money and time. A deterministic selector-based test that runs in milliseconds has no equivalent here: a vision step has to capture the screen, send it to a model, and wait for a decision.

That makes the SDK a poor fit for high-frequency regression suites on web applications, for headless CI where no display exists, and for any workflow where a wrong click has irreversible consequences and you need a guarantee rather than a probability. The model choice also matters operationally. The pyproject.toml constrains anthropic to >=0.86.0,<1 with a comment that version 1.x dropped a temperature keyword the SDK still sends and would crash every model call. Anyone who force-upgrades that dependency outside the declared range is on their own.

Platform support is stated as Windows, Linux, macOS and Android. The README's headline also mentions iOS and HMI devices, but the extras and dependency list show Android tooling through pure-python-adb; the README does not document an equivalent iOS driver setup, so treat iOS as unverified from the repository alone.

AskUI Python SDK compared with Playwright and Appium

The closest alternatives split along the same axis. Playwright drives browsers through the browser's own automation protocol and locates elements by role, text or CSS, which is fast, free per step and reproducible, but it only reaches web content and it depends on the page exposing an accessible structure. Appium targets mobile apps through platform drivers and element identifiers, which is the standard route for Android and iOS, but it inherits the same selector fragility when an app renders custom widgets or a canvas.

AskUI's difference is that it does not require an identifier at all. It finds elements by text, images or a natural language description, and it can be pointed at a screen that no other driver can introspect, including a remote desktop session or an HMI panel. The trade is determinism and cost: you exchange a selector that either matches or does not for a model decision that usually matches. For a smoke test on a desktop application with no automation hooks, that exchange is worth making. For a thousand-case web regression suite, it is not.

Maintenance, licence and upgrade cost

The repository is not archived and the last push was on 2026-09-09, four days before this writing. Releases are frequent: v0.44.0 on 2026-08-13, v0.45.0 on 2026-08-27 and v0.46.0 on 2026-08-31. The version is still 0.x, which in practice means the public API can change between minor releases, and the pyproject.toml shows the project pins dependencies tightly when a break is known, as it does for anthropic and anyio.

Licensing is MIT, declared both in the repository metadata and in the pyproject.toml as license = {text = "MIT"}. MIT permits commercial use and modification with attribution and without warranty. That covers the SDK code. It does not cover the model you connect to: the AskUI hosted provider, Anthropic, Google, AWS Bedrock and Vertex each carry their own terms and their own per-call pricing, and the README's free trial is a trial rather than a licence. Confirm separately what your chosen provider allows for the data your screens contain, since screenshots of a production system may include personal or regulated information. This is a description of the licence text, not legal advice.

The upgrade cost is mostly dependency management. With releases landing every two to four weeks and a 0.x version number, expect to re-read the release notes before bumping, and expect the Python ceiling of 3.13 to block you if your environment has already moved to 3.14.

Editorial conclusion

Adopt the AskUI Python SDK when your target is a desktop application, an Android device or an HMI panel where no stable accessibility tree exists, and when you can accept a model call per step. Do not adopt it for headless web testing, where Playwright or Selenium gives deterministic selectors at no inference cost, and do not adopt it if your environment is pinned to Python 3.14. Before writing production scripts, verify that your chosen model provider is reachable from the machine under test, confirm which optional extras you actually need, and check that the anthropic pin below 1.x still matches the version your package manager resolves.

Frequently asked questions

Official sources

1. askui/python-sdk on GitHub
2. License: MIT
3. Project website
4. README
5. Releases