pi-computer-use: giving AI agents access to desktop apps without an API

The gap that makes computer use the right tool

pi-computer-use is a Pi extension that gives an AI agent a way to work with desktop applications that have no API, no MCP server, and no terminal interface. This positioning is explicit in the README: an API or direct integration, when one exists, should be used first. Computer use is for when the only available interface is the app on screen.

On the agent side, this amounts to one install command and a set of eight tools. An active Pi session and, on some platforms, one or two permission grants are required before the tools become usable. Written for the Pi runtime rather than as a standalone library, it is not a candidate for projects that do not already use Pi.

GitHub's project description reads "Let Pi control your apps on MacOS & Windows." That description is narrower than what the README covers: the install section documents Linux support, and a dedicated file, docs/linux.md, exists for the full Linux capability matrix. Anyone reading only the repository description would miss the Linux path entirely.

Installing the package and granting the right permissions on each platform

The install command is one line:

pi install npm:@injaneity/pi-computer-use

After that, start Pi and complete the platform setup flow. Setup registers the helper and verifies permissions before the tools are available.

On macOS, the helper installs at ~/Applications/pi-computer-use.app by default. Existing system-wide installs remain at /Applications/pi-computer-use.app. Two permissions are required in System Settings: Accessibility and Screen Recording. On newer macOS versions, Screen Recording appears as "Screen and System Audio Recording." The setup flow registers the helper first, so both entries should already be listed; enable the toggles and choose Recheck. The macOS helper requires macOS 14 or newer. Earlier versions are not supported.

On Windows, no helper app or TCC permission flow is involved. The extension uses the platform accessibility APIs, and the README specifies an interactive desktop session. Headless Windows environments are out of scope.

On Linux, the extension must run inside the target user's graphical session with a working AT-SPI2 accessibility bus. The capability model on Linux differs by display server, and the README points to docs/linux.md for the full matrix.

Node.js 20.6.0 or newer is required, per the package.json engines field. Version 0.5.1 is published on npm under the name @injaneity/pi-computer-use.

Eight tools run from discovery through observation to action

The eight tools form a deliberate sequence. An agent starts by finding what is on screen, then observes and narrows the picture, and finally acts on a specific control.

find_roots enumerates open applications and their windows. observe_ui reads the structure of a window, returning a description of what is visible. search_ui narrows that description by looking for specific text, buttons, or controls. Rather than returning the full tree, it focuses the agent's view on what it is already looking for. expand_ui and inspect_ui then go deeper into the parts the top-level observation summarizes.

read_text pulls the textual content from a region of the interface without treating it as a control. act_ui performs actions: clicking, typing, scrolling, pressing controls. wait_for halts the agent until a specified change appears in the interface, preventing subsequent steps from running before the UI has caught up.

Tool names and one-line descriptions appear in the README, but the full reference is in docs/usage.md. Using the extension in production requires reading that file, because the README summary does not cover parameter shapes or error conditions.

A /computer-use command inside Pi shows the active configuration and where each setting came from, which is useful for debugging whether the permissions and helper are set up correctly before the first real agent task.

act_ui replaced screenshot, click and set_text; older code will break

Four tool names are listed in the README's development status section as no longer part of the public extension surface: screenshot, click, set_text, and computer_actions. These were direct tools in an earlier version of the API. Any code, tutorial, or automation script that calls them will fail against the current version.

act_ui is the replacement for all four. It accepts one or more intent steps in a single call, preserves focus across dependent input steps, verifies that each step was delivered, and recovers safely when something goes wrong. It stores one complete successor state, and it returns a compact diff when identity confidence allows. Rather than issuing individual clicks or keystrokes one by one, the agent passes a sequence of intent steps and act_ui coordinates the rest, including focus management and recovery.

No migration guide ships with the package to help with this transition. Anyone upgrading from an earlier version needs to check docs/usage.md to understand what act_ui expects in place of the old calls. Release records show v0.5.0 published on 2026-07-26, v0.5.1 on 2026-08-31, with the last push on 2026-10-04.

Immutable observations and a multi-root forest across desktop and CDP

Architecture in the README centers on immutable, state-scoped observations. Desktop surfaces and Chrome DevTools Protocol pages share one multi-root forest, which lets the agent treat a native window and a browser page through the same structural model.

Progressive outline queries are cached, so the agent does not re-read the full UI tree on every call. Live work is ordered per physical resource, which allows independent roots to run in parallel. This matters when the agent is watching multiple windows: operations on separate windows do not wait for each other.

act_ui's delivery verification and successor-state storage fit into this model. Nothing in the flow requires the agent to assume an action succeeded; the compact diff that act_ui returns lets it check whether the interface changed as expected before the next step runs. Confirmation is built into the call rather than being something the agent has to implement separately.

Linux has three capability tiers, and Wayland stays semantic-only

Linux capability splits into three layers. AT-SPI2 semantic operations are the base, running background-first on any session type regardless of display server. Observation and accessible control interaction work here without requiring focus or physical input, which makes AT-SPI2 the layer that runs everywhere.

X11 adds window metadata and focus information via EWMH, window capture, and policy-gated XTEST physical input. Policy-gated is the operative word: strict headless or background policies on X11 disable XTEST and focus entirely, which removes physical input and leaves only the AT-SPI2 semantic layer. An application that responds only to physical input and has no accessible UI tree is unreachable under those policies.

Wayland is the most limited tier, described in the README as semantic-only. Diagnostics reads portal capability properties without opening a portal session, and interactive portal use is disabled. An agent running on a Wayland session gets the AT-SPI2 base layer and nothing above it. Exact capability matrix and portal status for each configuration are documented in docs/linux.md, which the README treats as the authoritative source on Linux.

Where the extension cannot help: headless sessions and inaccessible apps

Three situations put pi-computer-use out of reach. First is a headless desktop session on Windows. An interactive desktop session is specified as a requirement in the README, and no documented fallback exists for server or container environments on that platform.

Second is Linux under strict background policies, where XTEST and focus control are disabled. If an application responds only to physical input and has no accessible UI tree, the semantic layer has nothing to drive.

Third is any application that does not implement accessibility APIs at all. observe_ui, search_ui, and act_ui depend on what the platform accessibility layer exposes. An app with no accessible UI tree gives those tools nothing to inspect or act on. Specific applications known to work or fail are not listed in the README, so testing a target application before committing to this approach is a required step.

Computer use is also not a match when a direct interface is available. If a reliable API, MCP server, or command-line path exists, that is the right choice according to the README. pi-computer-use covers the remaining cases where none of those options exist.

Editorial conclusion

pi-computer-use is worth considering when an AI agent needs to reach a desktop application that exposes no API, no MCP server, and no file-based interface. Before adopting it, verify that the target application has an accessible UI tree, because the tools depend entirely on what the platform accessibility layer exposes. On macOS, test on macOS 14 or later and confirm both Accessibility and Screen Recording toggles are on in System Settings. On Linux, check docs/linux.md against the specific display server in use, because Wayland gives substantially less than X11.

Frequently asked questions

Official sources

1. injaneity/pi-computer-use on GitHub
2. Issues
3. License: MIT
4. README
5. Releases