agent-desktop: driving macOS apps through the accessibility tree, not pixels

The problem is coordinate drift, not vision quality

Screenshot-driven computer use has a specific failure mode. The model looks at a frame, decides a button is at some pixel offset, and issues a click there. If a dialog appears, a window moves, or a list re-renders between the screenshot and the click, the action lands somewhere else. Retrying is not obviously safe, because a retry after a partial success can double-submit. The README frames the alternative directly: agent-desktop sees an app's real UI structure through OS accessibility trees, and the claim it makes is that refs stay stable and actions stay safe to retry. That is the whole pitch. The target reader is someone building an agent that has to operate a real desktop app, not a browser page, and who has concluded that pixel guessing is the part that keeps breaking. The project is written in Rust, ships as a single binary, and the README lists Finder, Safari, System Settings, Xcode and Slack as examples of apps it works with, which is a way of saying: anything that publishes an accessibility tree.

Snapshots, snapshot IDs and qualified refs

The core object is the snapshot. Running agent-desktop snapshot --app Finder -i returns interactive elements with refs and a snapshot_id. Refs are qualified, in the form @s8f3k2p9:e1 and @s8f3k2p9:e2, meaning the element identifier is scoped to the snapshot it came from. That scoping is what makes retries tractable: an action is expressed as click @e3 --snapshot s8f3k2p9, so the tool knows which observation the ref belongs to rather than trusting a bare index. The workflow the README gives is observe, act, re-observe. After any UI change you run snapshot -i again, and the documentation's own example for verifying a state change is to re-drill the same region. Type actions are equally explicit: type @e5 --snapshot s8f3k2p9 "quarterly report" inserts text into a field by ref, and press cmd+s handles keyboard shortcuts separately. Output is structured JSON with error codes and recovery hints, which matters more than it sounds: an agent that gets a machine-readable failure with a hint can branch, whereas an agent parsing prose has to guess.

Progressive skeleton traversal is the token argument

A full accessibility snapshot of a dense app is enormous. The README's own Slack comparison puts a regular snapshot at 30,743 tokens against 383 for a skeleton overview, and claims 78 to 96 percent token reduction on dense apps. Treat those numbers as the project's illustration of its own approach rather than an independent measurement. The mechanism is more useful than the figure. snapshot --skeleton --app Slack -i --compact produces a depth-3 map in which truncated containers report a children_count, and each truncated branch exposes a safe drill ref. You keep the snapshot_id, then call snapshot --root @e3 --snapshot s8f3k2p9 -i --compact to expand just that region, act on an element found inside it, and re-drill the same root to confirm the change. For simple apps the README says a full snapshot is fine, so the two modes are a deliberate choice rather than a default. The trade-off is real: skeleton traversal costs extra round trips, and an agent that drills into the wrong branch pays for the mistake in latency.

Getting it running, and the permissions wall

The recommended install is npm install -g agent-desktop, which downloads a prebuilt binary. Without installing, npx agent-desktop snapshot --app Finder -i works. From source it is git clone, cargo build --release, then copying target/release/agent-desktop into /usr/local/bin. The stated requirements are Rust 1.89 or later and macOS 13.0 or later. Permissions are where most first attempts will stall. Accessibility permission is required for the core path; screenshots additionally need Screen Recording; the Notification Center opener needs Automation permission for System Events. Plain permission checks never prompt, which is deliberate, so a check that returns denied will not fix itself. To request what is missing you run agent-desktop permissions --request, which the README describes as requesting permissions in a bounded isolated helper. The response is a structured object per permission, with a state field and, when denied, a suggestion string. Automation can report granted, denied, or unknown, and unknown specifically means macOS would need to prompt or System Events could not be probed without prompting. That third state is the honest part of the design and also the annoying one, because unknown is not actionable without an interactive prompt.

The stateless CLI, the FFI library, and what fails closed

The README counts 58 command names and 54 operational commands covering observation, interaction, keyboard, mouse, notifications, clipboard, window management, session lifecycle and trace read and export. The four remaining names are held-input commands reserved for a stateful daemon, and in the stateless CLI they fail closed. That is a design decision worth reading carefully: anything requiring held state, such as a mouse button held down across calls, is simply not available through the CLI. For agents that only click, type and press shortcuts, this costs nothing. For agents that need drag operations or sustained modifier state, it is a wall. The escape hatch is the C-ABI cdylib, libagent_desktop_ffi, shipped per release for macOS, Linux and Windows alongside the CLI tarballs. You dlopen it and call the functions declared in agent_desktop.h, which avoids fork-exec per command. The README's Python example calls lib.ad_init(4) to verify the ABI major version before any other call, then ad_adapter_create, then the observe-to-act sequence of ad_snapshot and ad_execute_by_ref, and finally ad_adapter_destroy. The consumer guide under skills/agent-desktop-ffi is where entrypoints, ownership, threading and error handling are documented. Note the ABI check: the example passes 4, so a mismatch against AD_ABI_VERSION_MAJOR is something your binding is expected to catch rather than ignore.

Chromium interop via CDP, and where the accessibility path stops

For Chromium-based apps the project offers a second path. launch --cdp opens a verified DevTools port so any framework that speaks CDP, including Playwright, Puppeteer, chrome-remote-interface and agent-browser, can drive the web contents, while native menus, dialogs and windows stay on the accessibility path. This is a sensible split, because web content inside Electron apps is exactly where accessibility trees get noisy and where the web tooling ecosystem is already strong. It also implies a boundary: if your target is a Chromium app and you only need the web contents, you may not need agent-desktop at all. The accessibility route earns its place when the interaction crosses the boundary, for example clicking a native menu item that then changes what the web view shows. The README does not describe how the two paths share state, so if you need a single coherent observation spanning native chrome and web content, verify that yourself before committing.

macOS-first, and the alternative you already have

The most concrete limitation is platform coverage. Requirements state macOS 13.0 or later, and the feature list names macOS in the topics while the examples are Finder, Safari, System Settings, Xcode and Slack. The FFI library ships for macOS, Linux and Windows, but the README does not claim the accessibility observation path works on all three, and the permissions model it describes is macOS-specific. If your fleet is Windows or Linux, this is not the tool yet. The obvious alternative for anyone already in the browser automation world is Playwright or Puppeteer driving a browser directly. The difference in approach is fundamental: those tools address a DOM through selectors and a protocol, which is precise and stable inside a page, and they have no view of native windows, menus or dialogs at all. agent-desktop addresses the OS accessibility tree, which covers any app that publishes one, at the cost of macOS permissions and a tree whose shape varies per app. A second alternative is writing raw macOS accessibility API calls yourself; agent-desktop's contribution there is the ref model, the snapshot scoping, the JSON error contract and the CLI surface, not the underlying API access.

Maintenance cost and the Apache-2.0 terms

The release cadence visible in the supplied material is roughly weekly through August and early September 2026, with v0.8.3, v0.8.4 and v0.8.5 landing about a week apart. A pre-1.0 project moving that fast means the CLI surface and the FFI ABI can shift, which is precisely why the FFI example checks AD_ABI_VERSION_MAJOR before calling anything. If you build a binding, pin a release and re-verify the ABI on upgrade rather than tracking main. The npm package is the low-friction path because it downloads the prebuilt binary; building from source ties you to a Rust 1.89+ toolchain. On licensing, the project is Apache-2.0, which permits commercial and closed-source use and includes a patent grant. Apache-2.0 also carries notice and attribution obligations, and the file-level details are in the LICENSE file rather than in the README. This is not legal advice; if you are redistributing the binary or the FFI library inside a product, read the licence text and your own counsel's reading of it.

Editorial conclusion

Adopt agent-desktop if your agent already runs on macOS 13 or later, you can grant Accessibility permission, and you want element refs with a re-observe loop instead of pixel coordinates. Do not adopt it if you need a cross-platform desktop driver today, or if you cannot tolerate a tool whose CLI is stateless and whose held-input commands fail closed by design. Before wiring it into anything, run agent-desktop permissions --request and confirm the JSON reports accessibility as granted, then run a skeleton snapshot on your densest target app and check whether the drill-down refs actually reach the controls you care about.

Official sources

1. Issues
2. lahfir/agent-desktop on GitHub
3. License: Apache-2.0
4. README
5. Releases