CLI tool
lahfir/agent-desktop avatar
lahfir/agent-desktop

agent-desktop drives desktop apps through accessibility trees, with refs that expire on purpose

Agent Desktop gives any agent reliable computer use on the desktop. Built with Rust, it sees any app's real UI structure through OS accessibility trees and operates it — refs stay stable and actions stay safe to retry, instead of guessing from pixels.

26,129 stars1,958 forksRustApache-2.0

At a glance

What is it?
lahfir/agent-desktop is a Rust CLI and C library that turns macOS, Windows and Linux UI trees into JSON refs an AI agent can act on. Its ref model makes actions safe to retry, but it pushes snapshot churn, permission setup and privacy handling back onto the caller.
Who is it for?
Use agent-desktop if you already run an agent loop and want it to act on native desktop apps through structured refs instead of pixels, on macOS 13+, Windows 10 1809+ or Ubuntu 24.04+. Skip it if you want a packaged assistant with a model inside, or if your task needs a key or mouse button held across steps.
Can I use it commercially?
Yes. Apache-2.0 is a permissive licence: you can use, modify and sell software built on it, as long as you keep its copyright and licence notices.
Is it still maintained?
Yes. The repository last received commits 1 day ago.
What is it written in?
Mainly Rust, according to GitHub's language statistics.

Answers come from the project's GitHub data, last synced on November 22, 2026, and from our analysis. They are not legal advice.

From daily snapshots

Star history

  • lahfir/agent-desktop26,129 ★ · +25,037
Star history
Repository2026-09-112026-11-22
lahfir/agent-desktop1.1k26.1k

25,037 Stars in the last 72 days, measured from the daily GitHub snapshot. Compare with other projects

Editorial analysis

Every ref is bound to the snapshot that produced it

agent-desktop does not look at pixels. It reads the accessibility tree of a running app and prints it as JSON, attaching a ref to every element an agent can act on. The basic loop is three commands: observe, act, observe again.

bash
agent-desktop snapshot --app Finder -i      # interactive elements, each with a ref
agent-desktop click @s8f3k2p9:e8            # act on a ref from that snapshot
agent-desktop snapshot --app Finder -i      # observe again to verify

The ref `@s8f3k2p9:e8` has two halves. `s8f3k2p9` is the `snapshot_id` from the envelope, and `e8` is the element inside that snapshot. A trimmed Finder snapshot reports `ref_count` 59, `complete` true, and a tree whose nodes carry `role`, `name`, `ref_id` and `available_actions` such as `Click`. A bare `@e8` without the snapshot prefix fails with `INVALID_ARGS`.

The point of binding refs to a snapshot is that a ref cannot quietly land on a different element after the window changes. If the UI moved on, the action returns `STALE_REF` or `AMBIGUOUS_TARGET`, and the fix is always the same: take a new snapshot and use its refs. The cost is round trips. An agent clicking through a window that re-renders after each click pays for a fresh snapshot every time, and its code has to treat those two error codes as an ordinary part of the loop rather than as a crash.

There is no model inside, so the retry policy is yours to enforce

agent-desktop ships as a CLI and a C library and nothing more. It does not plan, decide or recover; the observe, act, verify loop runs in whatever agent calls it. What the binary does carry is guidance that matches its own version:

bash
agent-desktop skills get desktop      # the loop, refs, verification and retry rules
agent-desktop skills get platform     # what differs on the OS you run on
agent-desktop <command> --help        # flags for one command

For harnesses that load skills, the install line points at skills.sh:

bash
npx skills add https://github.com/vercel-labs/agent-desktop --skill agent-desktop

That URL, the npm badges and the license link all name vercel-labs/agent-desktop rather than lahfir/agent-desktop, so confirm which repository you are tracking before pinning versions or filing issues. The lighter alternative is two lines in `AGENTS.md` telling the agent to run both `skills get` commands before first use.

Every command prints one JSON envelope, and failures carry `error.disposition.retry`. The rule is narrow: retry a failed command only when that field is `safe`. A tool loop that retries every failure by reflex breaks that contract. The envelope schema, error codes and exit codes live on the JSON output page at agent-browser.dev, so keep that page open while writing the error handler.

Headless input is the default, and not every action accepts it

Actions run headless unless told otherwise: no pointer movement and no focus change. That is what lets an agent work inside one app while a person keeps using another. Passing `--headed` switches to physical input, which moves the real pointer and takes focus.

Headless is not universal. Each platform page on agent-browser.dev (macOS, Windows, Linux) lists what works headless, what needs `--headed`, and what refuses outright. The consequence is that a workflow built headless on one OS can need `--headed` on another, and at that point the agent and the human sitting at the machine are competing for the same mouse and keyboard focus. If you plan to run agent-desktop on a machine someone is actively using, read the platform page for every action your workflow depends on before you commit to it.

Windows adds a quoting trap. In PowerShell a bare `@token` is the splatting operator, so every ref must be quoted, as in `'@s8f3k2p9:e8'`. An agent that assembles shell commands by string concatenation and runs them through PowerShell will pass the wrong argument unless it quotes refs on that platform. This is easy to miss when the same agent code was first written against a macOS or Linux shell where the unquoted form works.

Four of the 60 commands exist but refuse to run

The command surface is wide. agent-desktop has 60 command names, of which 56 are operational. They fall into groups: observe (`snapshot`, `find`, `get`, `is`, `screenshot`, `list-surfaces`), act on a ref (`click`, `type`, `set-value`, `clear`, `toggle`, `expand`, `scroll-to` and more), keyboard and mouse (`press`, `hover`, `drag`, `mouse-move`, `mouse-click`, `mouse-wheel`), apps and windows (`launch`, `close-app`, `list-windows`, `focus-window`, `resize-window`, `open-system-surface`), notifications (`list-notifications`, `dismiss-notification`, `notification-action`), clipboard (`clipboard-get`, `clipboard-set`, `clipboard-clear`), `wait`, sessions (`session`, `trace`, `cursor-overlay`) and system (`status`, `permissions`, `version`, `batch`, `skills`).

The four that do not run are `key-down`, `key-up`, `mouse-down` and `mouse-up`. They refuse until a daemon exists that can release a held key or button. A held key needs something that outlives a single CLI invocation to let go of it, and that piece is not there yet.

For you, that means any interaction which depends on holding a modifier or a button across separate commands is unavailable. You are limited to single-shot commands such as `press`, `drag` and `mouse-click`. If the target app needs a gesture those cannot express, agent-desktop cannot perform it in its current form, and your agent should be told so rather than left to call a command that will refuse.

Permissions and OS floors decide whether it runs at all

Installation is a global npm package followed by two checks:

bash
npm install -g agent-desktop
agent-desktop status                # platform, permissions, supported surfaces
agent-desktop permissions --guide   # macOS: walks you through the Accessibility grant

On macOS the Accessibility grant is required, and `permissions --guide` walks through it. On Windows no grant is needed for apps at your terminal's integrity level; the install note is scoped to that level, so check apps running at a different integrity level with `status` before depending on them. On Linux the tool has to run inside the user's graphical session with `at-spi2-core`, so a bare remote shell without that session is not the environment it is built for.

The version floors are fixed: macOS 13 or later on Intel or Apple Silicon, through the AX API; Windows 10 1809 or later on x64 or ARM64, through UI Automation; Ubuntu 24.04 or later on x86_64 or aarch64, X11 or Wayland, through AT-SPI. Older macOS releases and older Ubuntu LTS versions fall outside that table. Each binary is built for one OS and reaches the system only through a `PlatformAdapter`, so there is no single build to copy between machines. Release archives and building from source are covered on the Installation page at agent-browser.dev.

Snapshots carry whatever is on the screen

An accessibility tree is a text rendering of the UI. Even the trimmed Finder example exposes a window named Downloads and an item named Documents; in a mail client or a chat app, the same `name` fields carry subjects, message previews and contact names. The project states plainly that snapshots, clipboard reads, screenshots, command output and debug artifacts can contain private desktop content, and that they belong out of public logs and uploads.

This warning matters more for an agent tool than for a human one. If your agent harness records tool calls and their output, every `snapshot`, `get` or `clipboard-get` result lands in that record. If the harness sends transcripts to a hosted model or ships logs to an external service, the desktop content goes with them. If you paste a failing run into a public bug report, you publish whatever the app was showing at the time.

The practical consequence is that redaction and log retention are your responsibility, not agent-desktop's. Decide where command output is stored before pointing the tool at an app that holds anything sensitive, and keep debug artifacts on the machine that produced them.

The C library skips the CLI but pins you to an ABI version

Each release ships `libagent_desktop_ffi`, so a program can call the same core in process instead of spawning the CLI for every action. The Python example loads it through ctypes:

python
import ctypes
lib = ctypes.CDLL("./lib/libagent_desktop_ffi.dylib")
lib.ad_adapter_create.restype = ctypes.c_void_p
lib.ad_adapter_destroy.argtypes = [ctypes.c_void_p]
assert lib.ad_init(7) == 0  # AD_RESULT_OK when the header's AD_ABI_VERSION_MAJOR matches the library
adapter = lib.ad_adapter_create()
lib.ad_adapter_destroy(adapter)

`ad_init(7)` returns `AD_RESULT_OK`, which is zero, only when the header's `AD_ABI_VERSION_MAJOR` matches the library. A binding that hardcodes the major version has to move in step with the library, or initialization fails. The `.dylib` in the example is the macOS form, and the one binary per OS rule applies to the library too.

Building from source follows the same split. The Cargo workspace lists `crates/core`, `crates/macos`, `crates/windows`, `crates/linux`, `crates/ffi` and `src`, uses edition 2024 and requires Rust 1.89. `crates/ffi` is a member but not a default member. Platform adapters build only on their own OS, so cargo must be scoped to the host's packages:

bash
HOST_PKGS=(-p agent-desktop-core -p agent-desktop-macos -p agent-desktop -p agent-desktop-ffi)
cargo test "${HOST_PKGS[@]}"
cargo clippy "${HOST_PKGS[@]}" --all-targets -- -D warnings

On Windows swap in `agent-desktop-windows`, on Linux `agent-desktop-linux`. Releases are arriving quickly: v1.0.1 on 2026-10-28, v1.0.2 on 2026-10-30 and v1.1.0 on 2026-11-03, with the last push on 2026-11-03 as well.

Editorial conclusion

Use agent-desktop if you already run an agent loop and want it to act on native desktop apps through structured refs instead of pixels, on macOS 13+, Windows 10 1809+ or Ubuntu 24.04+. Skip it if you want a packaged assistant with a model inside, or if your task needs a key or mouse button held across steps. Before relying on it, run agent-desktop status on the target machine and check the platform page for which actions work headless there.

Frequently asked questions

What is agent-desktop?

agent-desktop is a Rust CLI and C library that reads a desktop app's accessibility tree, returns it as JSON with a ref for every actionable element, and lets an AI agent click or type on those refs on macOS, Windows and Linux.

Does agent-desktop include its own AI model?

No. agent-desktop has no model inside; the observe, act, verify loop runs in your agent, which calls the CLI or the C library.

Which operating systems does agent-desktop support?

macOS 13 or later (AX API), Windows 10 1809 or later on x64 or ARM64 (UI Automation), and Ubuntu 24.04 or later on x86_64 or aarch64 with X11 or Wayland (AT-SPI).

Why does agent-desktop return STALE_REF?

Refs belong to the snapshot that produced them. If the UI changed since that snapshot, an action returns STALE_REF or AMBIGUOUS_TARGET, and you take a new snapshot and use its refs.

Official sources

  1. Issues
  2. lahfir/agent-desktop on GitHub
  3. License: Apache-2.0
  4. README
  5. Releases
Add this badge to your README

If you maintain this project, the badge below links readers to this analysis and shows its maintenance status from the daily GitHub snapshot. Paste the markdown into your README; add ?metric=license or ?metric=stars to the image URL for a different field.

Add this badge to your README

markdown
[![Hysen Labs](https://hysenlabs.com/badge/lahfir-agent-desktop.svg)](https://hysenlabs.com/projects/lahfir-agent-desktop)

Weekly Top 10 badge

markdown
[![Hysen Labs](https://hysenlabs.com/badge/lahfir-agent-desktop.svg?metric=rank)](https://hysenlabs.com/projects/lahfir-agent-desktop)

Star history chart for your README

markdown
[![Star history](https://hysenlabs.com/chart/lahfir-agent-desktop.svg)](https://hysenlabs.com/projects/lahfir-agent-desktop)

Was this useful?