Ghost OS: Accessibility-Tree Computer Use for macOS Agents
Full computer-use for AI agents. Self-learning workflows. Native macOS. No screenshots required.
At a glance
- What is it?
- Ghost OS is a Swift-based MCP server that lets AI agents drive native macOS apps by reading the accessibility tree instead of screenshots, and by recording your clicks into replayable JSON recipes. Here is how it works, what it costs, and where it breaks.
- Who is it for?
- Adopt Ghost OS if your agents need to operate native macOS applications on a machine you control, and if you want each workflow recorded once and replayed as a readable JSON recipe rather than re-derived from pixels on every run. Do not adopt it if you need Windows or Linux coverage, if you are running macOS 13 or earlier, or if your target workflows live entirely inside a browser, where a DOM-based tool is a shorter path.
- Can I use it commercially?
- Yes. MIT is a permissive licence: you can use, modify and sell software built on it, as long as you keep its copyright and licence notices.
- Is it still maintained?
- Yes. The repository last received commits 176 days ago.
- What is it written in?
- Mainly Swift, according to GitHub's language statistics.
Answers come from the project's GitHub data, last synced on September 15, 2026, and from our analysis. They are not legal advice.
DEEP OPEN-SOURCE ANALYSIS
The gap Ghost OS is aimed at
An agent with shell access can write files, run tests, and grep a repository. It cannot click a button in a native app. The README frames the problem that way: the agent lives inside a chat box, and the apps on your Mac are outside it. Ghost OS is the bridge for that specific case. It is built for macOS 14 and later, written in Swift 6.2, and shipped as an MCP server so an MCP client such as Claude Code, Cursor, or VS Code can call its tools. The intended user is someone running an agent on their own Mac who needs it to reach Slack, Finder, Messages, or a desktop app that has no API. It is not a general automation framework and it is not a cross-platform tool. The platform badge says macOS 14+, and nothing in the material suggests anything else.
Accessibility tree first, vision model as fallback
The core mechanism is the macOS accessibility tree. Instead of capturing a screenshot and asking a model to guess what is on screen, Ghost OS reads structured, labeled data about the elements in each app. That gives an agent element roles, labels, and positions rather than pixels. When the AX tree is not enough, which the README attributes to web apps and dynamic content, the tool falls back to a local vision model, ShowUI-2B, for visual grounding. That two-tier design is the main architectural claim, and it is also the main thing to test. The AX tree is only as good as the app's accessibility implementation. Electron apps, canvas-rendered UIs, and custom-drawn controls are the usual places where the structured path thins out, and the README's own admission that web apps need the vision fallback is a signal about where the boundary sits. The fallback is a 2B model running locally, so expect it to handle grounding rather than open-ended visual reasoning.
How the self-learning loop actually records a task
The v2.2.1 feature is the part worth understanding in detail, because it is what separates Ghost OS from screenshot-driven computer-use tools. The flow has three tools. `ghost_learn_start` begins watching, taking a `task_description` such as `"send email in Gmail"`. You then perform the task by hand: clicking, typing, switching apps. Ghost OS observes through a CGEvent tap, and the README states that each observed event is enriched with accessibility tree context. `ghost_learn_stop` ends the recording and returns the enriched action sequence. `ghost_learn_status` reports progress during the capture. Claude then synthesizes that raw observation into a parameterized, replayable recipe, and `ghost_recipe_save` stores it. The README's worked example shows eight actions producing a recipe with three parameters: recipient, subject, and body. Later runs call `ghost_run` with the recipe name and a params object. The division of labor is the interesting part: a frontier model does the reasoning once, at record time, and a small model executes the stored recipe afterward. Recipes are JSON, which the README presents as a transparency property. You can read every step before running it.
Install, permissions, and the manual fallback
The documented install is two commands: `brew install ghostwright/ghost-os/ghost-os` followed by `ghost setup`. The README states that `ghost setup` handles permissions, MCP configuration, recipe installation, and vision model setup. The learning feature specifically requires Input Monitoring, granted under System Settings > Privacy & Security > Input Monitoring, which is consistent with a CGEvent tap reading your keystrokes and clicks. There is a documented failure mode at install time: Homebrew on macOS developer betas may demand an Xcode version that does not exist yet. The README gives a manual path for that case, downloading `ghost-os-2.2.1-macos-arm64.tar.gz` from the latest release, unpacking it, and copying `ghost` and `ghost-vision` into `/opt/homebrew/bin/`, plus `GHOST-MCP.md` and the `recipes` directory into `/opt/homebrew/share/ghost-os/`, and `vision-sidecar` into `/opt/homebrew/`. Note that this path is arm64 only as written. If you are on Intel, the README does not give you an equivalent snippet, and that is a real gap in the documentation rather than something you can infer your way around.
The recording model has a sharp edge
Recording by demonstration is convenient and it is also the weakest point. A recipe is a sequence of actions captured against one version of one app's UI. The README does not describe any selector-stability mechanism, any re-validation step, or any drift detection between record time and replay time. That means a UI change can turn a working recipe into a sequence of clicks on the wrong elements, and the failure may not be loud. The parameterization is done by a model synthesizing the raw observation, so a mis-inferred parameter boundary is possible too: the example recipe has recipient, subject, and body, and a recording where those three fields are not cleanly separable could produce a recipe that hardcodes something you wanted to vary. The Input Monitoring requirement is another boundary. This is a tool that watches everything you type and click while recording is active, which is a deliberate trade for the feature, not an incidental detail. Finally, the whole thing is macOS-only. If your agents run on Linux containers, Ghost OS is the wrong tool and no amount of configuration changes that.
Where it sits against a DOM-based browser tool
The README's own comparison table places Ghost OS against Anthropic Computer Use, OpenAI Operator, and OpenClaw. The sharpest contrast is with OpenClaw, which is also MIT-licensed and also keeps data local, but sees the browser DOM rather than the accessibility tree. That difference matters more than the feature list suggests. A DOM-based tool gets exact selectors and structured element identity inside a browser, which is a stronger foundation for web work than an AX tree plus a 2B vision fallback. Ghost OS trades that precision for reach: Slack, Finder, Messages, and any other native app are in scope, and no browser extension or driver is involved. If your workflows are entirely web, the DOM approach is the shorter path and you should take it. If they are not, the AX tree is the only structured interface macOS gives you, and Ghost OS is built around using it. The table also lists data locality, and the README claims your data never leaves the machine, which is consistent with a local vision model and no cloud calls, though the material does not document what the tool transmits, if anything.
Licence, maintenance, and upgrade cost
Ghost OS is MIT-licensed, which permits commercial use, modification, and redistribution provided the copyright notice and permission notice are preserved. That is the standard permissive arrangement, and it is worth noting that the sibling projects Shadow and Specter are separate repositories with their own terms; the MIT badge here covers Ghost OS. This is not legal advice, and if you are embedding it in a product you should read the LICENSE file yourself. On maintenance, the release cadence visible in the material is tight: v2.1.2 on March 11, v2.2.0 and v2.2.1 both on March 12, and the last push to the default branch on March 23, 2026. Frequent releases are good for fixes and bad for churn. The v2.1.2 notes mention pinned vision sidecar dependencies and a fix for the vision model download, which tells you the sidecar has been a moving part. The upgrade question you actually have to answer is whether saved recipes survive version bumps, and the material does not say. Treat your recipe directory as something to test after each upgrade rather than something to assume is stable. The manual install path also means you re-copy binaries and the `recipes` directory by hand on beta macOS, so budget for that if you are on a developer beta.
Editorial conclusion
Adopt Ghost OS if your agents need to operate native macOS applications on a machine you control, and if you want each workflow recorded once and replayed as a readable JSON recipe rather than re-derived from pixels on every run. Do not adopt it if you need Windows or Linux coverage, if you are running macOS 13 or earlier, or if your target workflows live entirely inside a browser, where a DOM-based tool is a shorter path. Before committing, verify three things: that `ghost setup` completes the Input Monitoring and Accessibility grants on your machine, that the recipes you care about survive a replay after the app updates its UI, and that the ShowUI-2B vision sidecar downloads and runs on your hardware, because that fallback is what covers web apps and dynamic content the AX tree does not describe.
Community notes