Clawd Cursor: a local MCP server that compiles the screen into a UI map

The gap Clawd Cursor fills: GUIs with no public surface

Most agent work has an escape hatch. A service exposes an API, a tool ships a CLI, a vendor publishes an integration. Clawd Cursor is aimed at the residue: the legacy desktop app, the internal admin panel, the installer, the canvas. The README states the positioning directly, calling it the fallback execution layer for when APIs, CLIs and direct integrations are not available, and adding the practical rule: if a native API exists, use it; if a CLI exists, use it; Clawd Cursor is for the click.

That framing matters because it sets the adoption bar. This is not a framework you build a product on. It is an MCP server you attach to an agent that already has tool calling, so the agent gains hands. The package description names the audience plainly: any tool-calling model, Claude, GPT, Gemini or Llama, on Windows, macOS or Linux. If your agent loop already speaks Model Context Protocol, the integration is a config entry rather than a library rewrite.

How the UI map is compiled: accessibility tree first, OCR second, pixels last

The engine is a tiered perception pipeline, and the ordering is the whole design. The accessibility tree is read first because it is free. OCR runs second because it is cheap. A screenshot is the last tier and the README is explicit that this is the only step that puts pixels into the model's context, noting that screenshot and vision are the same step. The agent climbs tiers only when the cheaper one cannot answer.

The output is not a list of coordinates. It is a fused, confidence-scored set of elements, each tagged with a stable el_NN id. Actions are addressed to those ids. Coordinates appear only in the last-resort tier, described as live pixels off the current frame, for canvas-only apps or tasks that genuinely need spatial reasoning. The practical consequence is that the same button keeps the same handle across observations, so a script can say click el_12 rather than click at (412, 688) and hope the window did not move.

Two claims in the README are worth separating from the marketing. The first, that token cost tracks task difficulty, follows from the tiering: simple forms resolve in the accessibility tree and never touch a vision model. The second, that vision-centric agents need a screenshot in the model for every observation, is a fair description of the OmniParser and UI-TARS approach, which the README classifies as parsing approaches you build an agent around rather than products you install.

Installing Clawd Cursor and running a first verified action

The package is published on npm as clawdcursor and the README badge states Node 20 or newer. The bin entry maps the clawdcursor command to dist/surface/cli.js, so the CLI is the entry point after install.

npm install -g clawdcursor
clawdcursor doctor

The second command is the one to run first. The .env.example file describes it as auto-detecting and configuring everything, and the environment file lists the variables it may write: AI_API_KEY, an optional AI_PROVIDER set to anthropic, openai, ollama or kimi, or provider-specific keys such as ANTHROPIC_API_KEY and OPENAI_API_KEY. If you prefer to configure by hand, copy the example file and fill it in.

cp .env.example .env

The install is not silent about its native dependencies. The postinstall script runs scripts/verify-install.js and then scripts/postinstall-native.js, each with a fallback that exits zero, and the published file list includes native/build.sh, native/Package.swift, native/Sources/ and native/entitlements.plist. That is a compiled helper, and on macOS the entitlements file is the part that determines whether the process is allowed to observe the screen at all.

For a first real use, the shape the README points at is a reusable script rather than a one-off click. The distinctive argument is expect: pass it on a consequential action and the engine re-checks the live screen after a short settle window for asynchronous UIs, then reports a DEVIATION instead of a success. A first script should therefore do exactly one thing and assert on it, so you learn how the engine reports a miss before you chain ten steps together.

The safety gate and what the verification actually catches

Every call routes through a single safety.evaluate() chokepoint that returns allow, confirm or block, and the README states that this holds whether the call arrives from an editor over stdio, an external agent over HTTP, or the built-in loop. The agent cannot bypass it. That is a stronger claim than most desktop automation makes, where the safety story is usually a prompt instruction the model is free to ignore.

The release history shows this area is still being worked. v1.5.9 is titled batch confirm-tier hardening (security), which tells you the confirm tier is where the interesting edge cases live: a batch of calls approved as a group is a different risk shape from one call approved alone. If you are evaluating the project for anything consequential, read the CHANGELOG entry for that release rather than the headline features.

Verification is the other half. The README's argument is precise: a completed task cannot be marked done on evidence that was already true before it acted. That is the failure mode of naive GUI agents, where the model clicks a button, sees a screen that looks plausible, and declares victory. Clawd Cursor's expect clause forces a fresh observation. What it does not do, according to the README, is roll back a failed action. A DEVIATION tells you the screen did not match; it does not undo the click. Recovery is the caller's problem, and that is a real design boundary.

Where Clawd Cursor is the wrong tool

The first wrong fit is the web. The README's own comparison table puts browser-use and Playwright MCP in the web-only column and Clawd Cursor in the any-desktop-app column, and that cuts both ways. If your target is a web application, a browser automation library gives you a DOM, deterministic selectors and a test runner. Clawd Cursor would be driving a browser through the same accessibility-and-OCR pipeline it uses for a native app, which is strictly more work for strictly less determinism.

The second wrong fit is a machine you do not control. The accessibility tree is the free tier, and on macOS and Windows reading it requires OS-level permissions. The repository ships native/entitlements.plist and a native build step, and the README does not document what happens when those permissions are denied. A headless CI runner or a locked-down corporate image is a poor host for this.

The third is any workflow where the GUI is incidental. If the task can be expressed as an API call, a CLI invocation or a file operation, Clawd Cursor adds a perception layer and a safety layer between you and something that never needed either. The project's own README makes this argument against itself, which is unusual and worth taking at face value.

Finally, the README does not document rollback, and it does not document a sandbox. There is no stated isolation boundary between the agent and your real desktop beyond the safety gate's allow, confirm and block decisions. Treat the host machine as the blast radius.

Alternatives, and the actual difference in approach

The README names its closest install-and-go peers as Windows-MCP and Terminator, both desktop MCP servers, and positions browser-use and Playwright MCP as adjacent browser-only tools. The comparison table is the useful artifact here, because it isolates three rows where Clawd Cursor claims to be alone: verifying its own actions, a single safety chokepoint, and perception without a vision model.

Against Windows-MCP and Terminator, the difference is breadth and the perception strategy. Windows-MCP is, by name, a Windows tool; Clawd Cursor's README claims Windows, macOS and Linux and ships a cross-platform test workflow at .github/workflows/cross-platform.yml. Whether that breadth holds on your specific Linux desktop environment is something the README does not answer, and the honest position is that you should test it on the target rather than assume it.

Against OmniParser and UI-TARS, the difference is architectural rather than a feature list. Those are vision-centric parsing approaches, in the README's phrasing, that you build an agent around. Every observation goes through a model that sees pixels. Clawd Cursor inverts the default: pixels are the exception, and the accessibility tree is the normal path. That is a real trade. Vision generalizes to anything a human can see, including a canvas with no semantics. An accessibility tree is empty or useless on exactly those surfaces, which is why the screenshot tier exists as an escape hatch rather than a fallback nobody uses.

Against the vendor computer-use APIs, the difference is locality and lock-in. The README's table marks those as Claude only and sandbox, with screens going to the cloud. Clawd Cursor is local-only, and the README notes that with a local model nothing leaves the machine, with ollama listed as a supported provider in .env.example.

Maintenance, licence and upgrade cost

The repository is not archived, and the last push was on 2026-09-14, two days before this writing. The most recent release listed is v1.5.9 from 2026-07-03, titled batch confirm-tier hardening (security), preceded by v1.5.8 (the agent path clicks straight) and v1.5.7 (accuracy + repo polish). The pattern is frequent small releases with security and accuracy fixes rather than long-lived major versions, which means the upgrade cost is low per release but the cadence is high enough that pinning a version is worth considering for anything you depend on.

The licence is MIT, per the LICENSE file and the badge in the README. That is permissive: you can use it commercially, modify it and redistribute it, subject to the usual attribution requirement. None of this is legal advice, and the native components deserve a look, since the published file list includes native/Package.swift, native/entitlements.plist and native/Sources/, so the repository is not pure TypeScript even though TypeScript is the primary language.

The operational cost that matters more than the licence is the permission surface. The postinstall step runs a native build, and the runtime needs screen observation rights. Those are per-machine grants, so a fleet rollout is a provisioning problem, not an npm problem.

Editorial conclusion

Adopt Clawd Cursor when the target is a GUI with no API and the work is repetitive enough to be worth turning into a script: it is the only one of these tools that reports a DEVIATION rather than a hollow success, and the single safety.evaluate() chokepoint is the reason to prefer it over wiring raw clicks into your own loop. Do not adopt it as a browser automation framework, as a cloud sandbox, or on a machine where you cannot grant accessibility and screen-recording permissions. Before trusting it, run clawdcursor doctor on the actual target machine, then write one script with an expect clause and watch whether the engine reports a DEVIATION when you deliberately break the flow.

Frequently asked questions

Official sources

1. AmrDab/clawdcursor on GitHub
2. License: MIT
3. Project website
4. README
5. Releases