computer-use-linux: Wayland-first desktop control for MCP hosts

The Linux gap that computer-use-linux fills

Computer-use MCP servers mostly grew up on macOS, where AppKit, AXUIElement and CGEvent give one vendor's answer to accessibility, input and screenshots. Linux has no equivalent single stack. The README states the project's own framing of the problem: the few Linux-targeting servers either drive xdotool against an X11 root window or shell out to OCR over screenshots. Neither survives a Wayland session, and OCR over pixels throws away the role, name and state information that applications already publish through AT-SPI.

The intended user is someone running an MCP host on a Linux desktop and wanting that host to operate real applications. The README names Codex Desktop's Linux build, Claude Desktop and Hermes Agent as hosts that can spawn the server, and notes the crate was extracted from codex-desktop-linux, which still bundles the binary as a built-in plugin. So the audience is narrow and concrete: agent developers and power users on Linux, not people looking for a cross-platform automation library. If you are on macOS, this is the wrong repository and the README says so indirectly by listing what the macOS servers already do.

How the input, window and screenshot backends are chosen

The architecture is a set of ordered fallback chains, and the ordering is the design. For pointer actions, the README says actions can use the org.freedesktop.portal.RemoteDesktop interface on Wayland, with ydotool and ydotoold over uinput as the deterministic fallback. For literal text, wtype is preferred on compatible Wayland compositors when portal keyboard input is unavailable, because it preserves Unicode and the active layout, before falling back to ydotool. Screenshots try the GNOME Shell DBus screenshot method first, then org.freedesktop.portal.Screenshot, then spawn gnome-screenshot for background and systemd contexts where both DBus paths are denied.

Window targeting is the same idea with more entries. The window registry tries the GNOME Shell extension, GNOME Shell Introspect, the COSMIC Wayland helper, KWin DBus scripting, hyprctl, i3 IPC, and generic X11/EWMH, in that order, and reports which backend won or why each one failed. That reporting is the part worth noticing. A fallback chain that silently degrades is hard to debug; one that tells you which rung it landed on is not.

On top of that sits AT-SPI. Tools such as click, perform_action and set_value accept role, name, text and states selectors, and get_app_state returns a screenshot plus an accessibility tree with element indices that the input tools accept. Pixel coordinates remain available, and the README is explicit that they exist as a fallback for rendering-only surfaces: canvas, games, X clients without ATK. The Cargo.toml comment about atspi is a useful detail: the dependency disables the atspi connection feature and depends on atspi-connection directly with p2p off, because p2p peer enumeration calls GetInterfaces on every a11y-bus peer and a single AppArmor-confined peer such as a snap-packaged Slack aborts the whole AccessibilityConnection. That is a real failure mode the maintainers hit and worked around at the dependency level.

Installing computer-use-linux and running a first readiness check

The README gives the npm route first. The package is @agent-sh/computer-use-linux and the binary it installs is computer-use-linux, so a global install puts the CLI on your PATH. The README also notes prebuilt binaries ship with the latest release, and the Rust crate is published on crates.io under the same name if you prefer cargo.

npm install -g @agent-sh/computer-use-linux
computer-use-linux doctor | jq .readiness

The second command is the one to run before anything else. According to the README, doctor returns a structured JSON document covering platform, portals, AT-SPI, windowing and input, plus a readiness summary with explicit blockers and a recommended next step. Piping it through jq .readiness narrows the output to that summary. If the summary reports blockers, the capability map in the same document tells you which backends were available, so you can see whether the problem is a missing portal, a locked-down Shell Introspect, or absent input tooling.

Two setup tools address the common blockers directly. setup_accessibility enables GNOME's org.gnome.desktop.interface toolkit-accessibility setting so toolkit apps expose AT-SPI trees, and setup_window_targeting installs and enables the bundled GNOME Shell extension when org.gnome.Shell.Introspect is locked down. Both are exposed as MCP tools, so a host can call them, and the repository ships a gnome-shell-extension directory for the second case. After that, the first useful call is usually list_windows to see what the window registry can actually target, followed by get_app_state on one application to get a screenshot and an element tree you can address by role and name.

Where the semantic layer breaks down

The accessibility approach is also the main limitation. Semantic selectors only work where an application publishes an AT-SPI tree. The README concedes that pixel coordinates remain a fallback for canvas, games and X clients without ATK, which means those surfaces get the weaker path: no roles, no names, no states, just coordinates that break when a window moves or a layout shifts. An agent that works well against a GTK form will be noticeably less reliable against a browser-rendered canvas or a game.

Screenshots carry a second constraint. Payloads are size-bounded by default before they reach the MCP host: max 1920 px width and height and 2 MiB of image bytes, with hard caps even when callers request more. That is sensible for agent context windows, but it means a full-desktop capture arrives downscaled. The README handles this by returning coordinate_width, coordinate_height, scale, format and quality in the screenshot metadata so callers can convert from the downscaled preview back to desktop coordinate pixels. Any client that ignores that metadata and clicks at raw image coordinates will be wrong, sometimes by a large factor. JPEG is offered as a way to trade lossless pixels for a smaller payload before the byte cap forces further resizing, which is a reasonable escape hatch but not a fix for wanting detail at native resolution.

There is also a platform boundary. The npm package declares os: linux and cpu x64, arm64, so it will not install elsewhere, and the whole backend selection assumes a Linux compositor. The README describes X11 as best-effort rather than a first-class target, which is honest but means X11-only users get the fallback rungs of every chain.

computer-use-linux compared with xdotool-based and OCR-based servers

The real alternative in this space is not another MCP server with the same design. It is the older approach the README contrasts with: a server that drives xdotool against an X11 root window, or one that screenshots the desktop and runs OCR to find text to click. The difference is structural rather than incremental.

xdotool against the X11 root window has one input model and one window model, and it works as long as you are on X11. On Wayland there is no root window to drive, so the approach either fails or needs a compatibility layer. computer-use-linux instead negotiates with whatever the compositor exposes: RemoteDesktop portals, hyprctl, i3 IPC, KWin DBus scripting, the GNOME Shell extension, the COSMIC helper. That is more moving parts and more places for a session to be misconfigured, which is exactly why the doctor report exists.

The OCR approach inverts the trade. It sees what a human sees, so it works on canvas and games where AT-SPI is absent, but it recovers text by guessing from pixels and has no notion of role, state or settable value. computer-use-linux's perform_action and set_value act on elements that declare those affordances, so a toggle reports its state and a text field accepts a value directly. On the surfaces where AT-SPI is missing, this project falls back to the same coordinate-level approach an OCR server would use, minus the OCR. The honest summary is that computer-use-linux is the better tool for toolkit applications on Wayland, and the OCR approach is more general on surfaces that publish nothing.

Maintenance, licence and upgrade cost

The repository is not archived, and the last push was on 2026-09-10, which is recent. Releases have been frequent: v0.5.0 on 2026-08-31, v0.4.10 on 2026-08-28, v0.4.9 on 2026-08-11. The Cargo.toml and package.json both report version 0.6.0, so the source tree is ahead of the newest listed release tag, which is normal for a project that tags after merging. A CHANGELOG.md sits at the repository root, and that is where upgrade notes would live; the README does not document rollback, so if you pin a version and it misbehaves, the changelog is your only in-repo record of what moved.

The licence is MIT for both the Rust crate and the npm package, per Cargo.toml, package.json and the LICENSE file. MIT is permissive and imposes no copyleft obligation on your own code, but this is a factual statement about the licence text, not legal advice; if you redistribute the binary, the usual attribution requirements apply and you should read LICENSE yourself. One licence-adjacent detail worth flagging: the npm package ships pi/extension/THIRD_PARTY_NOTICES.txt, which suggests the bundled Pi extension carries third-party code under its own terms. If you consume the npm package rather than the crate, that file is the one to read.

The upgrade cost is tied to the fallback chains. A new release can change which backend wins on your compositor, and the doctor report is the cheapest way to notice. Because the project deliberately reports why each backend failed, a version bump that changes backend selection is visible in that JSON rather than only in behaviour.

Editorial conclusion

Adopt computer-use-linux if you run a Wayland session on GNOME, KWin, Hyprland, i3 or COSMIC and you want an MCP host to drive real applications through accessibility semantics rather than pixel guessing. Skip it if your targets are canvas-only surfaces, games, or X11 clients with no ATK bridge, because the semantic selector path has nothing to bind to and you fall back to coordinates. Before wiring it into an agent, run computer-use-linux doctor and read the readiness block: it names the backends that won and the blockers that remain, and that report, not the feature list, tells you whether your session is actually supported.

Frequently asked questions

Official sources

1. agent-sh/computer-use-linux on GitHub
2. Issues
3. License: MIT
4. README
5. Releases