computer-use-linux: a Rust MCP server that drives a real Linux desktop

The gap: MCP desktop control that is not macOS-only

Most computer-use MCP servers target macOS, where the README says they lean on AppKit, AXUIElement and CGEvent. The Linux options it names are narrower: driving xdotool against an X11 root window, or shelling out to OCR over screenshots. Neither gives an agent a structured view of the interface. xdotool assumes X11, which is the wrong assumption on a modern GNOME or KDE session, and OCR over pixels throws away roles, names and states that the application already publishes.

This project is for people running an MCP host on a Linux workstation and wanting the agent to operate the same desktop a human uses. The README lists Codex Desktop's Linux build, Claude Desktop, Hermes Agent and custom clients as hosts that can spawn the binary. The scope is deliberately local: the server runs on the machine whose desktop it controls, so there is no remote transport story here.

Four backends in a row, and a report that says which one won

Window targeting is the clearest example of the design. The registry tries GNOME Shell extension, GNOME Shell Introspect, the COSMIC Wayland helper, KWin DBus scripting, Hyprland hyprctl, i3 IPC and generic X11/EWMH, in that order. It then reports which backend won, or why each one failed. That ordering is a real decision: it prefers compositor-native interfaces over the X11 fallback, so on a Wayland session you are not silently routed through an XWayland compatibility layer.

Input follows a similar cascade. Pointer actions can go through org.freedesktop.portal.RemoteDesktop on Wayland, with ydotool and ydotoold over uinput as the deterministic fallback. For literal text the server prefers wtype on compatible Wayland compositors when portal keyboard input is unavailable, which the README says preserves Unicode and the active layout, before falling back to ydotool. Screenshots try the GNOME Shell DBus method first, then org.freedesktop.portal.Screenshot, then spawn gnome-screenshot for background or systemd contexts where both DBus paths are denied.

Three different mechanisms for one operation is a maintenance liability and the README does not hide it. It is also the honest answer to a fragmented desktop stack. The useful part is that the failure is observable rather than silent: the doctor tool returns a capability map of available backends alongside the readiness summary.

Semantic selectors versus pixel coordinates

The input tools accept role, name, text and states selectors backed by AT-SPI. click, perform_action and set_value all take them, and get_app_state returns a screenshot plus an accessibility tree with element indices that the input tools accept. Pixel coordinates remain available as a fallback, which the README scopes to rendering-only surfaces such as canvas, games and X clients without ATK.

This is the part that separates the project from screenshot-driven agents. A click by role and name survives a window move, a theme change and a resolution change. A click by coordinate does not. The trade-off is that the semantic path depends entirely on the application publishing an accessibility tree. GTK and Qt apps generally do. A game engine, an Electron app with accessibility disabled, or a terminal emulator that exposes little will push you back to coordinates and to screenshot interpretation, at which point the agent is doing vision work and the selector advantage evaporates.

The README also notes that targeted press_key and type_text results append focused-element feedback from AT-SPI (role, name, editable) and warn when no editable element holds focus. That feedback loop matters more than it sounds: typing into the wrong window is the classic way an agent run goes wrong, and a warning attached to the tool result is cheaper than a screenshot round trip.

Getting it running, and what doctor tells you

Install is two commands. From npm:

npm install -g @agent-sh/computer-use-linux computer-use-linux doctor | jq .readiness

The Rust crate is published on crates.io as computer-use-linux, and prebuilt binaries ship with the latest release. The crate installs the main computer-use-linux binary plus a small computer-use-linux-cosmic helper used only for COSMIC Wayland window management.

Two setup tools exist for the cases where the environment is not ready out of the box. setup_accessibility enables GNOME's org.gnome.desktop.interface toolkit-accessibility setting so toolkit apps expose AT-SPI trees. setup_window_targeting installs and enables the bundled GNOME Shell extension when org.gnome.Shell.Introspect is locked down. Both are MCP tools, so a host can call them, but they change desktop settings and are worth running deliberately rather than letting an agent invoke them unattended.

Screenshot payloads are bounded before they reach the host: max 1920 px width and height and 2 MiB of image bytes, with hard caps even when callers ask for more. Callers can pass max_width, max_height, max_bytes, scale, format: "jpeg" or quality, and the returned metadata includes coordinate_width, coordinate_height, scale, format and quality so a downscaled preview can be mapped back to desktop coordinates. If you are building a client, that metadata is what you need to translate a model's guess about a pixel into a real click.

Where it stops working

The README is explicit that this is Wayland-first and X11 best-effort, which is the inverse of most existing Linux automation tooling and means your X11 experience may be the weaker one. More importantly, every backend it probes is a desktop-session backend. There is no headless mode described. A container with no compositor, no session bus and no AT-SPI registry will fail the windowing and accessibility probes, and doctor will say so rather than pretend.

Background and systemd contexts are a second weak spot, acknowledged in the screenshot path: gnome-screenshot is spawned only when both DBus screenshot routes are denied. If you run the server as a service rather than inside the user session, expect the portal and DBus paths to be unavailable and the fallback to do the work, with the permissions and display assumptions that implies.

Third, the semantic path is only as good as the target application's accessibility support. Nothing in the README suggests a workaround beyond coordinates. If your workload is a single custom-rendered application, the AT-SPI layer may contribute nothing and you are paying for a dependency you do not use.

Finally, the project is young. Releases v0.4.9 through v0.5.0 landed between August 11 and August 31, 2026, which is a fast cadence for something that changes desktop settings and injects input events.

The alternative it is actually replacing

The comparison the README draws is with xdotool-style X11 automation and OCR-over-screenshot agents. The difference is not just Wayland support. xdotool addresses a root window and injects events at coordinates; it has no concept of an element's role or name, so a script that clicks at (840, 512) breaks the moment anything moves. OCR agents recover some of that by reading pixels, but they reconstruct structure the application already exposes, and they pay a screenshot, a model call and a coordinate conversion for every step.

This project inverts the order: ask the accessibility tree what is on screen, act on the named element, and fall back to pixels only when the tree is empty. That is a different cost profile. You spend setup effort on AT-SPI, a compositor-specific window backend and an input daemon, and in exchange most interactions do not require a screenshot at all. If your applications expose accessibility trees, that is a straightforward win. If they do not, the fallback path leaves you roughly where the OCR approach started, minus the OCR.

Maintenance, licensing and the upstream relationship

The licence is MIT, which is permissive and imposes no copyleft obligation on a host that spawns the binary. That is a statement about the licence text, not legal advice; if you redistribute it inside a product, read the LICENSE file and get your own answer.

The crate was extracted from codex-desktop-linux, the Linux distribution of Codex Desktop, which still bundles this binary as a built-in plugin. The README calls this standalone repo the upstream. That is a useful signal and a useful risk at once: the code has a real consumer with its own release schedule, so regressions are likely to surface, but you are also adopting something whose original purpose was to serve one host's needs. Features that matter to Codex Desktop will get attention first.

Upgrade cost is dominated by the environment, not the package. A new version can change which backend wins a probe, and the compositor extension and the COSMIC helper are separate artifacts that have to stay in step with the main binary. Re-running doctor after every upgrade and diffing the capability map is the cheap way to catch that. The npm wrapper and the crates.io crate are two distribution channels for the same server, so pick one and pin it rather than mixing global installs.

Editorial conclusion

Adopt it if you are running an MCP host on a Wayland GNOME, KDE, Hyprland, i3 or COSMIC desktop and you want semantic selectors rather than screenshot OCR. Do not adopt it if your target is a headless CI container with no compositor, no AT-SPI registry and no portal, or if you need macOS or Windows control, because none of the backends it probes exist there. Before wiring it into anything, run computer-use-linux doctor and read the readiness blockers and the capability map: a green install means nothing if the windowing backend list shows every compositor probe failing and the input section reports neither the RemoteDesktop portal nor a working ydotoold socket.

Official sources

1. agent-sh/computer-use-linux on GitHub
2. Issues
3. License: MIT
4. README
5. Releases