AutoPy drives the mouse and keyboard from Python on a Rust core

One import over a Rust library

AutoPy is described as a simple, cross-platform GUI automation module for Python and Rust, and the interesting part of the design is where the work happens. The Python side is a thin binding layer over a Rust crate, and the Cargo manifest makes that explicit: the package is a cdylib named autopy, it depends on the autopilot crate at version 0.4.0, and it binds to Python through pyo3 at version 0.29.2. Alongside the automation work sit image handling, a choice combinator, and libc for the platform calls. The reason to prefer this shape over a pure Python implementation is straightforward. Talking to a window server, injecting input events and reading framebuffers are all operations where the language runtime overhead and the garbage collector get in the way, and moving them into a native library keeps the latency predictable. From Python you still get a normal module, so the ergonomics are unchanged for the script author.

maturin and PyO3 define the build

The build configuration is modern and short. The pyproject file names maturin with a version range of at least 1.5 and below 2.0 as the build backend, marks the bindings as pyo3, and enables the pyo3 extension-module feature so the compiled library loads as a native extension rather than linking the interpreter. The project requires Python 3.8 and onwards, and the classifiers list both CPython and PyPy, which is an unusual pairing for a native extension and a claim worth testing on your own runtime. The version is marked as dynamic in the project metadata, so the package version is not written in pyproject.toml, and the version metadata is instead read out of the Python package at build time. The repository also carries a setup.py that still performs the setuptools_rust style build, along with a requirements.txt listing setuptools-rust, plus a rust-toolchain.toml and a Makefile, so there are two build paths and the maturin one is the current one.

Install the wheel first, build from source only if you must

The installation section gives three routes, and the order matters. The first attempt is a binary wheel from PyPI:

$ pip install -U autopy

If that fails, the reason is usually a missing Rust toolchain, and the documented fix is to install rustup and then update it:

$ rustup update stable
$ pip install -U autopy

The third route builds from the GitHub source, which creates a virtual environment and compiles the extension in place with maturin:

$ git clone git://github.com/autopilot-rs/autopy-rs.git
$ cd autopy
$ python -m venv .env && source .env/bin/activate
$ maturin develop

The requirements list a Rust 1.98.1 toolchain for source builds, macOS 10.6 and up, Windows 7 and up, and X11 with the XTest extension. The README notes that additional and possibly outdated instructions for installing from source on Windows live in a separate setup document, which is a polite way of saying that Windows source installs are the rough path.

Mouse movement, from teleports to a sine wave

The mouse API is where most scripts start. There is an immediate move, a smoother variant that traverses the distance rather than jumping, and a full set of mouse controls documented in the API reference on the project site. The tutorial shows the pattern by writing a function that moves the pointer across the screen as a sine wave, which doubles as an example of how to read the screen size and how to pace the movement. The same script exists in the examples folder as sine_mouse_wave.py, alongside hello_world_alert.py and screengrab.py. The tutorial version reads the screen dimensions with autopy.screen.size(), halves the height and subtracts ten to stay inside the bounds, then steps along the x axis while computing the y value from the sine curve and sleeping for a random short interval between one and three milliseconds. That last detail is the useful lesson, because a script that moves the pointer instantly looks nothing like a person, and the random delay is what makes the trace look plausible.

Typing at a chosen speed and tapping with modifiers

The keyboard side has two levels of control. The first is a single call that types a whole string at a chosen speed, measured in words per minute, which is enough for most form filling tasks:

import autopy


autopy.key.type_string("Hello, world!", wpm=100)

The second is individual key events with modifiers held down, expressed through the Code and Modifier enums:

import autopy


autopy.key.tap(autopy.key.Code.TAB, [autopy.key.Modifier.META])
autopy.key.tap("w", [autopy.key.Modifier.META])

The modifier list being a list rather than a single value is what allows chords, and the META modifier is the interesting one on macOS, where the command key is spelled differently from the control key on other platforms. A script written with META will therefore behave differently on Windows or Linux, which is the kind of portability detail that bites when a script that was developed on a Mac is run in CI. The simplest use of the module is the alert, which is the hello world example and the fastest way to confirm that the native library loaded and the window server connection works at all.

Bitmaps: capture the screen, then look for the needle

The bitmap module is the part of AutoPy that other automation libraries are usually compared against, and it works in two stages. The first is capture and inspection. A screen capture returns a Bitmap object, and asking it for the colour of a pixel returns a packed integer that Python's hex function renders in the familiar form, which is why the tutorial goes through that conversion rather than presenting the raw number. The coordinate system has its origin at the top left, so the pixel you ask about first is the top left corner of the screen. The second stage is search. You open a needle image and a haystack image, then ask the haystack to find the needle, which returns a position if there is one:

import autopy


def find_image_example():
    needle = autopy.bitmap.Bitmap.open('needle.png')
    haystack = autopy.bitmap.Bitmap.open('haystack.png')

    pos = haystack.find_bitmap(needle)
    if pos:
        print("Found needle at: %s" % str(pos))

find_image_example()

The search scans from left to right and top to bottom, and it can be restricted to a rectangle by passing a rect argument, which is how you avoid matching a button icon that appears in twenty places on a page:

haystack.find_bitmap(needle, rect=((10, 10), (100, 100)))

Saving a capture to disk works too, with the file type parsed from the filename or passed in, and only JPEG and PNG are supported. A convenience method on the screen module returns a single colour more cheaply than capturing a whole bitmap, which matters in a loop that samples a pixel thousands of times.

X11 with XTest, and no Wayland story

The platform list is the hard limit to plan around. AutoPy is supported on macOS, Windows, and X11 with the XTest extension, and that third entry is more restrictive than it looks, because XTest is the injection mechanism the project targets for synthetic input. The project does not describe a Wayland path, so on a modern Wayland desktop you would be running the application under XWayland, where the coordinate space and the pointer behaviour are those of the X server rather than the compositor. The mouse coordinate system is also explicitly top-left origin, which differs from the bottom-left convention in some other toolkits and is worth stating in any script that has to work in both environments. Beyond that, the documentation surface is split: the README carries the tutorial, the full API reference for the mouse and the other modules lives on the separate project website, and the repository publishes no GitHub releases, so the version you get from PyPI is the version to pin. The last push was on 2026-09-28, and the crate version in the Cargo manifest is 4.0.2.

Editorial conclusion

AutoPy fits a test author or operations script that needs to move a pointer, type text, or find a picture on screen across macOS, Windows and X11 from one Python API, and the PyO3 layer is thin enough that the Rust crate underneath is a realistic fallback if you need to drop lower. It does not fit a Linux shop that has moved to Wayland, because the supported display server is X11 with the XTest extension, and it does not fit a team that wants a pure Python dependency, since a source install pulls in a Rust toolchain. Before you build from source, install rustup and run rustup update stable so the toolchain matches the Rust 1.98.1 the project asks for, prefer the binary wheel with pip install -U autopy, and confirm the crate version you get, because the Cargo manifest reads 4.0.2 while the published package version is dynamic and the repository publishes no GitHub releases.

Frequently asked questions

Official sources

1. autopilot-rs/autopy on GitHub
2. Issues
3. License: Apache-2.0
4. Project website
5. README