Self-Operating Computer: a multimodal model driving mouse and keyboard

The model sees the screen and answers with clicks and keys

The Self-Operating Computer Framework lets a multimodal model operate a computer using the same inputs and outputs a human operator has. The model views the screen, and it decides on a series of mouse and keyboard actions to reach an objective you give it.

The project says it was released in November 2023 and was one of the first examples of full computer-use. The mechanics are visible in the dependency list rather than in an architecture diagram. PyAutoGUI provides the mouse and keyboard control, mss and pyscreenshot capture the screen, PyGetWindow and PyRect deal with window geometry, and openai, google-generativeai, anthropic and ollama are the model clients. The framework itself is the loop that ties those pieces together: screenshot, prompt, parse the returned action, perform it, repeat.

The audience is a developer who wants to see computer use work, and a researcher comparing how different models handle the same task, since the model backend is a command line switch and the screen handling is identical across all of them.

One operate command, seven model backends

Compatibility with different multimodal models is the first listed feature, and the integrated set covers GPT-4o, GPT-4.1, o1, Gemini Pro Vision, Claude 3, Qwen-VL and LLaVa. Additional models are named as future plans.

The default is gpt-4o, which is what you get by typing operate. The other backends are selected with the -m flag, and the naming is not uniform, which tells you something about the shape of the project. The OpenAI models come in OCR variants, so you run operate -m gpt-4.1-with-ocr or operate -m o1-with-ocr. The others are named after the model alone: operate -m gemini-pro-vision, operate -m claude-3, operate -m qwen-vl, operate -m llava.

Each backend has its own key. Gemini needs a Google AI Studio key entered at the prompt, and the README notes that you may also need to authorize credentials for a desktop application. Claude needs a key from the Claude dashboard, and Qwen needs one from the Qwen dashboard. The OpenAI key is the odd one out: it is read from a .env file, and to change it later you run vim .env and replace the old value.

OCR mode turns the screen into a hash map of clickable elements

The most important mechanism in the project is the OCR mode, and it is the default. The mode gives the model a hash map of clickable elements by coordinates. GPT-4 then decides to click an element by its text, and the code looks that text up in the hash map to get the coordinates of the element the model wanted.

That is a meaningful change in what the model has to do. Instead of reading pixel positions and estimating where a button is, the model names the thing it wants in words and the code resolves the position. The README states that based on recent tests OCR performs better than both som and vanilla GPT-4, which is why it was made the default, and notes that plain operate and operate -m gpt-4-with-ocr both reach it.

The trade-off is that the model now depends on text being on the screen. An interface that is mostly icons gives the hash map less to work with, which is the weakness the alternative mode exists to address.

Set-of-Mark mode adds a YOLOv8 button detector you can replace

The second visual strategy is Set-of-Mark, or SoM, prompting, selected with operate -m gpt-4-with-som. The idea is to enhance the visual grounding of the model by marking regions of the screen, and the README links an arXiv paper on the technique.

The implementation is deliberately modest and deliberately replaceable. For this initial version a simple YOLOv8 model is trained for button detection, and the best.pt file is included under model/weights/. Users are encouraged to swap in their own best.pt file to evaluate performance improvements, and the project asks that a better model be sent as a pull request. setup.py confirms the arrangement: the package data entry ships best.pt as part of operate.models.weights.

So the two modes answer different failures. OCR resolves elements the model can read, SoM resolves elements it can only see, and the project lets you replace the detector without touching the loop.

Installing it: pip, then operate, then two macOS permissions

The installation is short, and the last step is the one people miss. The package installs from PyPI and the console script is called operate:

pip install self-operating-computer
operate

The first run asks for your OpenAI key, and the framework writes it to a .env file in the working directory. After that the Terminal app asks for permission for Screen Recording and Accessibility on the Security & Privacy page of macOS System Settings. Without those two grants the framework cannot see the screen or move the pointer, and the failure looks like a model that does nothing.

The three common starting points, with the default OCR mode and the two visual alternatives, look like this:

operate -m gpt-4-with-ocr
operate -m gpt-4-with-som
operate -m llava

Note the last one, which needs setup before it works.

Voice mode needs a cloned repo, and LLaVa needs Ollama

Two paths sit outside the pip install, and both change what you are testing.

Voice mode, run with operate --voice, takes the objective by speech. It needs the repository rather than the package, so you clone it, install the extra requirements and install PortAudio, which on macOS comes from brew install portaudio and on Linux from sudo apt install portaudio19-dev python3-pyaudio. Then you run operate --voice from inside the cloned directory.

git clone https://github.com/OthersideAI/self-operating-computer.git
cd self-operating-computer
pip install -r requirements-audio.txt

The other path is running the model locally instead of calling a hosted API. Ollama serves LLaVA on your own machine, which is the real alternative to the cloud backends because the screen and the model never leave the workstation. The setup is a pull and a serve:

ollama pull llava
ollama serve

The README is blunt about the result: error rates when using LLaVA are very high, and this is intended as a base to build on as local multimodal models improve. Ollama currently supports macOS and Linux, with Windows in preview, and the model download takes about 5 GB.

setup.py still says 1.5.8 while main keeps moving

Version and maintenance need care here, because the two signals disagree. The last tagged release is v1.5.8, published on 2025-02-28, preceded by v1.5.7 and v1.5.6 in January 2025. The version string in setup.py is also 1.5.8. Yet the repository is not archived and the last push to the main branch was on 2026-10-19, more than a year after that tag.

So the work continues on main while the packaged version and the release tags have not moved. Anyone installing from PyPI gets the 1.5.8 line, and anyone tracking main gets code that the setup.py version does not describe. The top level holds evaluate.py, the operate package, readme/, requirements-audio.txt, requirements.txt and setup.py, with no changelog file, so the main branch is where the evaluation and development work sits.

The licence is MIT, so the code can be reused and modified freely, and the dependency pinning in requirements.txt is unusually complete, with openai at 1.2.3, anthropic unpinned, ultralytics at 8.0.227, easyocr at 1.7.1 and ollama at 0.1.6. A pinned old openai client and a model family that has moved on quickly is the practical upgrade cost here.

Editorial conclusion

Use it to prototype computer-use agents and to study how a screen is turned into a sequence of clicks and keystrokes, not as production automation, since a wrong action is a real click on a real machine. Do not reach for it when the target is a web API or a structured UI, where a script is safer. First check the screen recording and accessibility permissions on macOS, and pin a release rather than tracking main, because setup.py still reports version 1.5.8 while the branch has moved on.

Frequently asked questions

Official sources

1. License: MIT
2. OthersideAI/self-operating-computer on GitHub
3. Project website
4. README
5. Releases