Open Computer Use: Driving an E2B Desktop Sandbox with Swappable Open Models
AI computer use powered by open source LLMs and E2B Desktop Sandbox
At a glance
- What is it?
- A Python agent that clicks, types and runs shell commands inside a cloud Linux desktop, with the grounding, vision and action models selected in a single config file. The interesting part is the provider swap; the uninteresting part is that nothing here is a packaged product yet.
- Who is it for?
- Adopt it if you want to experiment with computer-use agents on open weights and you are comfortable editing config.py and providers.py yourself. Do not adopt it if you need a supported release, a pinned dependency set or a headless deployment path, because none of those are visible in the repository material.
- Can I use it commercially?
- Yes. Apache-2.0 is a permissive licence: you can use, modify and sell software built on it, as long as you keep its copyright and licence notices.
- Is it still maintained?
- Yes. The repository last received commits 68 days ago.
- What is it written in?
- Mainly Python, according to GitHub's language statistics.
Answers come from the project's GitHub data, last synced on September 15, 2026, and from our analysis. They are not legal advice.
DEEP OPEN-SOURCE ANALYSIS
The gap between a chat model and a machine that has a mouse
A language model that can describe how to file a support ticket is not the same as a program that files one. Open Computer Use exists to close that distance. It gives an agent a real Ubuntu desktop in the cloud and the three input channels a human would use: keyboard, mouse and shell. The README states the project is powered by E2B Desktop Sandbox and controlled by open source LLMs, and that the agent can be paused and prompted at any time while its display streams live to the client machine. That pause-and-prompt loop is the actual product idea. Most computer-use demos run to completion or fail; this one keeps a human in the loop without taking the screen away from the agent. The intended audience is narrow and specific: engineers who already have an E2B API key and at least one LLM provider key, and who want to see how a grounding model, a vision model and an action model divide the work. It is not aimed at someone who wants a desktop automation library for a local machine. The sandbox is remote by design, and the README treats Ubuntu as a starting point rather than a constraint, noting the project is designed to work with any operating system.
Three models, one loop: grounding, vision and action
The architecture splits model duties rather than asking one model to do everything. In config.py the README shows three assignments: grounding_model = providers.OSAtlasProvider(), vision_model = providers.GroqProvider("llama3.2") and action_model = providers.GroqProvider("llama3.3"). That is the whole design in three lines. A grounding model turns a description of a screen element into coordinates. A vision model reads the display. An action model decides what to do next, which the agent then executes through the keyboard, mouse or shell inside the sandbox. Because the three roles are separate objects, you can run OS-Atlas for grounding while Groq serves the other two, or move everything to a single provider that covers all three capabilities. The README's provider table is explicit about which models cover which role. Llama 3.2 is vision only on Fireworks, OpenRouter and Llama API, while Llama 3.3 is action only there. Groq lists Llama 3.2 for vision and action, and Llama 3.3 for action only. DeepSeek is action only. Gemini 2.0 Flash, GPT-4o, GPT-4o mini and Claude are listed as vision plus action. OS-Atlas and ShowUI appear as grounding models hosted on Hugging Face Spaces. Moonshot and Mistral AI are listed as providers, with Pixtral for vision and Mistral Large for actions. The README also states the project supports 10+ LLMs and invites pull requests to providers.py for new models. That invitation is a fair signal of maturity: the provider layer is meant to be edited, not configured.
Getting it running: poetry, ffmpeg and a .env file
The prerequisites are Python 3.10 or later, git, an E2B API key and an API key for whichever LLM provider you select. Installation starts with brew install poetry ffmpeg, which tells you the README assumes macOS. On Linux you would need to install Poetry and ffmpeg through your distribution's package manager instead, and the repository material does not cover that path. After git clone https://github.com/e2b-dev/open-computer-use/ and cd open-computer-use, you create a .env file in the project directory. E2B_API_KEY is mandatory. The LLM keys are conditional: you only need the key for the provider or providers selected in config.py, and the README notes that Hugging Face Spaces do not require an API key. The listed variables are FIREWORKS_API_KEY, OPENROUTER_API_KEY, LLAMA_API_KEY, GROQ_API_KEY, GEMINI_API_KEY, OPENAI_API_KEY, ANTHROPIC_API_KEY and MOONSHOT_API_KEY. HF_TOKEN is separate and described as required to bypass Gradio rate limits, which matters if your grounding model is OS-Atlas or ShowUI. Then poetry install, followed by poetry run start. The agent opens and prompts for its first instruction. You can skip the prompt with poetry run start --prompt "use the web browser to get the current weather in sf". The README states the display stream should be visible a few seconds after the Python program starts. If it is not, the first thing to check is ffmpeg, since it is installed as a prerequisite and the stream is the only feedback channel you have.
The rate limit you inherit from Hugging Face Spaces
OS-Atlas and ShowUI are the default grounding options in the README's example, and both are reached through Hugging Face Spaces. That is convenient because Spaces need no API key, but it puts a third party's Gradio rate limiting directly in the agent's path. The README addresses this by listing HF_TOKEN as required to bypass those limits. The word required is doing real work there. Without it, a grounding call can fail mid-task, and a failed grounding call is not a graceful degradation: the agent has no coordinates, so the action model's decision cannot be executed. The failure lands in the middle of a mouse or keyboard sequence rather than at startup. If you swap grounding to a paid provider you remove that dependency, but the README's provider list does not show another grounding-only option besides the two Spaces. The practical consequence is that the cheapest configuration is also the one with the least predictable latency. That is a trade-off worth naming rather than a defect to file.
No releases, no version pins, no upgrade story
The repository shows no retrieved releases, and the README documents no version pinning beyond Python 3.10 or later. There is no changelog in the supplied material, no migration notes and no compatibility matrix for the provider SDKs. That shapes the maintenance cost in a particular way. You are tracking the master branch of a project whose provider layer is explicitly open to pull requests, so a merged change to providers.py can alter an interface you depend on without a version number to hold you back. Dependencies arrive through Poetry, which gives you a lock file once you run poetry install, and that lock file is your only stability guarantee. Treat it as one. Upgrading means re-running poetry install against a newer lock and re-checking that your three model assignments in config.py still map to provider classes that exist. If you fork, the README's request to open a pull request for new providers suggests upstream would rather absorb provider additions than maintain a plugin boundary, which is good for the ecosystem and mildly awkward if your provider is internal. The Apache-2.0 licence permits modification and redistribution, and it includes a patent grant; it also means the project ships without warranty. That is a statement about the licence text, not legal advice, and if you plan to redistribute a modified version you should read the notice and attribution requirements yourself rather than take this paragraph as sufficient.
Where a scripted browser driver is the better tool
Playwright and Selenium attack the same surface from the opposite direction. They drive a browser through a documented API: selectors, DOM events, deterministic waits, and a test runner that reports pass or fail. Open Computer Use drives pixels and keystrokes through model inference, which means the same instruction can produce different actions on two runs and there is no assertion primitive in the supplied material. If your target is one web application with stable markup and you need a regression suite that runs in CI, a scripted driver wins on every axis that matters: speed, determinism, debuggability and cost. The case for this project is the opposite one. When the target is an arbitrary desktop, when the workflow crosses applications that expose no API, or when you are evaluating how well a grounding model locates a button described in words, a scripted driver cannot express the task at all. The honest framing is that these are different problem classes. Using Open Computer Use for a task Playwright can already do reliably is paying inference cost and accepting nondeterminism for nothing.
Who this is for, and what to check before you commit
The strongest fit is an engineer evaluating computer-use models who wants a working harness rather than a paper. The three-slot config makes provider comparison cheap: change the grounding or action assignment, restart with poetry run start, and watch the stream. The pause-and-prompt control means a bad trajectory does not have to run to completion before you intervene. The weakest fit is a team that needs a supported dependency with a release cadence, because none is visible here. A second weak fit is anyone deploying headless. The display stream and the interactive prompt are central to how the project is described, and the README does not describe a server mode or a CI entry point. Before you build on it, confirm that the model you intend to use covers the role you are assigning it, since the provider table is capability-specific and a vision-only assignment will not drive actions. Confirm that HF_TOKEN is set if grounding runs through Hugging Face Spaces. Confirm that ffmpeg is on PATH, because the stream depends on it. Beyond that, the repository is small enough to read end to end, and config.py is the file that decides what your agent actually is.
Editorial conclusion
Adopt it if you want to experiment with computer-use agents on open weights and you are comfortable editing config.py and providers.py yourself. Do not adopt it if you need a supported release, a pinned dependency set or a headless deployment path, because none of those are visible in the repository material. Before committing, verify three things: that the provider you intend to use is listed for the capability you need (Llama 3.2 is vision only on Fireworks, OpenRouter and Llama API, while action requires Llama 3.3 or another action-capable model), that your HF_TOKEN is set if you rely on the OS-Atlas or ShowUI Hugging Face Spaces, and that your E2B_API_KEY and chosen LLM key are both present in the .env file before running poetry run start.
Community notes