Surf: A Next.js Reference App for AI-Controlled Virtual Desktops via E2B and OpenAI

What Surf Demonstrates: Natural Language Control of a Virtual Desktop

Surf addresses a specific problem in AI agent development: giving an AI model a real operating environment to act in, not a browser tab or a code interpreter. The application wraps E2B's cloud-hosted Linux desktop sandbox with a chat interface, so a user can type an instruction like "Open Firefox and go to google.com" and watch an AI agent execute it on a real (virtual) desktop, clicking, typing, and navigating in real time.

The README describes this as a Next.js application that integrates E2B's desktop sandbox with OpenAI's API. It is primarily a demonstration of how computer use agents can be built, not a finished product to deploy for end users. The target audience is developers exploring how to wire together a cloud desktop environment, an AI vision and action API, and a streaming frontend into a working application.

Architecture: How the Sandbox, the AI, and the Frontend Connect

The application has four main components, as described in the README. The frontend is a Next.js application that renders the virtual desktop view and provides a chat interface. The E2B Desktop Sandbox runs a Linux-based desktop in an isolated cloud environment and streams its display back to the frontend. OpenAI's computer use capability processes user instructions and generates the actions (mouse clicks, keyboard input) to perform. A streaming API layer using Server-Sent Events (SSE) carries the AI's responses and action results from the backend to the browser in real time.

The core flow starts when the user clicks "Start new Sandbox," which triggers the createSandbox server action. E2B initializes the virtual desktop and returns a URL for streaming. When the user sends an instruction through the chat interface, the /api/chat endpoint processes it using the AI provider's API. The AI generates action steps, those steps execute on the sandbox, and the results stream back to the frontend. The user sees both the AI's reasoning in the chat panel and the desktop state updating live.

The README names SSE as the transport layer, which is a one-way server-to-client push mechanism. This is appropriate for streaming action output but means the client cannot interrupt an in-progress action sequence through the same channel.

Setting Up Surf Locally

The setup requires Node.js (version specified in package.json), an E2B API key, and an OpenAI API key. Clone the repository and install dependencies:

git clone https://github.com/e2b-dev/surf
cd surf

npm install

Create a .env.local file in the root directory. The .env.example file shows the required variables:

E2B_API_KEY=your_e2b_api_key
OPENAI_API_KEY=your_openai_api_key

The .env.example also includes a NEXT_PUBLIC_GTM_ID variable for Google Tag Manager, set to a placeholder GTM-XXXXXXX. This is used in deployed environments and is not required locally.

Start the development server:

npm run dev

The dev script pipes Next.js output through pino-pretty for formatted logging. Navigate to http://localhost:3000 to open the application. Once running, click "Start new Sandbox" to initialize a virtual desktop, then type an instruction in the chat input.

AI Provider Support: OpenAI and Several Others

The README describes Surf as powered by OpenAI, but the package.json tells a more complete story. The dependencies include AI SDK packages for OpenAI (@ai-sdk/openai), Google (@ai-sdk/google), Groq (@ai-sdk/groq), Mistral (@ai-sdk/mistral), and xAI (@ai-sdk/xai), alongside the Vercel AI SDK (ai 4.1.25) that unifies them under a common interface.

This suggests the provider is selectable, though the README does not document how to switch providers or what the configuration looks like for non-OpenAI options. The .env.example only shows E2B_API_KEY and OPENAI_API_KEY, implying that switching to a different provider likely requires both adding its API key to .env.local and changing the provider reference in the application code.

The inclusion of @gradio/client as a dependency is notable. Gradio is a Python-based library for building AI interfaces and model endpoints; its JavaScript client is used here, though the README does not explain for what purpose. This suggests Surf may support routing requests to a Gradio-hosted model in addition to the named AI SDK providers.

For developers who want to experiment with non-OpenAI computer use models, the dependency set gives a broader starting point than the README alone suggests.

Sandbox Lifecycle: Time Limits and Auto-Extension

Sandboxes in E2B are time-limited. The README notes that the interface shows a timer with the sandbox's remaining time and that the sandbox auto-extends its timeout when it is about to expire. The increaseTimeout server action handles this extension. You can also stop the sandbox explicitly by clicking the "Stop" button, which calls the stopSandboxAction server action.

This lifecycle model has a direct implication: any work done inside the sandbox (files created, software installed, settings changed) is ephemeral. When the sandbox stops or its time expires without being extended, that state is lost. The README does not document a way to persist sandbox state between sessions or to resume a stopped sandbox. Each "Start new Sandbox" action creates a fresh environment.

For most computer use demonstrations this is acceptable. For workflows that need to carry state forward across sessions, the ephemeral model is a hard constraint that would require changes to the application architecture.

Limitations and Comparison with Playwright

Several limitations follow from the architecture. The README does not document error handling for cases where the AI model misidentifies a UI element, executes the wrong action, or gets stuck in a loop. The troubleshooting section lists three failure modes (sandbox not starting, AI not responding, actions not working) and traces each to an API key problem, but does not address the broader question of how to recover from a mid-task failure.

The application is also tightly coupled to E2B's desktop sandbox. There is no documented way to substitute a different sandbox provider or run against a local virtual machine. Teams who want to use a different infrastructure would need to replace the @e2b/desktop SDK and the createSandbox, increaseTimeout, and stopSandboxAction server actions.

A common alternative for UI automation is Playwright, the cross-browser testing library from Microsoft. Playwright automates browser-level interactions through programmatic selectors and recorded flows: you define what to click, fill, or navigate using code. Surf, by contrast, uses an AI agent that interprets a natural language instruction and then decides which actions to take based on what it sees on the desktop. Playwright is more reliable for repeatable, well-defined workflows; Surf is more useful for exploratory or variable tasks that are difficult to encode as deterministic selectors.

The last push to the repository was on 2026-09-09 and the project is not archived. No GitHub releases have been published; the version in package.json is 1.0.0. The Apache-2.0 license applies, which permits commercial use and modification but requires preserving the license notice.

Editorial conclusion

Surf suits developers who want a concrete reference implementation for building computer use applications on top of E2B and an OpenAI-compatible provider. It is not a production framework or a reusable library; it is a runnable demo with a chat interface and a virtual desktop view. Teams evaluating it should check that their target AI provider is among those listed in package.json and that they have valid E2B and OpenAI API keys configured in .env.local before running.

Frequently asked questions

Official sources

1. e2b-dev/surf on GitHub
2. Issues
3. License: Apache-2.0
4. Project website
5. README