ESP32 iPhone Agent: AirPlay, USB HID, and On-Device Control

Why ESP32 iPhone Agent Exists

ESP32 iPhone Agent addresses a specific gap in AI phone automation: the model needs to see the real iPhone screen and produce real touch, swipe, and keyboard input, rather than operate a desktop application through a Mac or a remote browser session. The project targets users experimenting with agentic phone control, computer-use prompts, and local or cloud vision models. Its README presents the system as a combination of an inexpensive ESP32-S3 board and the phone itself. The board supplies the input hardware, and an iOS app supplies decoding, planning, acting, and task review.

The approach does not require a jailbreak, a Mac during normal use, or an Apple developer account for the basic app-signing path. It does require hardware preparation. The phone must be an iOS 18 or later device with cellular data, AssistiveTouch must be enabled, and the board must have native USB and PSRAM. The documented board is the plain Seeed Studio XIAO ESP32S3 with 8 MB PSRAM. The Sense and Plus variants are explicitly excluded from the stated test configuration.

This is not a general-purpose phone farm or background task runner. The app controls the phone through an on-screen pointer and keyboard emulation, so it is most appropriate for experiments where the model must interact with visible iOS interfaces. It also introduces dependencies that are unusual for an iPhone project: BLE communication, a board-hosted Wi-Fi hotspot, AirPlay screen mirroring, USB HID, and an external vision endpoint.

The ESP32, AirPlay, BLE, and App Pipeline

The system separates display input from model control. The iPhone sends Screen Mirroring over Wi-Fi to the ESP32-S3, which appears as “ESP32 Agent” in the phone's Screen Mirroring list. The board opens its own Wi-Fi hotspot and acts as the AirPlay receiver. Its README states that the H.264 stream is passed through without decoding on the board, then forwarded to the app over BLE.

The iPhone app decodes the mirrored video, sends screenshots to an OpenAI-compatible vision endpoint, and runs a planner and actor. The planner decides what should happen next, while the actor can tap, swipe, draw, type, press keys, go home, or finish. The same app communicates with the board over BLE, where the board exposes a USB HID mouse and keyboard. AssistiveTouch turns the pointer into visible touch interaction, so a click becomes a tap and a drag becomes a swipe.

The pointer is relative rather than absolute. According to the README, the app tracks the pointer and re-homes it in a corner. Before a precise tap, it wiggles the pointer, locates it in the mirrored image, and corrects the error. After each action, the app waits for the screen to settle and tells the model whether the screen changed. If the board, hotspot, or mirroring connection is lost, the run pauses and resumes from the same step. Each run is retained on the phone with step-by-step replay and overlaid taps and swipes.

Preparing the Hardware and iOS App

The first setup session is hardware-heavy rather than purely software-driven. Install `arduino-cli`, the ESP32 core 3.3.x, and NimBLE-Arduino 2.5.x on a Mac connected to the board. The firmware build uses `hardware/flash.sh` because it applies the BLE buffer sizes in `hardware/hid_bridge/nimble_cfg.h`.

The board must be flashed before it is moved to the iPhone. The phone powers the board through USB. USB-C models connect directly, while Lightning models require a Lightning to USB Camera Adapter. The board's firmware handles BLE GATT, the hotspot, the AirPlay receiver, and USB HID.

The iOS application is a SwiftUI project generated with XcodeGen from `ios/project.yml`. Xcode is needed once to build and install the app, even though no Mac is required during the normal run. A free Apple ID can sign the application, but the README says it must be re-signed every seven days. The paid Hotspot Configuration capability enables one-tap hotspot joining. Without that capability, the app displays the hotspot password for manual entry.

A First iPhone Agent Task

After flashing the board, copy the example Xcode configuration and set the signing values required by Apple. Open the project in Xcode and run the `IPhoneAgent` scheme on the iPhone. The app then guides the user through hardware pairing, screen access, pointer setup, and model configuration.

The documented setup sequence is: plug in the board for Bluetooth pairing, join its Wi-Fi network, select `ESP32 Agent` from Screen Mirroring, enable AssistiveTouch, allow the app to calibrate the pointer, and enter the model endpoint and API key. The key is stored in the Keychain.

A first task should be deliberately small. The README gives “Open Settings and go to Wi-Fi” as an example. Type that goal into the app and press `Start`. The app can be stopped from the Dynamic Island or by returning to it. Siri, Shortcuts, widgets, and Control Center can run saved tasks after the initial setup. Because the agent depends on screen recognition and pointer correction, a short navigation task is a better first check than an unattended transaction or a long workflow.

Model Compatibility and Prompt Customization

The app does not require one fixed model vendor. The README says any OpenAI-compatible chat API that accepts images can serve as the vision endpoint, including vLLM, SGLang, llama.cpp, Ollama, cloud APIs, or a user-operated server. The app lists models exposed by the endpoint and allows the user to select one. It also supports separate planner and actor models, which can be useful when one model handles task decomposition and another handles step execution.

The development example uses `nvidia/Qwen3.8-Flash-Next-NVFP4`, served on two DGX Sparks through TensorFold. That is a development example, not a stated requirement or a guarantee of supported model performance. The repository does not provide benchmark results, latency measurements, or accuracy claims.

Prompts are plain text under `prompts/` and are bundled into the app. Changing a planner or actor prompt requires editing the file, rebuilding the app, and reinstalling it. This is a direct customization path, but it also means prompt changes are not hot-swapped at run time. The model endpoint and key are configured in the app, while the prompt files are compiled into the application.

Operational Limits and Upgrade Cost

The main limitation is the dependency chain. A run requires the board to be connected, the BLE link and Wi-Fi hotspot to be available, AirPlay mirroring to remain active, and the phone to retain cellular data for the model request. The README specifically states that the phone continues using cellular data because the board's hotspot occupies Wi-Fi. A network, model, or board failure therefore interrupts an otherwise local-looking setup.

Pointer control also depends on AssistiveTouch and image-based correction. Relative pointer movement can be corrected by locating the pointer in the mirrored screen, but the repository does not document behavior for every iOS interface, accessibility state, orientation change, modal interruption, or rapidly changing screen. The project is consequently better suited to supervised experiments than to unattended actions involving sensitive data.

The repository is not archived, and its last push was on 2026-10-16. Its MIT license permits use and modification under the license terms, subject to the license text. The practical upgrade cost includes rebuilding the iOS app after firmware or prompt changes, resolving Apple signing requirements, and re-signing a free-account build every seven days. The repository keeps the previous computer-mediated implementation in the `legacy` branch and `v0.1.0` tag, while `server/` is documented as unused by the current version.

Installation Commands and Model Setup

The README provides the following initial firmware command. It installs the documented NimBLE-Arduino version, then the script builds or flashes the board. The optional serial port and debug arguments are passed to `hardware/flash.sh`.

arduino-cli lib install "[email protected]"
hardware/flash.sh            # optional: [serial port] [--debug]

The app installation is a local Xcode project setup. Copy the example configuration, set `DEVELOPMENT_TEAM` and `IA_BUNDLE_ID`, and open the project:

cp ios/Config/Local.xcconfig.example ios/Config/Local.xcconfig   # set DEVELOPMENT_TEAM and IA_BUNDLE_ID
open ios/IPhoneAgent.xcodeproj

The README does not provide a CLI command for deploying the iOS app to the phone. Xcode handles that build and installation step. After the app is installed, open it, plug in the ESP32-S3, complete Bluetooth pairing, join the board's hotspot, select `ESP32 Agent` from Screen Mirroring, enable AssistiveTouch, calibrate the pointer, and enter the OpenAI-compatible endpoint and key. These steps are prerequisites for entering the first task, such as `Open Settings and go to Wi-Fi`.

Editorial conclusion

Adopt ESP32 iPhone Agent if you want to prototype computer-use tasks on a physical iPhone and can flash the board and build the app. Do not adopt it as a production automation platform without first verifying iOS signing, model latency, network requirements, AirPlay stability, and the reliability of pointer calibration.

Frequently asked questions

Official sources

1. Issues
2. License: MIT
3. onehumanbeing/esp32-iphone-agent on GitHub
4. README
5. Releases