iPhone Use: USB iPhone Automation for Codex

Why iPhone Use targets a physical iPhone

iPhone Use addresses a specific gap between an AI assistant and a real iPhone. The project lets Codex, WorkBuddy, or Doubao Work open apps, read pages, tap controls, scroll, enter text, and collect lists through USB. It also presents a live phone screen in the host interface and returns screenshots when control-level element lookup fails. The intended users are developers working on a Mac with their own unlocked iPhone, Apple developer signing access, and an assistant capable of following a structured setup and interaction workflow.

This is not a general mobile device farm or a cross-platform automation suite. The documented environment is macOS with full Xcode, Python, Node.js, npm, and a USB-connected iPhone. The iPhone must trust the Mac, use developer mode when required, and remain unlocked during installation and startup. No jailbreak or separate Appium Server is required, but WebDriverAgentRunner still has to be signed, built, deployed, and kept running on the device.

The project is aimed at app exploration and assistant-driven operations on real hardware. That distinction matters because real-device behavior can differ from simulator behavior. The documentation also keeps sensitive or human-controlled steps outside automatic automation. Passwords, verification codes, Face ID, Apple login, device trust, developer mode, and unlocking must be handled by the device owner. This boundary means a workflow can pause until the user returns.

The WebDriverAgent path from MCP calls to iOS actions

The central mechanism is a local Python MCP service connected to WebDriverAgent, or WDA. WDA is an Appium-maintained service that runs on the iPhone and exposes device automation capabilities. iPhone Use forwards access through a loopback connection, while the phone-side runner handles app interaction. The project states that it uses a pinned, verified WDA 16.14.0 commit and manages acquisition, dependency installation, signing, building, deployment, and startup through its setup flow.

The service is not limited to fixed UI scripts. It can ask WDA for a compact control tree that includes text, element positions, and screenshot information. It can also request a screenshot directly without generating XML. When an element cannot be located reliably, the documented fallback is to inspect the returned image. Image pixels are multiplied by the returned `image.pixel_to_point` value to convert the visual coordinate into an iPhone point coordinate, which can then be supplied to `pua_tap`.

Actions include taps, swipes, drags, waits, Unicode text entry, and combined batches. The model should submit sensitive fields only after separately entering them because input and submission are separate operations by default. Long text is entered in segments, and continuation is tied to the original input field so that changing the app or focus stops continuation instead of writing into an unrelated control. For list work, the project provides bounded scrolling, overlapping reads, and deduplication rather than claiming that one snapshot contains an entire feed.

A live screen widget is separate from the automation evidence used for decisions. Users can watch the screen, click directly, drag with a mouse, or use a trackpad for two-finger scrolling and paging. The widget can show connection and pause states. When authentication requires takeover, it retains the last frame beneath a dark overlay and instructs the user to act on the phone. Model decisions remain tied to the actual image or control data returned by tools rather than to visual decoration in the preview.

Codex installation and first device connection

The Codex installation starts from source access. The README says the repository is currently private and requires GitHub access permission, although it also supports an already obtained source archive. Clone it with the documented command, enter the directory, and run the install script. The script validates and stages the source, registers a local marketplace through Codex CLI, installs the plugin and skills, and registers the cached service as the standard MCP server named `iphone_use`.

The following commands clone the project and run its installer. The installer may stop with a specific missing dependency or request for host-side action, so resolve those messages before opening a new Codex chat.

git clone https://github.com/zhongerxin/iPhone-use.git
cd iPhone-use
sh scripts/install.sh

After installation, reconnect or open a new Codex chat so the tools can load. In that chat, enable iPhone Use and ask the setup skill to diagnose the Mac, discover the USB device, reuse an existing WDA installation when healthy, and start WDA when needed. The documented readiness test is `pua_ready` returning `ready=true`. Do not begin app operations before that result is available.

The iPhone must remain connected and unlocked during setup. The assistant may ask the owner to sign in to Apple, trust the Mac, enable developer mode, or approve system prompts. Those are manual boundaries, not missing plugin commands. The project does not document a command that bypasses them.

Host installation and shared-device operating rules

Codex is not the only documented host. WorkBuddy and Doubao Work use their own installers, but they still need macOS, full Xcode, Python, Node.js, npm, and a USB iPhone. These adapters do not require Codex CLI, and the README states that their installed versions disable anonymous usage statistics by default.

For WorkBuddy, run this command from the source directory, then trust `iphone_use` in Connector Management and start a new session.

python3 workbuddy/install.py

For Doubao Work, run this command in the source directory. The installer output supplies the absolute command and arguments for a new STDIO connector. The `iphone-use` and `iphone-use-setup` skill folders are then uploaded through the host interface, and a new local computer task loads them.

python3 doubao/install.py

Codex, WorkBuddy, and Doubao Work can reuse the same local device configuration, WDA build, and operating lock. That reuse is useful when the same Mac manages one phone, but it also means concurrent hosts are not independent operators. If two sessions attempt to use the same iPhone, they should follow the shared lock's waiting instructions instead of sending competing taps or text input.

Screenshots, shared locks, and uncertain action outcomes

The screenshot fallback is useful because visual interfaces often contain canvas elements, icons without stable accessibility identifiers, or controls obscured by transient state. A screenshot gives the assistant a fresh observation when the control tree is incomplete. It does not guarantee correctness. Coordinate selection remains dependent on the returned image scale, current screen state, and whether another action changed the display between observation and tapping.

The project also introduces conservative failure semantics. Multiple processes share an operating lock. Setup requests are asynchronous and return work identifiers. The first connection should call `pua_setup(action="status")`, reuse a healthy WDA service or active work, and start a new setup only when necessary. A start request waits up to 20 seconds by default; after a timeout, the same job is queried rather than starting another build or launch process.

Timeout or disconnection after an action can leave its result uncertain. The documented response is to read the actual phone state instead of blindly replaying a tap, text entry, or submission. Replaying a non-idempotent action can duplicate a submission or enter text in the wrong place. This behavior is especially relevant for forms, messaging apps, payments, and list operations that lack clear completion markers.

Screen streaming is local and is not written to disk according to the README. Connection recovery reuses background startup work, rebuilds invalid sessions, and restores temporary snapshot settings. The refresh control can reconnect the preview. Locked-screen state and an intentional pause are presented separately, which allows a READY state after unlocking to restore the locked-screen preview.

Alternatives to iPhone Use and the cost of adopting it

The closest functional alternative is Appium's standard XCUITest workflow with WebDriverAgent. Appium provides the underlying device automation foundation, but iPhone Use adds a local MCP service, host plugins for Codex, WorkBuddy, and Doubao Work, setup and signing guidance, a live screen widget, screenshot fallback, and model-oriented compact page inspection. A team already operating Appium through its own test harness does not necessarily need iPhone Use merely to automate an iPhone. That approach adds an Appium Server and test code, while iPhone Use explicitly says no separate Appium Server is required. Neither route removes the need for WDA on the device.

The immediate adoption cost is primarily environment setup. Full Xcode must support the iPhone's current iOS version, and Command Line Tools are explicitly insufficient. Python 3.9 or later is required for the MCP service. Node.js must be 20.19 or later, 22.12 or later, or 24 or later, with npm 10 or later; the README says the project's engines and doctor checks are authoritative. Apple account login, signing, device trust, developer mode, and initial unlocks can require manual work.

The repository uses the MIT License. The repository also includes `THIRD_PARTY_NOTICES.md`, so adopters should account for bundled or integrated third-party components rather than treating the top-level MIT grant as the only applicable notice. This is a repository reading, not legal advice.

The repository is not archived, and the last push was on 2026-10-16. Releases v0.3.11, v0.3.12, and v0.3.13 were published on 2026-10-13 and 2026-10-15. The README does not document a formal compatibility policy for every WDA, Xcode, or iOS combination. It pins WDA 16.14.0 and directs users to project engine checks. Upgrade work may include reviewing `CHANGELOG.md`, rerunning diagnostics, rebuilding or replacing a cached plugin, and checking WDA signing and device state. The documentation does not promise zero-reinstallation upgrades.

Editorial conclusion

Adopt iPhone Use when the target is your own USB-connected iPhone, a full Xcode installation is acceptable, and the assistant must operate real apps rather than a simulator. Do not choose it for unattended Android testing, remote device fleets, or environments without Apple signing access. Verify macOS, full Xcode, supported Python and Node.js versions, USB trust, developer mode, and whether an existing WebDriverAgentRunner build can be reused before installation.

Frequently asked questions

Official sources

1. Issues
2. License: MIT
3. README
4. Releases
5. zhongerxin/iPhone-use on GitHub