PhoneAgent: An Experimental Bridge for Driving iPhone and Android Apps Over JSON-RPC

The gap PhoneAgent fills between a phone and a coding agent

Most desktop automation stacks assume a browser or a shell. A phone has neither in a form an external agent can drive. PhoneAgent's answer is to put a JSON-RPC server on the device side and let a host process send actions to it. The README describes two operating modes: an in-app iPhone agent (SwiftUI app plus XCTest runner plus the OpenAI Responses API) and an external bridge that lets Codex or OpenClaw control iOS and Android devices. The audience is narrow and specific: developers who already write UITest or adb automation and want a uniform action vocabulary instead of memorising two toolchains. It is not aimed at end users. The README labels it experimental software and a personal project, and there are no retrieved releases, so there is no versioned artifact to pin against. That matters more than it sounds: with no releases, you consume the main branch, and the API surface described below can move under you.

Two servers, one action vocabulary

The architecture is split by platform but unified by method names. On iOS, the RPC server is test-hosted: SimulatorRPCServer.swift and PhoneAgent.swift live under PhoneAgentUITests, which means the bridge runs inside an XCTest process rather than inside the shipped app. On Android, android_rpc_bridge.py is a localhost JSON-RPC bridge backed by adb commands. Both expose the same shared surface: get_tree, get_screen_image, get_context, set_api_key, open_app, tap, tap_element, enter_text, scroll, swipe, stop. One method is iOS-only: submit_prompt, which powers the in-app agent loop. Transport is newline-delimited JSON-RPC objects, and the default endpoint is 127.0.0.1:45678. The data flow for a typical action is: host script sends a JSON-RPC object to localhost, the platform bridge translates it into either an XCTest-hosted action or an adb/UiAutomator/input/screencap call, and the result returns as JSON. Because the iOS server lives in the test target, launching the bridge is closer to running a UI test than starting a daemon, and the helper script start_rpc_bridge_local.sh exists to make that less awkward, including localhost forwarding for physical devices.

Getting a first RPC call through

For the in-app agent, the README's Quick Start is four steps: open PhoneAgent.xcodeproj in Xcode, run the PhoneAgent scheme on an iPhone or simulator, enter the OpenAI API key when prompted, then submit tasks by keyboard or microphone. Note that the README's step one gives an absolute path from the author's machine, /Users/rounak/Developer/PhoneAgent-cli/PhoneAgent.xcodeproj, so you will need to substitute your own checkout location. For the external bridge, the CLI is the entry point. The README gives these examples: ./.agents/skills/phoneagent/scripts/rpc.py open-app com.apple.Preferences for iOS, ./.agents/skills/phoneagent/scripts/rpc.py open-app com.android.settings for Android, ./.agents/skills/phoneagent/scripts/rpc.py get-tree, and ./.agents/skills/phoneagent/scripts/rpc.py get-screen-image --print-metadata, which writes a PNG under /tmp/phoneagent-artifacts. The CLI accepts --host and --port for non-default endpoints. Two parameter details are easy to get wrong. First, open_app takes bundle_identifier, and the value differs by platform: a bundle identifier such as com.apple.Preferences on iOS, a package name such as com.android.settings on Android. Second, tap_element and enter_text use coordinate rectangles in the format {{x, y}, {w, h}}. For wireless Android, the README documents adb pair <PHONE_IP:PAIRING_PORT>, then adb connect <PHONE_IP:ADB_PORT>, then adb devices -l to verify, followed by start_android_rpc_bridge_local.sh --serial <PHONE_IP:ADB_PORT>. Host requirements are Xcode, Python 3, and Android SDK tools for the Android side.

Where the bridge breaks down

The Known Limitations section is unusually direct, and it should shape expectations. The Android bridge does not implement the submit_prompt agent loop, so the in-app agent experience is iOS-only. UI tree snapshots can be noisy or stale during animations, which is the failure mode that will bite hardest in practice: if get_tree returns a stale hierarchy, a subsequent tap_element computed from it may land on the wrong target. Keyboard and text reliability varies by app and platform, so enter_text is the least predictable method in the shared surface. Long-running tasks may need explicit polling and retries, which means the caller owns the retry loop. There is also a data-handling boundary worth stating plainly: the README's disclaimer says app contents may be sent to the OpenAI API when using the agent flow, and that model or tool actions can be incorrect, with a recommendation to verify important operations. The security model is localhost-oriented: the bridge binds around 127.0.0.1, iOS physical-device workflows use localhost forwarding, and the Android bridge executes only through the selected adb serial. That is a reasonable default, but it also means the bridge is not designed to be exposed on a network interface, and nothing in the material suggests authentication beyond localhost scoping.

PhoneAgent versus Appium and the platform test frameworks

The obvious comparison is Appium, which also drives iOS and Android from an external process and also exposes a client-server protocol. The difference in approach is where the automation runs and what it depends on. Appium relies on platform drivers (XCUITest under the hood on iOS, UiAutomator or Espresso on Android) and has a large, versioned ecosystem with element-locator strategies beyond raw rectangles. PhoneAgent instead runs its own JSON-RPC server inside an XCTest host on iOS and a Python adb wrapper on Android, with a small fixed method list. That smaller surface is the point: eleven shared methods plus submit_prompt is easy to reason about, and the CLI is a single Python script. The cost is that PhoneAgent gives you less than Appium in exactly the areas where automation gets hard. There is no described accessibility-identifier-first locator strategy; tap_element and enter_text take coordinate rectangles, so you are responsible for deriving coordinates from get_tree yourself. There is no described session management, no device farm integration, and no release artifacts. If your goal is a maintained automation framework with a support community, PhoneAgent is the wrong tool. If your goal is a small local bridge you can read end to end, the trade is defensible.

The in-app agent loop and its wake-word mode

The iOS app is not just a bridge host. It stores the OpenAI API key in the iOS Keychain, accepts prompts from the keyboard or the microphone, and offers an optional always-on mode with a custom wake word. Completion is signalled through a notification, and there is a quick-reply follow-up loop, so a task can continue through a notification rather than requiring the user to reopen the app. This is the part of the project with the least external precedent, and it is also the part that depends most on the OpenAI Responses API, which means the agent mode inherits whatever latency and cost profile that API has. The README does not describe a local model option, so if you cannot send app contents to OpenAI, the agent flow is not usable for you. The bridge mode, by contrast, makes no such requirement: it is a transport for actions you decide on yourself, and set_api_key is exposed as an RPC method, which implies the key can be supplied over the bridge rather than typed into the UI. The README does not state whether set_api_key persists to Keychain the way the in-app entry does, so treat that as unverified.

Maintenance cost, licence, and what to check before adopting

PhoneAgent is MIT licensed. That is permissive, and it means you can vendor the scripts into your own tooling without a copyleft obligation, but it also means there is no warranty and no support commitment, which the README's own disclaimer reinforces by calling the software experimental and a personal project. This is not legal advice; read the licence text yourself if you plan to redistribute. On maintenance, the practical cost is version coupling. The iOS bridge depends on Xcode and the XCTest runner, so an Xcode upgrade that changes UITest behaviour can break the bridge independently of anything in this repository. The Android bridge depends on adb and the UiAutomator surface, so Android platform updates and device-specific behaviour are your problem to absorb. The last push recorded is 2026-08-15, and no releases were retrieved, so there is no changelog to consult when something breaks. The concrete checks before you commit: confirm the PhoneAgent scheme runs on your Xcode version, confirm rpc.py get-tree returns a usable hierarchy for the app you care about, and confirm whether the methods you need exist on the platform you target, since submit_prompt does not exist on Android.

Editorial conclusion

Adopt PhoneAgent if you are already comfortable running Xcode UITest schemes or adb commands and you want a localhost RPC surface for scripted app control rather than a polished product. Do not adopt it if you need a supported SDK, an Android agent loop, or stable element identification during animations. Before wiring it into anything, verify three things: that the iPhoneAgent scheme builds and launches on your Xcode version, that rpc.py reaches 127.0.0.1:45678 on your chosen device, and that the Android bridge behaves acceptably without submit_prompt, since the README states that method is iOS-only.

Official sources

1. Issues
2. License: MIT
3. README
4. rounak/PhoneAgent on GitHub