# PhoneAgent: RPC Bridge for AI-Controlled iOS and Android Automation

> PhoneAgent is an experimental open-source project that lets AI coding agents like Codex and OpenClaw control iOS and Android devices through a local JSON-RPC bridge. It also includes a standalone SwiftUI iPhone app that runs an agent loop using the OpenAI Responses API directly on the device.

**rounak/PhoneAgent** — An AI agent that can get things done across iPhone apps.

- Repository: https://github.com/rounak/PhoneAgent
- Stars: 791 · Forks: 93
- Language: Swift
- License: MIT
- Published: 2026-09-10 · Updated: 2026-09-10 · Language: en
- Canonical page: https://hysenlabs.com/projects/rounak-phoneagent

## What PhoneAgent Does and Who It Is For

PhoneAgent addresses a gap between AI coding agents and mobile device control. Agents like Codex and OpenClaw can edit code and run terminal commands, but they have no native way to interact with a running iPhone or Android application. PhoneAgent bridges this by running a local JSON-RPC server on the device or simulator, which the agent calls over localhost to tap, scroll, enter text, open apps, and capture screenshots.

The project has two operating modes. The first is an in-app iPhone agent: a SwiftUI application running on a physical iPhone or simulator that accepts natural-language prompts from the user and uses the OpenAI Responses API to drive the device. The second is an external bridge mode, where an AI coding agent running on a Mac calls the RPC server to automate a connected or simulated iOS or Android device.

The README describes the project as experimental personal-project software. It is not a production-grade automation framework.

## The RPC Action Surface for iOS and Android

Both iOS and Android bridges expose a shared set of RPC actions over a newline-delimited JSON-RPC transport on `127.0.0.1:45678` by default. The shared actions are:

- `get_tree`: returns the accessibility tree of the current screen
- `get_screen_image`: captures a screenshot
- `get_context`: returns combined context (tree plus screenshot)
- `set_api_key`: sets the OpenAI API key on device
- `open_app`: opens an app by bundle identifier (iOS) or package name (Android)
- `tap`, `tap_element`, `enter_text`, `scroll`, `swipe`: interaction methods
- `stop`: stops the current operation

The iOS bridge adds one method not available on Android: `submit_prompt`, which powers the in-app agent loop. An agent calls `submit_prompt` with a natural-language instruction and the in-app loop handles the multi-step execution.

The `tap_element` and `enter_text` methods use coordinate rectangles in the format `{{x, y}, {w, h}}` as taken from the accessibility tree. The `open_app` method takes a bundle identifier on iOS (for example `com.apple.Preferences`) or a package name on Android (for example `com.android.settings`).

## Setting Up the iOS and Android Bridges

The iOS bridge requires macOS with Xcode installed. Open the project at `PhoneAgent.xcodeproj` in Xcode, run the `PhoneAgent` scheme on a simulator or physical iPhone, and use the included launch script to start the RPC server.

For a physical device, localhost forwarding is needed. The iOS bridge launcher script handles this automatically. The scripts directory is at `.agents/skills/phoneagent/scripts/`.

For the Android bridge, start with verifying the device connection:

```bash
adb devices -l
```

For wireless Android without USB, pair and connect first:

```bash
adb pair <PHONE_IP:PAIRING_PORT>
adb connect <PHONE_IP:ADB_PORT>
```

Then start the Android RPC bridge with the network serial:

```bash
./.agents/skills/phoneagent/scripts/start_android_rpc_bridge_local.sh --serial <PHONE_IP:ADB_PORT>
```

The `rpc.py` CLI in the scripts directory supports both platforms and accepts `--host` and `--port` for non-default endpoint settings:

```bash
./.agents/skills/phoneagent/scripts/rpc.py get-tree
./.agents/skills/phoneagent/scripts/rpc.py get-screen-image --print-metadata
```

## The In-App iPhone Agent: Microphone, Wake Word, and Keychain

When running as a standalone iPhone app, PhoneAgent connects to the OpenAI API using a key stored in iOS Keychain. The app accepts prompts via the keyboard or through the microphone. An optional always-on mode listens for a custom wake word, allowing hands-free triggering.

After the agent completes a task, it sends an iOS notification. A quick-reply follow-up loop lets the user respond to the notification to continue the task without reopening the app.

The core source files for this mode are `PhoneAgent/PhoneAgentApp.swift` as the app entry point, `PhoneAgent/ContentView.swift` for the main UI, `PhoneAgent/PromptView.swift` for prompt submission, and `PhoneAgent/SettingsView.swift` for API key configuration.

Because the agent loop sends app contents to the OpenAI API when executing tasks, the README includes a disclaimer that model actions can be incorrect and that important operations should be verified before executing them.

## Known Limitations and Security Model

The README documents four known limitations. First, the Android bridge does not yet implement the `submit_prompt` agent loop, which limits Android automation to discrete RPC calls rather than autonomous multi-step task execution. Second, UI tree snapshots from the accessibility system can be noisy or stale during animations. Third, keyboard and text input reliability varies by app and platform. Fourth, long-running tasks require explicit polling or retry logic in the calling agent.

The security model is localhost-oriented by design. The RPC server binds to `127.0.0.1:45678`, which prevents external network access. iOS physical-device workflows route through localhost forwarding, and the Android bridge executes only through a selected `adb` serial. This scoping limits the attack surface but means the bridge is not accessible from other machines on the network without additional tunneling.

The app contents sent to the OpenAI API during the agent loop represent a privacy consideration: any text visible on screen during a task is potentially sent to OpenAI's servers.

## Comparison with Appium and Commercial Mobile Automation Tools

Appium is the dominant open-source mobile automation framework. It exposes a WebDriver-compatible HTTP interface and supports both iOS and Android through platform-specific drivers. Appium is designed for test automation: it integrates with test frameworks, produces test reports, and is maintained by a large open-source community.

PhoneAgent differs in two ways. First, it is designed for agent-driven automation rather than test scripts. The RPC surface is intended for an AI agent to call at runtime based on natural language instructions, not for hand-written test sequences. Second, the in-app iPhone agent mode has no equivalent in Appium: it runs as a first-class iOS application with microphone access, Keychain integration, and push notification support.

For teams needing reliable, CI-compatible mobile test automation, Appium is the more mature choice. PhoneAgent is experimental software aimed at developers exploring AI agent control of mobile apps, not at replacing established test infrastructure.

## Conclusion

PhoneAgent is worth evaluating if you want to automate iOS or Android apps with an AI agent and are comfortable working with experimental, personal-project software. It requires a macOS host with Xcode for the iOS bridge and Android SDK tools for the Android bridge. The Android bridge does not yet implement the submit_prompt agent loop, so autonomous Android automation is limited to explicit RPC calls. The project is MIT-licensed and last pushed on 2026-08-15. Confirm that the version of the OpenAI Responses API you target is still supported before building a workflow on the in-app agent loop.

## FAQ

### what is phone agent

PhoneAgent is an experimental open-source project that provides a local JSON-RPC bridge for AI coding agents to control iOS and Android devices. It also ships as a standalone SwiftUI iPhone app that runs an agent loop using the OpenAI Responses API directly on the device.

### Does PhoneAgent work with Android as well as iPhone?

Yes. The Android bridge uses adb and UiAutomator to expose the same shared RPC action surface as the iOS bridge. However, the Android bridge does not yet implement the submit_prompt autonomous agent loop, which is currently iOS-only.

### Does the PhoneAgent RPC server accept connections from the network?

No. The RPC server binds to 127.0.0.1:45678 by default, which limits it to localhost connections. Physical iPhone access uses localhost forwarding. The README describes the security model as localhost-oriented.

## Sources

- [Issues](https://github.com/rounak/PhoneAgent/issues)
- [License: MIT](https://github.com/rounak/PhoneAgent/blob/main/LICENSE)
- [README](https://github.com/rounak/PhoneAgent/blob/main/README.md)
- [rounak/PhoneAgent on GitHub](https://github.com/rounak/PhoneAgent)

---

Hysen Labs editorial analysis, written from the project's own repository and release notes. Cite the canonical page: https://hysenlabs.com/projects/rounak-phoneagent