PhoneCLI compiles mobile app screens into YAML commands

The app map replaces the screenshot loop with a compiled command list

PhoneCLI starts from a criticism of how phone agents usually work. Nearly every GUI agent runs the same loop: take a screenshot, send it to a vision-language model, emit an action. The project argues that this loop is slow, costly and fragile, because every screenshot is an API call and models hallucinate coordinates, and that it forces the screen off the device, which is the opposite of what a phone is for. The insight underneath that critique is that most of what the loop does is navigation, and navigation is static, ordered, and repeated. So PhoneCLI does not try to make the model bigger. Offline, a BFS crawler walks an app and compiles its navigation into a YAML app map, in which every screen becomes one deterministic command. The runtime then replays those commands on the device, through WebDriverAgent on iOS and ADB on Android, in sub-second time and at zero VLM cost. A GUI mode remains for screens the map has never seen. That split is the whole design, and it is also where the limits sit.

Two execution modalities, and a fallback that quietly restores the VLM bill

The two modalities are not equal partners. The CLI path is a replay: a command from the app map, executed against a live device, with no model call at all. The GUI path is the familiar one, where a vision model looks at the screen and decides. The project states the fallback rule plainly, that a failed CLI path falls back to GUI, and draws the conclusion that compilation can only help. That is true in the sense that a working compiled command is cheaper and faster than a model call. It also means the cost saving is conditional on the app map matching reality. When a compiled path fails, the request does not get cheaper, it becomes the thing the harness was built to avoid. A rollout that works on one build of an app and breaks after an interface change will fall back on every affected step, and the harness gives no published figure for how often that happens in the wild. Any evaluation of a PhoneCLI deployment should therefore track fallback rate, not just success rate.

Three tiers: CLI first, on-device model second, cloud last

The request path is ordered, CLI to on-device model to cloud. The CLI tier absorbs navigation at zero model cost, so only the steps the app map cannot cover ever reach a model. The claim in the project documentation is that this cuts cloud calls by roughly 10 percent end to end, and that the 3B on-device model the team ships is the engine rather than the headline, since the CLI path needs no model at all and the fallback path can take any model, a general LLM or a GUI-tuned one. The other half of the story is memory. The documentation describes an efficient memory of 10 to 20 steps of context, and that budget is what lets one phone keep running a long task without the context window becoming the bottleneck. Read those two numbers together and the design becomes clearer: PhoneCLI is not primarily an accuracy system, it is a containment strategy that keeps as much of the work as possible off a paid cloud endpoint and inside a small local model.

AndroidLab numbers: 63.0 percent against a 50.7 percent pure-VLM baseline

The published evaluation runs on AndroidLab with 9 apps and 138 tasks, Qwen3.7-Plus as the cloud model, and one judge used throughout, which is the detail that matters most for reading the result. The headline is 63.0 percent task success against 50.7 percent for the pure-VLM loop, a gain of 12.3 points. Efficiency moves in the same direction: 7.52 steps down to 6.72, roughly 11 percent fewer, and 40.1k tokens down to 34.4k, roughly 14 percent fewer, on the same tasks. A compiled command executes with 0 model calls. Read the caveats before quoting the 12.3 points. The suite is 9 apps and 138 tasks, which is narrow, the comparison is against a specific baseline configuration, and a single judge scores both sides. The project also documents the evaluation in phonecli_android/README.md, including the app map built for each app, the macro agent, its pure-VLM baseline, and the judging procedure, so the setup can be inspected rather than taken on trust.

Getting the code: a dependency list, three READMEs and a model on Hugging Face

The project points at dedicated documentation for iOS real devices, covering setup, WebDriverAgent, app map building, a CLI reference and troubleshooting, in phonecli/README.md, and at phonecli_android/README.md for the AndroidLab evaluation setup. A paper on arXiv and a companion project page for OpenPhone are linked alongside a Hugging Face dataset and a Hugging Face model, and a Chinese README is included in the repository root. The dependency list is published in requirements.txt and starts like this:

backoff==2.2.1
InquirerPy==0.3.4
lxml==4.9.3
openai
jsonlines
numpy
protobuf

Those pins describe a broad stack: an HTTP client, a terminal prompt library, BeautifulSoup4, pandas, datasets, opencv-python, dashscope and zhipuai clients, flask, tiktoken, and vLLM-related tooling in the repository's vllm_script directory. Expect to run a mobile device or emulator, and expect the build phase to need the app in front of you.

Where PhoneCLI is the wrong tool

Three limits deserve plain statement. First, the offline build phase is not optional. A BFS crawler has to walk each app to produce the map, so the approach fits apps you can install and operate yourself, and fits poorly when the target is an arbitrary third-party app that changes without notice. Second, the evidence is Android only. The 9 apps and 138 tasks are from AndroidLab, and the iOS path depends on WebDriverAgent being installed and authorised on your devices, which is a different operational burden from driving Android over ADB. Third, the model story is not a finished product. The repository ships a 3B on-device model, and the table of contents lists sections on shipped artefacts, on why 3B, on training with SFT and reinforcement learning, and on inference speed, but the quality of that checkpoint relative to the cloud fallback cannot be judged from the README alone. There are also no GitHub releases, so there is no published version to pin against.

A young MIT-licensed project with its own papers

PhoneCLI is released under the MIT licence, and the homepage is an arXiv paper rather than a product site, arXiv 2609.35671, with the underlying OpenPhone work on arXiv 2510.22009 marked as an ACL 2026 Findings paper. The harness is described as based on AndroidLab, and the dataset and model are distributed through Hugging Face under the hkuds organisation, with a separate OpenPhone project page for the wider effort. The last push was on 2026-10-07. The repository layout matches the story: agent/, page_executor/, phonecli/, phonecli_android/, recorder/, evaluation/, model_training/, prepare_data/, templates/, configs/ and tools/ sit side by side, which tells you a reader will find the crawler, the executor, the training code and the benchmark harness in one tree. With the MIT licence there is little friction to using the code commercially, and the practical upgrade cost is reading the harness yourself before trusting it on a device fleet.

Editorial conclusion

PhoneCLI fits teams building phone agents for apps they can inspect ahead of time, especially on Android where the evaluation was run, and teams willing to build an app map per app. It does not fit products that need to drive arbitrary third-party apps with no preparation step, and it does not remove the need for a vision fallback on unseen screens. Before committing, read phonecli_android/README.md to see how the nine app AndroidLab suite is judged, confirm the WebDriverAgent setup on iOS is acceptable for your device fleet, and check the MIT licence against your distribution model.

Frequently asked questions

Official sources

1. HKUDS/PhoneCLI on GitHub
2. Issues
3. License: MIT
4. Project website
5. README