sim-use: an observe-act CLI that gives agents hands on iOS Simulator and Android

What sim-use is for, and who it is not for

The README frames the problem narrowly: an AI agent that writes mobile code has no way to see what it produced or interact with it. sim-use closes that gap by exposing a CLI that reads the accessibility tree of an iOS Simulator or Android emulator/device and emits a compact outline, then accepts taps against that outline. The intended user is someone building an agent loop, not a QA engineer writing a test suite. The project's own phrasing is "designed from day one for agent loops, not human testers." That distinction matters because the output format is optimized for token count, not for human readability. A screen becomes something like "@9 Button \"General\"" rather than a full JSON tree. If you want a general-purpose mobile test framework with assertions, reporters, and CI integrations, this is the wrong shape. If you want an LLM to see a screen, decide on one element, and tap it, the shape fits.

The observe-act loop and the alias cache

The core mechanism is a three-step cycle the README spells out as observe, act, verify. You run sim-use ui to read the screen, sim-use tap @9 to act, then sim-use ui again to confirm. The @N alias is the interesting part: it is cached from the last ui call, so the tap does not need to re-resolve the element tree. That is what makes the round trip fast, and the README claims roughly 300 ms per observe-act cycle after the first command, with a per-device background daemon amortizing initialization. I cannot verify that number, but the mechanism explains it: the daemon holds state, and the alias cache avoids a second tree walk. Selectors come in several forms for different trade-offs. @N is fastest but tied to the last ui. #<id> is stable across layout changes because it uses the accessibility identifier. --label matches on visible text and works with --wait-timeout for scripted flows. --point is the fallback when no accessibility data exists. The README notes that accessibility-derived selectors self-calibrate rotation on iOS, while explicit coordinates are interpreted in device-native portrait space. That is a real constraint: if you mix selector styles, you are mixing coordinate systems.

Cross-platform means two backends behind one verb set

sim-use routes calls by device ID shape. A UUID goes to iOS Simulator. emulator-5554, a serial like R5CT1ABCD12, or an IP:port goes to Android. A physical iPhone or iPad UDID is recognized separately and gets a restricted verb set. The README describes the iOS side as driving Apple's Accessibility APIs and the iOS Simulator HID pipeline, and the Android side as going through an AccessibilityService via a bridge APK that you install once with sim-use android init --device <serial>. So the cross-platform claim is real at the command surface, but the underlying stacks are entirely different. The README says the same verbs, flags, and --json shape apply to both, which is what lets you write one agent loop. It does not claim behavioral parity, and the physical-device section explicitly warns against assuming capability parity. For Android setup beyond the init command, the README defers to AGENTS.md for toolchain details.

Install paths and the XCFramework toolchain lock

The recommended install is Homebrew: brew tap lycorp-jp/tap then brew install lycorp-jp/tap/sim-use. On Homebrew 6.0.5 and later, an untrusted tap error requires brew trust lycorp-jp/tap first. Building from source is more involved. sim-use is a Swift package targeting macOS 14 or later, and it links against static XCFrameworks built from Meta's idb. Those frameworks are not checked into the repository because they are large, so you generate them locally with ./scripts/build.sh dev, which uses XcodeGen (installed via brew install xcodegen). Then make build, and the binary lands at .build/debug/sim-use. The README states a specific maintenance cost here: the XCFrameworks are built without library evolution, so their Swift modules are locked to the toolchain that produced them. Switching Xcode versions means re-running ./scripts/build.sh dev. That is a recurring cost, not a one-time setup step. The README also notes Xcode 27 beta support from Beta 4 onward, since earlier betas ship no usable SimulatorKit.framework, and that Xcode 27 no longer bundles Simulator.app, though the one from an Xcode 26.x install still works.

Where the accessibility tree runs out

The README makes a strong claim that sim-use walks the full tree including WebViews, system overlays, and embedded content, with no elements silently skipped. It also describes a specific failure mode and the recovery path. When the frontmost app exposes an empty tree because a remote process owns the visible UI, such as a system document picker, ui automatically retries with cross-process discovery and flags the recovered flat hierarchy via the advisory envelope key. That is a useful design detail because it tells you two things. First, the tool can hit a case where the obvious tree is empty. Second, the recovery produces a flat hierarchy, which is a different shape than the normal outline, and you need to handle the advisory key in your agent logic. The physical iOS device path is a harder boundary. The README states the channel exposes no element geometry, so coordinate taps, swipes, and gestures are unavailable, and the remaining verbs reject with the reason and the nearest alternative. That is a genuine limitation, not a temporary gap. For physical devices, ui and tap also require the foreground app to be development-signed with get-task-allow=true, though screenshot captures any screen.

What you give up compared with XCUITest or Appium

The obvious alternatives are Apple's XCUITest and Appium. XCUITest is Apple's own UI testing framework, written in Swift or Objective-C, and it runs inside the test process with direct access to the accessibility hierarchy and gesture APIs. Appium exposes a WebDriver-based interface across platforms and languages. sim-use differs from both in a specific way: it is a CLI that returns a text outline and accepts element aliases, and it is not a test runner. It has no assertions, no test organization, and no reporting. XCUITest gives you a full test harness and coordinate-level gestures on both simulators and physical devices, at the cost of writing and compiling test code. Appium gives you a language-agnostic WebDriver API and a large ecosystem, at the cost of a heavier server architecture. sim-use gives you a fast observe-act primitive with a token-efficient output format, at the cost of a narrower verb set and no test framework. If your goal is regression testing, XCUITest or Appium is the right category. If your goal is an agent that verifies its own work, sim-use is built for that loop.

Licence, maintenance, and what to check before adopting

sim-use is Apache-2.0, which permits commercial use, modification, and distribution with the usual conditions around notices and patent grants. The idb-derived XCFrameworks are built locally from Meta's idb, so their licence terms travel with that dependency, and the README does not restate them. Check the idb licence before redistributing a built binary. On maintenance: the repository shows releases v0.12.0 in late July 2026, v0.13.0 in early August, and v0.14.0 on 2026-08-27, with the last push on 2026-09-09. That cadence suggests active work, but a version below 1.0 means the command surface and JSON envelope can still change between releases, and the README already documents a capability matrix that differs between simulator, emulator, and physical device. Before adopting, verify three things. First, that your target app actually exposes a usable accessibility tree, since --point is the fallback and it is a weaker signal for an agent. Second, that your Xcode version matches the one used to build the XCFrameworks, or plan to re-run ./scripts/build.sh dev after each toolchain switch. Third, that the Android bridge installs cleanly on your device serial with sim-use android init --device <serial>, since that step is a prerequisite for any Android verb.

Editorial conclusion

Adopt sim-use if you are building an agent loop that needs to observe and act on a simulator or emulator, and you want one CLI rather than two platform-specific harnesses. Do not adopt it if you need coordinate-level gesture control on a physical iPhone, or if you cannot run a macOS 14+ build with the idb XCFrameworks. Verify first that your target app exposes a usable accessibility tree, that your Xcode version matches the XCFramework toolchain, and that the idb-derived static frameworks build on your machine.

Official sources

1. Issues
2. License: Apache-2.0
3. lycorp-jp/sim-use on GitHub
4. README
5. Releases