# sim-use: a CLI that gives AI agents eyes and hands on iOS Simulator and Android devices

> sim-use turns a simulator or emulator screen into a compact accessibility outline and lets an agent tap elements by alias. It is built for agent loops on macOS, and its Android and physical-device paths come with real constraints.

**lycorp-jp/sim-use** — Give your AI agent eyes and hands on iOS Simulator and Android emulator/devices.

- Repository: https://github.com/lycorp-jp/sim-use
- Stars: 1,384 · Forks: 94
- Language: Swift
- License: Apache-2.0
- Published: 2026-09-10 · Updated: 2026-09-10 · Language: en
- Canonical page: https://hysenlabs.com/projects/lycorp-jp-sim-use

## The gap sim-use targets: agents that build mobile UI but cannot check it

An agent that writes SwiftUI or Compose code can compile it. What it usually cannot do is look at the running app and confirm the button it added is actually there, enabled, and labelled correctly. The README frames this as the last gap in the agentic mobile development loop: plan, code, verify, ship. sim-use occupies the verify step.

The intended user is not a human QA engineer writing XCUITest cases. It is an LLM loop that needs a cheap, repeatable way to read a screen and act on it. The README says the tool was designed from day one for agent loops, not human testers, and the design choices follow from that: alias-cached taps, structured JSON envelopes, and a bundled skill file that teaches an AI client the command surface.

If you already have a mature Appium or XCUITest suite, sim-use is not replacing it. It is a narrower tool aimed at the moment an agent needs to confirm its own output before handing control back to a person.

## How the observe-act loop actually works

Every interaction follows the same three-step cycle the README demonstrates: read the screen, act on what you see, read again to verify. The observe step emits a text outline rather than a raw accessibility tree. Instead of nested JSON, you get a flat list of elements with aliases, grouped by vertical region (Top, Content, Bottom) with y-coordinate bounds. The README claims this outline is roughly 16x more compact than raw JSON, which matters when the consumer is paying per token.

The act step resolves a selector to a real touch. Four selector styles exist, and the trade-offs are explicit in the README: `@N` aliases are fastest because they are cached from the last `ui` call, `#<id>` selectors survive layout changes, `--label` works for scripted flows with `--wait-timeout`, and raw `-x -y` coordinates are described as a last resort when no accessibility data exists.

Underneath, sim-use drives three different backends. On iOS Simulator it uses Apple's Accessibility APIs and the Simulator HID pipeline. On Android it goes through an AccessibilityService, which is why the bridge APK must be installed once per device. The device ID shape decides which backend handles a call: a UUID means iOS Simulator, `emulator-5554` or a serial or an `ip:port` address means Android.

A per-device background daemon amortises initialisation. The README states that after the first command, each observe-act round trip completes in about 300 ms. That figure is the project's own claim, not an independent measurement.

One detection detail is worth noting because it affects reliability. When the frontmost app exposes an empty tree because a remote process owns the visible UI, such as a system document picker, `ui` retries with cross-process discovery and flags the recovered flat hierarchy through an `advisory` key in the envelope. A flat hierarchy means the parent-child structure is lost, so an agent reading it should not assume nesting.

## Installing sim-use and running a first tap

The README recommends Homebrew. The tap is separate from the formula, so both lines are needed.

```bash
brew tap lycorp-jp/tap
brew install lycorp-jp/tap/sim-use
```

On Homebrew 6.0.5 and later, the README notes that an untrusted tap error may appear, in which case `brew trust lycorp-jp/tap` must run first.

Building from source is heavier and macOS-only. sim-use is a Swift package targeting macOS 14+, and it links against static XCFrameworks built from Meta's idb. Those frameworks are not checked into the repository and must be produced locally by the build script, which needs XcodeGen.

```bash
brew install xcodegen
git clone https://github.com/lycorp-jp/sim-use.git
cd sim-use
./scripts/build.sh dev
make build
.build/debug/sim-use --help
```

The `./scripts/build.sh dev` step is first-time only, but it has a sharp edge: the XCFrameworks are built without library evolution, so their Swift modules are locked to the toolchain that produced them. Switching Xcode versions means re-running the script.

With a simulator booted and a single device active, the first real use is the loop itself. The README shows this exact sequence.

```bash
sim-use ui
sim-use tap @9
sim-use ui
```

The first command prints the outline, including lines like `@9 Button "General"`. The tap command prints a confirmation line such as `Tap at (201.0, 452.0) completed successfully`. The second `ui` call is how an agent confirms the screen changed.

For Android, one extra step is required before any of this works: `sim-use android init --device <serial>` installs the bridge APK. The README points to `AGENTS.md` for the Android toolchain setup.

If you want the bundled skill installed into an AI client so it knows the command surface, `sim-use init` auto-detects installed clients, and `sim-use init --client claude` runs non-interactively. `sim-use init --print` shows the content without installing anything.

## Where sim-use stops: physical devices and geometry

The clearest limitation is the physical iOS path, which the README labels experimental. It routes through the same top-level verbs, and `sim-use ui`, `sim-use tap '#<id>' / --label` and `sim-use screenshot` work against a plugged-in device's UDID. But the channel exposes no element geometry. That means coordinate taps, swipes and gestures are not available there, and the remaining verbs reject with a reason and the nearest alternative rather than silently failing.

The README states this plainly: never assume capability parity, and points to a capability matrix. This is the right call for a tool that would otherwise produce confusing partial failures in an agent loop, but it also means an agent that learned to tap by alias on a simulator may need a different strategy on hardware.

There is a signing constraint too. sim-use installs and signs no runner and needs no Developer Disk Image, which is a genuine simplification. In exchange, `ui` and `tap` require the foreground app to be development-signed with `get-task-allow=true`. Only `screenshot` captures any screen. An agent pointed at a release build of the same app will fail on inspection, not on tapping.

The macOS 14+ requirement is another boundary. The install path is Homebrew or a local Xcode build, so Linux CI runners cannot host sim-use itself. The Android bridge APK does not change that, since the CLI is still a macOS binary.

## How sim-use differs from Appium and XCUITest

Appium and XCUITest both drive iOS and Android, and both have far longer track records. The difference is who consumes the output.

XCUITest is a test framework. You write assertions in Swift, run them, and read a pass or fail. The screen representation is internal to the framework. An LLM that wants to decide what to do next has to be handed a test result, not a description of the screen.

Appium exposes a WebDriver endpoint, and a client can query the page source. That source is a full accessibility tree serialised as XML or JSON. It is complete, and it is large. Feeding it into a model context on every step is expensive, and the agent still has to locate elements by XPath or accessibility ID rather than by a short alias.

sim-use inverts the priority. The outline is lossy by design, grouped into regions with aliases, and the alias cache lets the next action be a two-character reference. The README's own framing is that the tool is AI-native rather than built for human testers. If your primary consumer is a human writing test cases, Appium or XCUITest will fit better, because sim-use deliberately does not offer the assertion vocabulary those tools provide.

## Maintenance, licence and the real upgrade cost

The repository is not archived, and the last push was on 2026-09-09, which is recent. Releases are frequent: v0.14.0 landed on 2026-08-27, v0.13.0 on 2026-08-06, and v0.12.0 on 2026-07-29. The version number is still 0.x, so the command surface and JSON envelope shape should be treated as movable.

The upgrade cost is dominated by the Xcode toolchain rather than by sim-use itself. Because the XCFrameworks are built without library evolution, their Swift modules are locked to the toolchain that produced them. Anyone building from source must re-run `./scripts/build.sh dev` after switching Xcode versions. Homebrew users avoid that, but they inherit whatever toolchain the published build used.

Xcode 27 support has its own caveats. The README states that Xcode 27 betas are fully supported from Beta 4 onward, because earlier betas ship no usable SimulatorKit.framework. Xcode 27 also no longer bundles Simulator.app; the README says the one from an Xcode 26.x install still works, as does Device Hub. That is a workaround, not a fix, and it is worth checking before upgrading a working machine.

The licence is Apache-2.0. That is a permissive licence with an explicit patent grant and a requirement to preserve notices. The repository also carries a NOTICE file and a THIRD_PARTY_LICENSES file, which matters here because sim-use links against XCFrameworks built from Meta's idb. If you redistribute a binary, read those files rather than assuming the Apache-2.0 text covers everything in the build. This is not legal advice; check with your own counsel if redistribution is part of your plan.

## Conclusion

Adopt sim-use if you are running an agent loop against booted iOS Simulators on macOS and want a stable observe-act-verify cycle without writing XCUITest scaffolding. Do not adopt it if you need physical iOS device parity, if your team is not on macOS 14+, or if your workflow depends on coordinate-level gesture fidelity on real hardware. Before committing, verify that the Android bridge APK installs on your target serial, that your Xcode version matches the one that built the XCFrameworks, and that the verbs you need appear in the physical iOS capability matrix.

## FAQ

### What is sim-use used for?

It gives an AI agent the ability to observe and act on iOS Simulator and Android emulator or device screens, so an agent can verify the mobile UI it just built. The README describes the loop as observe, act, verify, using `sim-use ui` and `sim-use tap @9`.

### How do I install sim-use?

The README recommends Homebrew via `brew tap lycorp-jp/tap` followed by `brew install lycorp-jp/tap/sim-use`. Building from source requires macOS 14+, XcodeGen, and running `./scripts/build.sh dev` once to produce the XCFrameworks.

### Does sim-use work on Android devices and emulators?

Yes. Android devices and emulators are driven through the same command surface as iOS, but the README says you must run `sim-use android init --device <serial>` once to install the bridge APK. The README points to AGENTS.md for the Android toolchain setup.

### Can sim-use control a physical iPhone or iPad?

The README describes physical iOS support as experimental. `sim-use ui`, `sim-use tap` with `#<id>` or `--label`, and `sim-use screenshot` work against a plugged-in device's UDID, but the channel exposes no element geometry, so coordinate taps, swipes and gestures are unavailable.

### What does sim-use require to inspect an app on a physical iOS device?

The README states that sim-use installs and signs no runner and needs no Developer Disk Image, but `ui` and `tap` require the foreground app to be development-signed with `get-task-allow=true`. Only `screenshot` captures any screen.

## Sources

- [Issues](https://github.com/lycorp-jp/sim-use/issues)
- [License: Apache-2.0](https://github.com/lycorp-jp/sim-use/blob/main/LICENSE)
- [lycorp-jp/sim-use on GitHub](https://github.com/lycorp-jp/sim-use)
- [README](https://github.com/lycorp-jp/sim-use/blob/main/README.md)
- [Releases](https://github.com/lycorp-jp/sim-use/releases)

---

Hysen Labs editorial analysis, written from the project's own repository and release notes. Cite the canonical page: https://hysenlabs.com/projects/lycorp-jp-sim-use