Harness: AI-Driven User Testing for iOS Simulator, macOS and Web

The gap Harness fills between scripted UI tests and real user sessions

Scripted UI tests answer a narrow question: does this sequence of taps still produce this assertion? They cannot tell you that a first-time user gets stuck on the sign-up screen because the primary button reads like a secondary action. Harness targets that second question. You write a goal in plain language, for example "I want to sign up and create my first list" or "delete my account", attach a persona such as "first-time user, never seen this app", and an LLM agent reads screenshots, clicks, types and scrolls until it succeeds, fails, or would give up. The README frames this as "real-user simulation" rather than scripted UI testing, and the distinction matters: the agent is not replaying a recorded path, it is deciding what to do next from what it sees.

The audience is narrow and specific. This is a macOS 14+ developer tool written in Swift 6, distributed as a ~12 MB Universal build for Apple Silicon and Intel. It assumes you have Xcode tooling, because the iOS path runs xcodebuild and simctl, and it assumes you can grant Screen Recording and Accessibility permission on the host. If your team tests on Linux CI, nothing here applies to you. If your team is a small iOS or Mac shop where nobody has time to maintain a UI test suite, the trade is different: you write a sentence instead of a page of XCUITest code.

How the agent drives three different target kinds

Each Application declares its kind once, at create time, and that declaration reshapes the agent's tool schema and its system-prompt context block. Clicks, swipes, key shortcuts and navigation are not the same verbs on a simulator and in a web view, so the schema changes per platform. Run history, replay and friction reporting stay platform-neutral.

For iOS Simulator, Harness runs xcodebuild against your project and scheme, boots and installs with simctl, and uses WebDriverAgent for input. WebDriverAgent is vendored as a git submodule under vendor/, which is why the README opens with a note about the first clone. For a macOS app, Harness launches a pre-built .app or a macOS scheme through NSWorkspace, and the input story is the interesting one. Contained input is the default: accessibility actions (AXPress, AXSetValue) first, then CGEvent.postToPid to the app's own queue for scroll, shortcuts and raw clicks. The real pointer never moves, focus is never stolen, and no application other than the target is touched. There is deliberately no global-HID fallback, so a control that is unreachable through the accessibility tree fails honestly instead of being clicked by luck. Setting HARNESS_MACOS_INPUT=hid restores the legacy global-HID and foregrounding backend. Capture uses CGWindowListCreateImage, which grabs the window even when it is in the background.

For web, Harness embeds a WKWebView at a chosen viewport, 1280x1600 tall desktop by default or 375x812 mobile, synthesises JavaScript events for input, and captures with WKWebView.takeSnapshot. The mirror deliberately shows flat browser chrome with no device bezel so a snapshot covers more of the page, which the README ties to fewer scrolls per goal and lower API cost.

Targeting is Set-of-Mark across all three platforms: numbered overlays are drawn on interactive elements, the agent clicks by id rather than by pixel, and the overlays are agent-only and never written to disk. On web, the accessible name resolution follows screen-reader order (aria-label, then labelledby, then label, then placeholder, then title, then value, then text, then name), and each structured mark reports which rule won as label_source.

Installing Harness and running a first goal

The README points at a Universal download for macOS 14+ and also expects a source checkout, because WebDriverAgent is vendored as a submodule. A plain clone without submodules leaves the iOS input path incomplete, so initialise submodules as part of setup:

git clone https://github.com/awizemann/harness.git
cd harness
git submodule update --init --recursive

The repository also ships a project.yml, which indicates an XcodeGen-style project definition rather than a checked-in .xcodeproj at the root. The README does not spell out the build invocation for the app itself, so treat the downloaded release as the supported path and the source tree as the place you read and modify code.

Before the first run, macOS needs Screen Recording and Accessibility permission for Harness. Without both, capture and input will not work, and the macOS input backend will fail rather than fall back.

A first goal is written in the UI, not the shell: you create an Application, declare its kind, then give the agent a goal and a persona. The README's examples are short sentences, such as:

I want to sign up and create my first list

Pair that with a persona like "first-time user, never seen this app". After the run you get three artifacts: whether the goal completed (success, failure or blocked, plus a summary), the replayable path of screens and actions, and the timestamped friction events the agent flagged. For scripted or agent-driven use there are two other entry points: the harness-mcp binary, which is standalone and relocatable, and harness-cli for dev-time driving.

Authenticated flows and the credential handling trade-off

Web sessions are fresh-user by default, which makes SSO-only products unreachable. The 0.8.1 release adds two web-only parameters to start_ui_session. Setting visible: true brings the session window on screen so a human can complete password, SSO or MFA login. The session_state parameter takes cookies and localStorage and injects them before the first navigation. A companion tool, export_ui_session_state, returns the live session's state in exactly the shape session_state accepts, so a human-authenticated visible session can round-trip into later headless runs.

The credential handling is stated explicitly: cookie values are never logged, never written to steps.jsonl, never written to disk, and never stored on the persisted run model. That follows the same precedent as fill_credential. This is a sound default for a tool that reads screenshots of production-like environments, but it also means session state lives in memory and in whatever you pass across the tool boundary. If your threat model requires auditable storage of session material, this design gives you nothing to audit.

One more behaviour worth knowing: act_ui arms observation before dispatch and waits on the page's own pending work, setTimeout up to 2 seconds and in-flight fetch or XHR, so a same-URL React state swap lands in the action's own observation rather than the pre-action frame. Idle pages return at a 250 ms floor and the ceiling is 3 seconds. That is a reasonable compromise, and it is a compromise: a page that schedules work beyond those bounds will still be observed mid-flight.

Where Harness is the wrong tool

Web is WebKit-only. Chrome via CDP is listed on the roadmap, not in the product. If your users are on Chrome and your layout bugs are Blink-specific, Harness cannot see them, and no amount of goal phrasing fixes that. This is the single largest adoption blocker for web teams.

The macOS input model is honest but restrictive. Because there is no global-HID fallback, any control that is not reachable through the accessibility tree will fail rather than be clicked. That is the right default for a tool that runs on your machine while you work, but it means a custom-drawn canvas control with no accessibility exposure is untestable through the default backend. Switching to HARNESS_MACOS_INPUT=hid changes the guarantees: the real pointer moves and focus can be stolen, so runs are no longer contained to the target app.

The status line says v0.8.1 (alpha), while the release list shows v0.8.4 dated 2026-09-02. Version numbering in the README and the release tags do not line up, so pin to a tag rather than trusting the badge. The README also does not document rollback for a bad run, and it does not describe how to reproduce a run deterministically; the replay artifact shows the path, not a script that re-executes it.

Finally, cost. Every step is a screenshot plus a model call. The README treats the flat web chrome as a cost lever, which tells you cost is a real constraint. Cloud providers are billed per token; local inference through Ollama avoids that but requires the hardware.

Harness compared with XCUITest and Playwright

The closest alternative is the platform's own UI testing framework: XCUITest for iOS and macOS, Playwright or Selenium for web. The difference is not quality, it is what each one can assert. XCUITest executes a fixed sequence and checks fixed expectations, so it is deterministic, fast, free of model calls, and runs headless on CI. It will never tell you that a label is ambiguous, because you already encoded what you expected. Harness inverts that: it discovers the path, which is why it can surface friction, and also why it is non-deterministic and costs tokens per step.

For web specifically, Playwright plus an accessibility audit covers a lot of the same ground with none of the model cost, and it runs in Chrome. What it does not do is pursue a goal the way a confused user would, including giving up. That is the behaviour Harness is built to observe.

On the model side, Harness is not tied to one vendor. The README lists Anthropic Opus 4.7, Sonnet 4.6 and Haiku 4.5; OpenAI GPT-5 Mini and GPT-4.1 Nano; Google Gemini 2.5 Flash and Flash Lite; and local Mac inference through Ollama with Qwen3-VL 8B, Gemma 4 Vision 9B, Llama 3.2 Vision 11B, plus a custom-model field. Credentials are stored per provider in the Keychain, with per-Application credential storage as well. That breadth is unusual for a tool at this stage and it is the main reason to look past the alpha label.

Maintenance, licence and upgrade cost

The last push to main was on 2026-09-02, and the repository is not archived. Three releases landed within two days of that date (v0.8.4, v0.8.3, v0.8.2), which is a fast cadence for a single-maintainer tool. Fast cadence cuts both ways: fixes arrive quickly, and the surface you integrated against can move between minor versions. The README's own status line still says v0.8.1 while tags are at v0.8.4, which is exactly the kind of drift to expect.

Upgrade cost has two components. The app itself has Sparkle auto-update built in, so the binary is not your problem. The integration surface is: the MCP tools (start_ui_session, act_ui, export_ui_session_state, fill_credential) and their parameters, plus the outputSchema that now declares label_source. If you drive Harness from another agent, those schemas are your contract, and a release that changes a parameter name is a breaking change for you even if the app updates silently. The harness-mcp binary is standalone and relocatable, which helps: you can bundle and pin it rather than tracking the app.

Licence is MIT. That permits commercial use, modification and redistribution, and it requires the copyright notice and permission notice to be included in copies or substantial portions. It provides no warranty. Nothing here is legal advice; if you ship Harness inside another product, have counsel read the MIT text against your distribution model. One practical note: the vendored WebDriverAgent under vendor/ is a submodule with its own upstream, so your licence review should cover the submodule tree, not just the top-level LICENSE file.

Editorial conclusion

Adopt Harness if you already ship a macOS 14+ build machine and want goal-level feedback on an iOS Simulator, macOS or WebKit web target without writing scripted UI tests, and you are comfortable with an alpha (v0.8.1) release cadence. Do not adopt it if your web app must be tested in Chrome, if you need a headless Linux runner, or if you cannot grant Screen Recording and Accessibility permissions on the machine. Verify first that your target launches under the documented paths: xcodebuild plus simctl for iOS, an .app or macOS scheme for desktop, and an embedded WKWebView at the default 1280x1600 viewport for web.

Frequently asked questions

Official sources

1. awizemann/harness on GitHub
2. License: MIT
3. Project website
4. README
5. Releases