TipTour for macOS: a voice-driven computer-use agent built on Gemini Live and CUA

What TipTour actually solves for macOS users

Most desktop automation is written ahead of time. You script a click at a coordinate, and the script breaks the moment a window moves. TipTour takes the opposite route: the instruction is spoken at the moment of use, and the target is resolved from what is on screen right now. The README describes it as a menu bar companion that "understands your screen, listens to your voice, and controls your computer for you."

The intended audience is narrow and technical. This is a source-build macOS app, not a download-and-run utility. It requires macOS 14 or later, Xcode 16 or later, and your own Gemini API key. The repository also ships a Cloudflare Worker directory, which the README calls optional for distribution builds. If you are not willing to open an Xcode project, set a signing team and grant system permissions, TipTour is not aimed at you.

The problem it targets is the gap between knowing what you want done and knowing which menu item, button or field does it. Freeform highlighting is the clearest expression of that idea: you paint over a region with Control+Shift and drag, then speak about "this" rather than naming the element.

Gemini Live, CUA Driver Core and Accessibility: the mechanism

TipTour is a coordinator, not a model. The README lists four pieces. Gemini Live handles realtime voice, screen understanding, transcription and tool calling. CUA Driver Core performs the actual computer control: clicks, typing, hotkeys, app launch, URLs, scrolling and browser coordinates. macOS Accessibility supplies native app structure so text and elements can be targeted exactly rather than by pixel. Focus Highlight carries the "this part" commands.

The data flow implied by that split is: microphone audio and screen frames go to Gemini Live, which returns a tool call; the tool call is executed by CUA Driver Core or through the Accessibility layer, depending on whether the target is a browser surface or a native control. The README states that TipTour "sees the app/window you are working in, understands the highlighted or hovered area, and keeps actions inside that context." That scoping is the design claim worth testing: an agent that acts on the wrong window is worse than no agent.

CUA is credited to the trycua/cua project, and the optional pixel cat sprites come from oneko. Three modes are documented. Autopilot performs actions and is on by default. Tour Guide teaches step by step and is off by default. Neko Mode is described as an optional playful cursor mode, also off by default. Tour Guide is the more interesting of the three, because a step-by-step walkthrough fails more gracefully than an agent that clicks the wrong button.

Building TipTour from source and running a first command

The README gives one command to open the project, then a numbered Xcode procedure. Node 20 or later is needed only if you are working on the Cloudflare Worker.

open tiptour-macos.xcodeproj

In Xcode, select the TipTour scheme, set your signing team, and press Cmd+R. The README adds a specific warning: do not build with terminal xcodebuild while you are actively testing permissions, because it can invalidate local TCC permission state. That is a real constraint, not a footnote. On macOS, TCC is the database that records which app holds which permission, and rebuilding through a different path can reset it.

After launch, paste your Gemini API key into the visible Gemini API key field in the menu bar panel. The README states the key is stored in macOS Keychain. Grant the four permissions the table lists: Microphone for voice input, Screen Recording for visual context, Accessibility for reading UI structure and controlling apps, and Screen Content for ScreenCaptureKit capture.

Once running, the controls are Ctrl+Option to toggle voice mode and Ctrl+Shift plus drag to paint a freeform focus highlight. A first use that exercises the whole chain is to hold Ctrl+Option, say "Open Apple Notes and write a short essay", then hold Ctrl+Shift and drag over a phrase, and say "Change this word." The first command tests app launch and typing. The second tests whether the highlight is being resolved to the right text.

Where TipTour breaks down or is the wrong tool

The permission model is the first failure mode. Four permissions, one of which is Accessibility, is a lot of trust to hand a menu bar process. If any one is missing or revoked, the corresponding capability stops working, and the README does not describe what the app does when a permission is absent. There is no documented error surface for that case.

The second limitation is dependency on a hosted model. Everything the app sees is routed to Gemini Live for screen understanding and tool calling. If you cannot send screen content to a third-party API, TipTour is the wrong tool regardless of how well it works. The README does not describe an offline or local-model path.

Third, the README documents no rollback, no undo, and no action history. Autopilot is on by default, which means the default posture is the app acting for you. For destructive operations in a document you care about, Tour Guide is the safer mode, and the README does not explain how to make Autopilot ask before acting.

Finally, the release situation is thin. No releases were retrieved, and the README describes source builds only. Anyone expecting a signed installer will be disappointed. The last push to the repository was on 2026-05-30.

How TipTour differs from other computer-use agents

The closest comparison in the README's own credits is CUA, the trycua/cua project, which TipTour uses for its computer control primitives. CUA is a set of primitives: clicks, typing, hotkeys, app launch, URLs, scrolling and browser coordinates. TipTour is the layer above it that adds voice input, screen understanding through Gemini Live, a macOS Accessibility path for native controls, and the highlight gesture. If you already have a scripted CUA pipeline, TipTour is not a replacement for it; it is a different front end with a spoken interface and a model in the loop.

The other comparison people search for is Google's AI pointer and similar pointer agents. The repository description frames TipTour as an open-source alternative to that category. The practical difference is control: you supply the Gemini API key, you build the app, and you can read the source and the AGENTS.md technical notes. The cost of that control is that you also own the build, the signing and the permission state.

If your automation is deterministic and repeatable, a scripted tool is a better fit than a model-driven agent. TipTour earns its place when the target changes between runs and describing it in words is easier than locating it in code.

Licence, packaging and the cost of keeping a build current

The README badge and the License section both point to MIT, and the repository contains a LICENSE file. The repository metadata reports the licence as NOASSERTION, which means the automated classifier did not confirm a standard licence identifier. Those two signals disagree, so read the LICENSE file itself before you rely on MIT terms. This is a note about what the files say, not legal advice.

Upgrade cost is dominated by the build method, not by the code. Because the README warns that terminal xcodebuild can invalidate TCC permission state, the recommended upgrade path is to pull the new source and rebuild through Xcode with the same signing team, then re-check the four permissions. Each rebuild is a manual step, and there is no documented auto-update mechanism. The appcast.xml file at the repository root suggests a Sparkle-style update feed exists in the project layout, but the README does not describe an update flow, so treat that as unconfirmed.

The Worker adds a second deployment surface if you want a distribution build. The README gives the sequence: cd worker, npm install, put the key as a Wrangler secret, then deploy.

cd worker
npm install
npx wrangler secret put GEMINI_API_KEY
npx wrangler deploy

After deploying, set TipTourWorkerBaseURL in the app bundle or build settings. Source builds do not use the maintainer's Worker URL.

Editorial conclusion

Adopt TipTour if you are a macOS developer comfortable building from Xcode, holding your own Gemini API key, and granting Microphone, Screen Recording, Accessibility and Screen Content permissions to a menu bar process. Do not adopt it if you need a signed, notarized installer, a documented upgrade path, or a tool that works without sending screen context to a hosted model. Before you commit, verify the licence text in LICENSE, confirm the Gemini API key is stored in Keychain after you paste it, and check whether the Accessibility permission survives a rebuild on your machine.

Frequently asked questions

Official sources

1. Issues
2. milind-soni/tiptour-macos on GitHub
3. Project website
4. README