TipTour: macOS Menu Bar App for AI-Directed Desktop Control

Two Modes, One Menu Bar Button

TipTour sits in the macOS menu bar and activates one of two modes depending on user configuration. Gemini realtime mode is triggered by holding Ctrl+Option. The user speaks, Gemini processes the screen and audio together, and the result is one desktop action per turn: clicking, typing, opening apps or URLs, scrolling, editing highlighted text, or creating an Apple Notes entry. Pressing the shortcut again stops the session.

JEV text mode activates with Ctrl+K. The user types a task description, and JEV reads screen labels and control locations through local detection, selects the best target, and executes the action. JEV then evaluates whether the task is complete and loops until the goal is reached, a step fails, or the 12-action hard limit is hit. The modes are mutually exclusive: JEV is the default, and switching is done in Settings under Models, which shows only the current mode's API key field.

Keys stay in the macOS Keychain. There are no shared keys or hosted key proxies.

How JEV Differs from Gemini Mode in Practice

JEV and Gemini handle fundamentally different input types. JEV reads screen labels and control locations from local on-device detection. According to the README, it cannot see images or generate replacement text, so tasks that require interpreting visual content or composing new text belong to Gemini mode. JEV supports single, double, and right clicks; it acts on its top-ranked target without a confidence cutoff, meaning it always picks a target rather than asking for clarification. If no target is detected or JEV reports completion, the loop stops.

Gemini mode streams microphone audio and optionally screenshots to Google's servers. It supports a broader action vocabulary: the README lists clicking, typing, keyboard shortcuts, app and URL opening, scrolling, highlighted text edits, and Apple Notes creation. Holding Ctrl+Shift and painting over an area focuses Gemini on that region. Holding Ctrl+Option+Command opens a shortcut chooser for Speak, Type, or Highlight modes.

The auto-click setting controls whether TipTour executes actions autonomously. Point-only mode lets Gemini indicate targets for the user to click manually. JEV requires auto-click to function.

Building TipTour from Source

TipTour requires macOS 14.2 or later and a current Xcode with Swift support. The README gives these steps: open `tiptour-macos.xcodeproj`, select the TipTour scheme, set a signing team, and build and run through Xcode. Package dependencies resolve through Xcode's package manager automatically.

The README explicitly prohibits running `xcodebuild` from the terminal. The reason given is that terminal builds do not preserve the installed app's macOS permissions correctly, which breaks accessibility and screen recording access that the app needs to function.

For testing JEV logic without launching the full app, the repository provides `scripts/test-jev.sh`, which runs isolated JEV decision tests. This separation lets developers iterate on JEV's control selection logic without reinstalling or re-granting permissions. Source documentation is in `docs/source-layout.md` and `docs/tiptour-agent-contract.md`.

Privacy and Permission Model

TipTour requires several macOS permissions. Accessibility permission is needed to inspect and control applications. Screen Recording or Screen Content permission enables screenshots and local screen detection. The README clarifies this does not mean TipTour records video. Microphone access is required only for Gemini voice mode; JEV does not use the microphone.

In Gemini mode, microphone audio and (when enabled) screenshots are sent to Google's servers. In JEV mode, the user's typed task, locally detected labels and locations, and recent action history are processed, but screenshots stay on the device.

The README states there is no recording, video creation, or image-generation pipeline. There are also no shared keys, no hosted key proxies, and no keys loaded from external projects. Every API key is stored per user in the macOS Keychain and never leaves the device except as part of the API call to the relevant provider.

Limitations: JEV's Scope and Action Budget

JEV has three documented limitations. First, it cannot interpret images or screen content visually; it reads only text labels and control locations from the accessibility layer. Second, it cannot generate replacement text, so tasks like rewriting a paragraph or filling in a form with composed content require switching to Gemini mode. Third, JEV stops unconditionally at 12 actions per task regardless of whether the goal is complete. A task requiring more than 12 sequential UI steps will fail at the limit without recovery.

Gemini mode has its own constraint: it handles one desktop action per turn. Complex multi-step tasks require multiple turns, which means the user must re-invoke Ctrl+Option for each step or structure the request to complete within one turn.

A close alternative in this space is computer use via Anthropic's Claude API, which also provides desktop control capabilities. TipTour's distinguishing characteristic is that it is a self-contained macOS native app using a user's own API key, with local UI detection for JEV rather than full-screen vision inference for each action.

License, Maintenance, and Repository Layout

TipTour is licensed under MIT. The repository lists NOASSERTION in its license metadata field, but the LICENSE file is present and the README states MIT. The last push was on 2026-09-19. The repository has no GitHub releases; installation is by building from source.

The main branch integrates both modes. Feature branches are described as temporary PR work. The top-level `TipTour/` directory holds the application source; `TipTourTests/` and `TipTourUITests/` contain tests; `scripts/` holds the JEV test script. The `docs/` directory has source layout and agent contract documentation.

The project homepage is trybluey.com, though the repository description positions TipTour as an alternative to Google's AI-enabled pointer feature.

Editorial conclusion

TipTour suits macOS developers who want to experiment with agent-driven desktop automation using their own Gemini or JEV API key without a subscription service. JEV's 12-action limit per task and its inability to see images or generate replacement text make it unsuitable for long workflows or tasks that involve interpreting visual content. Before building, verify that Xcode can resolve the package dependencies and that the target machine meets the macOS 14.2 minimum. The repository prohibits terminal xcodebuild invocations to preserve installed app permissions, so the build must go through Xcode directly.

Frequently asked questions

Official sources

1. Issues
2. milind-soni/tiptour-macos on GitHub
3. Project website
4. README