Ollama Grid Search: A Desktop Runner for Model and Prompt Sweeps

The gap it fills between a shell script and a full eval suite

Anyone running local models eventually writes the same loop: pick a prompt, pick three models, run it, read the outputs side by side. Doing that in a shell script works until you want to change a parameter, re-run one cell, or look at what you ran last week. Ollama Grid Search packages that loop as a desktop application. The README frames the goal as automating "the process of selecting the best models, prompts, or inference parameters for a given use-case", and the target user is someone with Ollama already installed and serving endpoints on localhost or a remote host. The project is explicit that its use of "grid search" is loose. Real grid search tunes training hyperparameters such as batch_size or learning_rate; here the grid is over inference-time choices, not training runs. That distinction matters when you compare it to tooling built around training loops.

How a sweep is executed

The mechanism described in the README is a nested iteration. You select a set of models, write a prompt, and define parameter combinations. The prompt is submitted once for each parameter value, for each selected model, producing a set of responses. Models are fetched automatically from the configured Ollama server, and the model list can be filtered by name. Two execution modes are offered: limited concurrency, which the README links to a blog post on React limited concurrency, or synchronous inference calls. The synchronous mode exists, in the README's words, "to prevent spamming servers", which is a real concern when a sweep multiplies models by parameter values by iterations. Each combination can be run multiple times, and the app can optionally emit inference parameters and response metadata: inference time, token counts and tokens per second. Individual inference calls can be refetched, which is the practical escape hatch when one call in a large sweep fails or returns something unrepresentative. A configurable inference timeout bounds how long a single call may take.

A/B testing, the prompt archive and the slash autocomplete

Two comparison workflows sit on top of the sweep engine. The first is A/B testing: select different models and compare results for the same prompt and parameter combination, or hold the model fixed and vary the prompt. The second is the prompt archive, a database of saved prompts with examples. Prompts can be inserted by typing "/" in an input, an autocomplete behaviour the README says was inspired by Open WebUI, and the project states an intention to make prompts compatible with Open WebUI. That compatibility is stated as a goal, not a shipped guarantee, so treat exported prompt formats as something to check rather than assume. Experiments are listed inside the app, can be inspected in readable views, downloaded as JSON, and re-run by cloning or modifying the parameters used previously. Cloning a past experiment is the feature that turns the tool from a one-off comparison into something you can iterate with, because it removes the need to re-enter a parameter set by hand.

Installation, Settings and the llmman endpoint

There is no package manager install path in the README. You download a build from the releases page: .dmg for macOS, plus Windows and Linux builds. On Apple Silicon the app uses ad-hoc code signing rather than a paid Apple Developer certificate, so the documented first-launch procedure is to right-click or Control-click the app, choose Open, and confirm in the security dialog; later launches work with a double-click. The README states the app is not notarized and that this is standard for open-source macOS software distributed outside the App Store. Configuration lives in a Settings screen. The key documented there is the Ollama Server URL. The README gives a concrete example for llmman, a local model runner that serves the Ollama API as a drop-in replacement: set the Ollama Server URL to http://localhost:17434, and models pulled with llmman are listed and selectable like any other. The same screen is where custom default parameters and system prompts are defined, and where the inference timeout is set.

What the tool does not do

Grading is listed under Future Features, alongside importing, exporting and sharing prompt lists and experiment files. That means the app produces responses, timings and token statistics, and stops there. There is no scoring, no rubric, no automatic winner. If your question is "which model is better" in any sense that requires a judgement criterion, the judgement is yours, made by reading the outputs in the inspection views. The same Future Features list implies that sharing prompt lists and experiment files between people is not yet a first-class flow, so a team wanting a shared evaluation corpus will be moving JSON around by hand. Prompt compatibility with Open WebUI is phrased as an aspiration. And the whole application assumes an Ollama-compatible endpoint; the llmman example works precisely because it mimics that API, which tells you the boundary is the API shape rather than the vendor.

Where a scripted harness or a hosted eval tool fits better

The honest alternative depends on what you are optimising for. If you need sweeps inside CI, with results checked into a repository and asserted against thresholds, a small script that calls the Ollama HTTP API and writes JSON will fit your pipeline better than a GUI application, because the app's outputs are files you download rather than a library you import. If you need grading, hosted model providers, or dataset-level metrics, an evaluation framework built around those concepts is the right shape, and this project's own Future Features list acknowledges the grading gap. The difference in approach is worth stating plainly: Ollama Grid Search is an interactive instrument for a person sitting in front of a screen, comparing a handful of outputs. A scripted harness is a batch instrument. They overlap on the sweep loop and diverge on everything after it.

Maintenance, releases and the MIT licence

The repository is not archived, and the release history shows v0.9.0 in December 2024, v0.9.1 in April 2025 and v0.9.2 in November 2025, with the last push to the default branch dated September 2026. The version numbers have stayed in the 0.9.x range across those releases, so the project presents itself as pre-1.0 software. That has a practical consequence for anyone cloning experiments: expect the shape of stored experiment JSON and the Settings keys to be able to change between minor versions, and keep your own copies of anything you need to reproduce later. The licence is MIT, which permits commercial and private use and modification provided the copyright notice and permission notice are preserved; the repository does not include a separate licensing note in the material reviewed here, so read the LICENSE file in the tree before redistributing a build. Nothing in the README describes a paid tier, telemetry or a hosted component, so the operational surface is the app plus your Ollama server.

Editorial conclusion

Adopt Ollama Grid Search if you already run Ollama and need to compare several models or parameter values against one prompt without writing a script, and you are willing to store experiments as JSON files on disk. Do not adopt it if you need automatic grading, hosted model providers, or CI integration: grading is listed as a future feature, and the app assumes an Ollama-compatible endpoint. Before relying on it, verify that your Ollama server is reachable at the URL you configure in Settings, confirm the inference timeout suits your largest model, and check how the limited concurrency setting behaves against your server.

Official sources

1. dezoito/ollama-grid-search on GitHub
2. Issues
3. License: MIT
4. README
5. Releases