What is aidatatools/ollama-benchmark?

llm-benchmark is a Python CLI that drives Ollama, picks models based on your RAM, and reports throughput. It is a quick way to compare local models on one machine, not a controlled benchmark harness.

Can I use aidatatools/ollama-benchmark commercially?

Yes. MIT is a permissive licence: you can use, modify and sell software built on it, as long as you keep its copyright and licence notices.

Is aidatatools/ollama-benchmark still maintained?

Yes. The repository last received commits 8 days ago.

What programming language is aidatatools/ollama-benchmark written in?

Mainly Python, according to GitHub's language statistics.

llm-benchmark: Ollama tokens per second CLI

At a glance

What is it?: llm-benchmark is a Python CLI that drives Ollama, picks models based on your RAM, and reports throughput. It is a quick way to compare local models on one machine, not a controlled benchmark harness.
Who is it for?: Adopt llm-benchmark if you already run Ollama and want a repeatable tokens-per-second number for the models you actually use, especially across several machines with different RAM. Skip it if you need controlled, statistically rigorous comparisons, since the tool auto-selects models by RAM and the README does not document warm-up handling or run-to-run variance.
Can I use it commercially?: Yes. MIT is a permissive licence: you can use, modify and sell software built on it, as long as you keep its copyright and licence notices.
Is it still maintained?: Yes. The repository last received commits 8 days ago.
What is it written in?: Mainly Python, according to GitHub's language statistics.

Answers come from the project's GitHub data, last synced on September 15, 2026, and from our analysis. They are not legal advice.

DEEP OPEN-SOURCE ANALYSIS

Hysen Labs Editorial · September 16, 2026 · aidatatools/ollama-benchmark

What llm-benchmark measures and who it is for

Running a local model through Ollama tells you it works. It does not tell you how fast it generates tokens, and that number decides whether a 7B model is usable for interactive chat or only for batch jobs. llm-benchmark exists to produce that number. The README describes it as a cross-platform CLI that reports "the tokens-per-second truth" for local LLMs served by Ollama.

The audience is narrow and practical: people who already have Ollama installed and want to compare models on their own hardware. The README states that a working Ollama installation is a prerequisite, so the tool assumes the daemon and its model store are already in place. It is not a hosted leaderboard and it does not benchmark remote APIs. The homepage at llm.aidatatools.com is listed in the repository metadata, but the README itself points users to the CLI.

One design decision stands out. Rather than asking you to name models, the tool inspects system RAM and chooses a model set that fits. That makes the first run trivial and makes results comparable across machines of similar memory, but it also means two people running the default command on different laptops are not benchmarking the same models.

How model selection and result reporting work

The README lays out four RAM tiers. Between 4GB and 7GB, the tool checks for gemma:2b and pulls it implicitly. Between 7GB and 15GB, it works with phi3:3.8b, gemma2:9b, mistral:7b, llama3.1:8b, deepseek-r1:8b and llava:7b. Between 15GB and 31GB, the set becomes gemma2:9b, mistral:7b, phi4:14b, deepseek-r1:8b, deepseek-r1:14b, llava:7b and llava:13b. Above 31GB, it uses phi4:14b, deepseek-r1:14b and gpt-oss:20b. The word "implicitly" matters: the tool will call ollama pull for missing models, so the first run downloads gigabytes before any measurement starts.

The second mechanism is telemetry. The default command is llm_benchmark run, and the README's example list shows that this sends system information and benchmark results to a remote server. A separate example, llm_benchmark run --no-sendinfo, does not. That is an explicit opt-out rather than an opt-in, and anyone benchmarking on a work machine should read that section before the first run.

The third mechanism is the custom benchmark file. A YAML file with a file_name, a version and a models list replaces the RAM-driven selection. The README's example uses version 2.0.custom and model entries such as deepseek-r1:1.5b and qwen:0.5b, with a note to keep model names in double quotes. The flag is --custombenchmark, pointing at the file path.

Installing llm-benchmark and running a first custom benchmark

Ollama must be installed and running first; the README lists that as the only installation prerequisite. The package supports Python 3.10 and above and is published as llm-benchmark on PyPI. The README gives three install routes: pip, pipx and uv. For a CLI you will run occasionally, pipx keeps it out of your project environments.

bash

pipx install llm-benchmark

After installation the console script is llm_benchmark, defined in pyproject.toml as llm_benchmark.main:app. Running the default command benchmarks the RAM-selected model set and, per the README, sends system information and results to a remote server.

bash

llm_benchmark run

To keep the run local, add the flag from the README's second example.

bash

llm_benchmark run --no-sendinfo

The more useful path for comparison work is a custom model list. Create a YAML file with the structure the README documents, then point the flag at it. The README warns that model names need double quotes.

yaml

file_name: "custombenchmarkmodels.yml"
version: 2.0.custom
models:
  - model: "deepseek-r1:1.5b"
  - model: "qwen:0.5b"

bash

llm_benchmark run --custombenchmark=path/to/custombenchmarkmodels.yml

If you built Ollama from source, the README provides a flag for the binary path: llm_benchmark run --ollamabin=~/code/ollama/ollama. There is also a hello subcommand shown in the developer section as llm_benchmark hello jason, used after poetry install.

Where the benchmark is weaker than it looks

The RAM tiers are the biggest limitation. Because the default run picks models by memory, a comparison between two machines is really a comparison between two different model sets unless you override it with --custombenchmark. The README does not document how many iterations each model gets, whether there is a warm-up pass, or how variance between runs is handled. Those omissions matter: a single cold run includes model load time, and a second run on the same model can look dramatically faster for reasons that have nothing to do with the hardware.

The telemetry default is a second caveat. Sending system information and results to a remote server is the documented behaviour of the plain run command, and the README does not describe what fields are transmitted or where they are stored. For a tool whose output is a machine profile, that is worth knowing before you run it on a company laptop.

Finally, the tool measures throughput, not quality. It tells you tokens per second; it does not tell you whether the answers were any good. A model that generates quickly and answers badly will still look excellent in the output. The README makes no accuracy claims, so treat the numbers as one axis only.

llm-benchmark compared with running Ollama's own timing output

The closest alternative is not another package; it is Ollama itself. The Ollama CLI and API report timing statistics for each generation, including token counts and durations, so you can compute tokens per second from a single prompt without installing anything else. The difference is scope. Ollama reports one request at a time, against whichever model you happened to load, and leaves the aggregation to you.

llm-benchmark adds three things on top: a model set chosen from detected RAM, an implicit pull step so the models exist before measurement, and a single command that produces a consolidated result. It also adds the custom YAML path, which is what makes it more than a convenience wrapper. If your goal is a quick sanity check on one model, Ollama's built-in timing is enough and avoids the extra dependency. If your goal is a repeatable sweep across a fixed list of models on several machines, the YAML route in llm-benchmark saves you from writing that loop yourself.

There is a middle option worth naming: writing your own script against the ollama Python package, which llm-benchmark itself depends on (ollama==0.6.1 in requirements.txt). That gives you control over warm-up and iteration count, which the README does not expose as flags.

Maintenance, licence and upgrade cost

The repository is MIT licensed, stated in both the LICENSE file and the pyproject.toml metadata. MIT is permissive: you can use, modify and redistribute the code, including in commercial settings, provided the copyright notice and licence text are kept. That is a statement about the licence text, not legal advice; if you plan to redistribute a modified version inside a product, have your own counsel read the file.

The project is not archived. The last push to main was on 2026-09-08, and the most recent release, v0.5.3, is dated the same day. The two releases before it, v0.5.2 and v0.5.1, landed on 2026-01-17 and 2025-12-29 respectively, so the cadence has been uneven: a burst of releases at the end of 2025, a gap through most of 2026, then v0.5.3.

Upgrade cost is low but not zero. The dependency pins are tight. setup.py pins typer==0.21.0, ollama==0.6.1, pyyaml==6.0.1, requests==2.33.0, psutil==5.9.8, GPUtil==1.4.0, lib-platform==1.2.10 and setuptools==83.0.0, while pyproject.toml uses caret ranges. The wide pin on setuptools is worth noting for anyone installing into a shared environment: setuptools==83.0.0 may conflict with an existing toolchain. Installing with pipx sidesteps that by giving the CLI its own environment.

Editorial conclusion

Adopt llm-benchmark if you already run Ollama and want a repeatable tokens-per-second number for the models you actually use, especially across several machines with different RAM. Skip it if you need controlled, statistically rigorous comparisons, since the tool auto-selects models by RAM and the README does not document warm-up handling or run-to-run variance. Before trusting a number, install it with pipx install llm-benchmark, run llm_benchmark run --custombenchmark=path/to/custombenchmarkmodels.yml against a fixed model list, and compare repeated runs on the same machine.

Frequently asked questions

How do I install llm-benchmark?

Install Ollama first, since the README lists a working Ollama installation as the only prerequisite. Then install the package with pipx install llm-benchmark, pip install llm-benchmark, or uv pip install llm-benchmark. Python 3.10 or newer is required.

How do I stop llm-benchmark from sending my results to a remote server?

Run llm_benchmark run --no-sendinfo. The README shows the plain llm_benchmark run command as sending system information and benchmark results to a remote server, and this flag as the way to avoid that.

Can I benchmark my own list of Ollama models instead of the RAM-selected ones?

Yes. Create a YAML file with file_name, version and a models list, keeping model names in double quotes, then run llm_benchmark run --custombenchmark=path/to/custombenchmarkmodels.yml. The README's example uses version 2.0.custom.

Which models does llm-benchmark download on the first run?

It depends on detected RAM. Between 4GB and 7GB it checks for gemma:2b; between 7GB and 15GB for phi3:3.8b, gemma2:9b, mistral:7b, llama3.1:8b, deepseek-r1:8b and llava:7b; between 15GB and 31GB for gemma2:9b, mistral:7b, phi4:14b, deepseek-r1:8b, deepseek-r1:14b, llava:7b and llava:13b; above 31GB for phi4:14b, deepseek-r1:14b and gpt-oss:20b. Missing models are pulled implicitly.

Official sources

Community notes