whichllm: Benchmark-Aware Local LLM Selection for a Specific Machine
Find the local LLM that actually runs and performs best on your hardware. Ranked by real, recency-aware benchmarks, not parameter count. One command, run it instantly.
At a glance
- What is it?
- whichllm is a Python CLI that detects your GPU, CPU and RAM, pulls model data live from HuggingFace, and ranks candidates by merged benchmark scores rather than parameter count. The interesting part is the ranking logic; the interesting risk is how much of that ranking rests on estimated speed and VRAM.
- Who is it for?
- Adopt whichllm if you already run local models through llama.cpp, LM Studio or similar and want a ranked shortlist instead of a spreadsheet of parameter counts. Skip it if you need a guaranteed fit or an exact tokens-per-second figure before downloading tens of gigabytes, because the tool's own documentation labels its speed numbers as estimates.
- Can I use it commercially?
- Yes. MIT is a permissive licence: you can use, modify and sell software built on it, as long as you keep its copyright and licence notices.
- Is it still maintained?
- Yes. The repository last received commits 1 day ago.
- What is it written in?
- Mainly Python, according to GitHub's language statistics.
Answers come from the project's GitHub data, last synced on September 15, 2026, and from our analysis. They are not legal advice.
DEEP OPEN-SOURCE ANALYSIS
The gap whichllm targets: fit is solved, choice is not
Most local-model tooling answers one question: will this quantized file load? That question is largely mechanical. You add weight bytes, a KV cache, some runtime overhead, and compare against VRAM. whichllm's README states the premise directly: fitting a model into your VRAM is the easy part, and the hard part is knowing which of the models that fit is actually the best. The tool is aimed at people who already have a working inference stack and are tired of picking a model by parameter count. Its own worked example makes the case: on an RTX 4090 it ranks a 27B model first over a 32B model, and the README explains that the 32B fits the card fine but loses because the 27B scores higher on real benchmarks and is a newer generation. A size-only tool would have returned the 32B. That divergence between what fits and what is worth running is the product.
How the ranking is assembled, and what the confidence tags mean
The score is not computed from the model card. According to the README, whichllm merges real benchmarks including LiveBench, Artificial Analysis, Aider, multimodal and vision evaluations, Chatbot Arena ELO, and the Open LLM Leaderboard. Each resulting score carries an evidence tag: direct, variant, base, interpolated or self-reported. The tag then discounts the score by confidence. Two failure modes are explicitly rejected: fabricated uploader claims, and cross-family inheritance where a small fine-tune borrows the score of a much larger base model. There is also a recency rule. Stale leaderboard entries are demoted along each model's lineage so a 2024 model cannot outrank a current-generation one on an outdated number, and the benchmark snapshot date is printed under every ranking. That last detail is the most useful design decision here. It means a stale recommendation is visible rather than silently trusted, which is more than most ranking tools offer.
VRAM and speed are modeled, not measured
The README describes the fit math as weights plus GQA KV cache plus activation plus overhead. Speed is described as bandwidth-bound, with per-quant efficiency, per-backend factors, an active-versus-total parameter split for mixture-of-experts models, and separate modeling for unified memory versus discrete PCIe partial offload. The MoE split is the part worth understanding. In the README's sample output, a 30B MoE model at Q5_K_M is listed at 102 t/s, far above the dense 27B at 27 t/s, because speed is ranked on active parameters while quality is ranked on total parameters. That is a genuinely different ranking from a dense-only heuristic. The caveat is that all of this is estimation. The README marks confidence with tilde and question-mark symbols, and the default table shows estimated generation speed. Nothing in the supplied material indicates that whichllm benchmarks your machine. It predicts, then prints the prediction.
Running it: one command, then a set of flags that change the answer
The README's quick start is a single no-install invocation: uvx whichllm@latest. Installation for regular use is uv tool install whichllm, with uv tool upgrade whichllm to update. Homebrew and pip paths also exist: brew install andyyyy64/whichllm/whichllm and pip install whichllm. The default behavior is deliberately permissive. The README says whichllm is ambitious by default, ranking the best model that looks runnable including partial RAM offload and near-edge VRAM fits. The conservative invocation it recommends is uvx whichllm@latest --gpu-only --speed usable --vram-headroom 1GB, which restricts results to models that fit fully in VRAM, drops slow estimates, and reserves headroom for runtime overhead. If LM Studio still reports the model as slightly too large, the README suggests raising the headroom to 1.5GB. Hardware can be overridden with --gpu "RTX 4090", multi-GPU with --gpu "2x RTX 4090", and detected limits with --vram 8 --ram-bandwidth 68. Other documented flags include --speed fast, --markdown for pasteable output, --json for scripts, and subcommands upgrade, plan, run and snippet.
Where it will mislead you
The default mode is the main hazard. Because it includes partial offload and near-edge fits when they seem usable, the top of the default list can be a model that technically loads and runs slowly. The README's own framing acknowledges this: the conservative flag combination exists precisely because the default is ambitious. If you read only the first line of the default table and start a download, you may end up with a model that the tool's own speed coloring marks red, which the README defines as under 4 tokens per second. The second limitation is the evidence pipeline. Scores are merged from external leaderboards, so a model released recently enough to have no direct benchmark entry will be represented by a variant, base or interpolated score, discounted accordingly. A brand-new architecture with no comparable lineage may rank poorly for lack of evidence rather than lack of quality. The third is scope: the README describes auto-detection across NVIDIA, AMD, Intel, Apple Silicon and CPU-only, but the supplied material does not document what happens when detection fails or returns a wrong VRAM figure, beyond the manual override flags.
How it differs from llama.cpp's own tooling and from LM Studio
The closest comparison the README itself draws is LM Studio, but the relationship is complementary rather than competitive. LM Studio is a runtime with a GUI that tells you whether a specific model fits once you have chosen it. whichllm never loads a model to answer that question; it ranks candidates before you choose, and the README's troubleshooting path routes you back to LM Studio to confirm the fit. Against llama.cpp, the difference is sharper. llama.cpp ships llama-bench and a quantization toolchain, so you can measure tokens per second on your own hardware for a model you have already downloaded. whichllm inverts that order: it estimates speed across many models you have not downloaded, then ranks them. One measures one model precisely; the other estimates many models roughly. If you already know which model you want, llama-bench is the correct tool and whichllm adds nothing. If you are choosing among twenty, the estimate is the point.
Licence, releases and the cost of staying current
whichllm is MIT licensed, which permits commercial use, modification and redistribution provided the copyright notice and permission notice are retained. That is the permissive end of the spectrum, and it is a plain statement of the licence terms rather than legal advice; check the LICENSE file in the repository for the binding text. Upgrade cost is low by design. Because the tool is distributed through uvx, uv tool and pip, updating is a single command, and the README lists uv tool upgrade whichllm for existing installs. The data side is where maintenance actually lives. Rankings track live HuggingFace data with curated frozen fallbacks for offline or rate-limited use, so the ranking quality depends on the maintainer keeping the benchmark merge and the fallback snapshot current. The release cadence visible in the supplied material shows v0.5.14, v0.5.15 and v0.5.16 across roughly six weeks, with the most recent release on the same day as the last push. That is active, but it also means the benchmark snapshot date printed under each ranking is worth reading rather than assuming.
Editorial conclusion
Adopt whichllm if you already run local models through llama.cpp, LM Studio or similar and want a ranked shortlist instead of a spreadsheet of parameter counts. Skip it if you need a guaranteed fit or an exact tokens-per-second figure before downloading tens of gigabytes, because the tool's own documentation labels its speed numbers as estimates. Verify first by running the conservative invocation the README gives, uvx whichllm@latest --gpu-only --speed usable --vram-headroom 1GB, and then cross-check the top pick against the model's own card in LM Studio before committing to the download.
Community notes