# llama-benchy measures any OpenAI-compatible server the way llama-bench measures llama.cpp > llama-benchy is a Python benchmarking tool that brings llama-bench style context-depth measurements to any server speaking the OpenAI API, and its reason for existing is a list of specific flaws in the existing options. It separates prefill from cached prompt processing, reports three different latency figures, and can run a command between runs so you can clear the cache. **eugr/llama-benchy** — llama-benchy - llama-bench style benchmarking tool for all backends - Repository: https://github.com/eugr/llama-benchy - Stars: 763 · Forks: 77 - Language: Python - License: MIT - Published: 2026-10-21 · Updated: 2026-10-21 · Language: en - Canonical page: https://hysenlabs.com/projects/eugr-llama-benchy ## The motivation is a list of specific measurement flaws llama-benchy is a script that benchmarks OpenAI-compatible LLM endpoints, generating statistics similar to llama-bench. The README explains why that tool cannot simply be reused, and the reasoning is specific rather than a complaint about coverage. llama-bench is a CLI tool that is part of the llama.cpp inference engine, widely used in the LLM community to benchmark models, and it allows measurement at different context sizes. But it is available only for llama.cpp and cannot be used with other inference engines, like vllm or SGLang. It also performs measurements using the C++ engine directly, which is not representative of the end user experience, which can be quite different in practice. The second half of the motivation is a critique of vLLM's own benchmarking tool, which the README acknowledges is powerful and can be used with other inference engines, but has three problems. First, it is very tricky and even impossible to calculate prompt processing speeds at different context lengths. The suggested workaround, `vllm bench sweep serve`, only works well with vLLM with prefix caching disabled on the server, because even with random prompts it reuses the same prompt between multiple runs, which hits the cache in llama-server for instance, producing very low median TTFT times and very high prompt processing speeds. Second, the TTFT measurement it uses is not actually until the first usable token. It is until the very first data chunk from the server, which may not contain any generated tokens in /v1/chat/completions mode. Third, the random dataset is the only one that allows specifying an arbitrary number of tokens, but a randomly generated token sequence does not let you adequately measure speculative decoding or MTP. The closing note is dated: as of January 2nd, 2026, the author was not able to find any existing benchmarking tool that brings llama-bench style measurements at different context lengths to any OpenAI-compatible endpoint. ## Three latency numbers, because they answer different questions The metrics are where the tool earns its name, and reporting three of them rather than one is a deliberate choice. The tool reports Time To First Response (data chunk), which is TTFR, Estimated Prompt Processing Time, est_ppt, and End-to-End TTFT. The distinction between TTFR and end-to-end TTFT is the flaw the README identified in the vLLM tool, taken seriously. TTFR is when the first data chunk arrives from the server, which in chat completions mode may contain no generated tokens at all. End-to-end TTFT is when the first token you can actually use arrives. For a chat interface, only the second number corresponds to what a user perceives as responsiveness. The est_ppt figure exists because the prefill time is not directly observable over the API in the same way generation speed is. It is an estimate derived from the difference between the two measurements, and reporting it separately means you can see how much of the time to first token was spent processing the prompt. The output table shows all three columns, with the pp rows carrying ttfr, est_ppt and e2e_ttft, and the tg rows carrying the generation throughput and a peak t/s figure. A row labelled pp2048 @ d4096 means a 2048 token prompt measured at a context depth of 4096. The output is also versioned and timestamped, with a line like llama-benchy (0.2.2.dev1+g52d2b0d55.d20260206) followed by the date and the latency mode, so a result can be tied to the exact build that produced it. ## Depth sweeps separate cached prefill from fresh prefill The feature list describes the measurement matrix rather than a single number, and the depth dimension is the interesting one. The tool measures Prompt Processing and Token Generation speeds at different context depths. It can measure separate context prefill and prompt processing over existing cached context at different context depths, and the prompt length, generation length and context depth are all configurable through `--pp`, `--tg` and `--depth`. So the same 2048 token prompt gets measured at depths of 4096, 8192, 16384 and 32768 in the example output, and the point of that is the difference between a server that has to prefill a long context from scratch and one that can reuse cached work for the prefix. That distinction is the whole reason the tool exists. A prefix-caching server will look artificially fast at prompt processing if you measure it without controlling the cache, and the README's criticism of the sweep tool is precisely that it does not control it. llama-benchy attacks the problem with two mechanisms. The first is the `--depth` sweep itself, which lets you see the prefill cost growing with context while the prompt stays fixed. The second is the option to support a command after each run, for example to clear the cache. That turns cache state from an uncontrolled variable into something the test harness manages, which is the only way to get a measurement you can trust across servers with different caching behaviour. The tool also runs a coherence test after warmup to verify the model responds correctly, which can be skipped with `--skip-coherence`, and it uses HuggingFace tokenizers for accurate token counts rather than estimating from characters. ## Installing without installing, four ways Using uv is recommended, and the four options cover everything from a one-off run to a system-wide install. Option 1 runs without installation using uvx, taking the release version from PyPI: ```bash uvx llama-benchy --base-url --model ``` The same option runs the latest version from the main branch: ```bash uvx --from git+https://github.com/eugr/llama-benchy llama-benchy --base-url --model ``` Option 2 installs into a virtual environment, and the sequence is clone, create the environment, install: ```bash # Clone the repository git clone https://github.com/eugr/llama-benchy.git cd llama-benchy # Create virtual environment uv venv # Install with uv (installs into a virtual environment automatically) uv pip install -e . ``` You then activate the environment and run the tool: ```bash source .venv/bin/activate ``` ```bash llama-benchy --base-url --model ``` Option 3 runs without installing at all using uv run, which creates a virtual environment if it does not exist: ```bash uv run llama-benchy --base-url --model ``` Option 4 installs into the system path, either the release version from PyPI with `uv pip install -U llama-benchy` or the current main branch with `uv pip install git+https://github.com/eugr/llama-benchy --system`. One convenience removes the need to pass the model name at all. The tool auto-detects the HuggingFace model name from the endpoint's `/models` endpoint when `--model` is not specified. ## A real run, with the flags that matter The full invocation takes the base URL, the model, and the three size parameters: ```bash llama-benchy --base-url --model --pp --tg [OPTIONS] ``` And the worked example shows the depth sweep in action: ```bash llama-benchy \ --base-url http://spark:8888/v1 \ --model openai/gpt-oss-120b \ --depth 0 4096 8192 16384 32768 \ --latency-mode generation ``` The latency mode flag is the one to pay attention to, because the README recommends generation mode to get prompt processing speeds closer to real numbers, especially on shorter prompts. The default behaviour adapts the prompt size to something, so the measurement depends on how it is configured. Other options cover the reporting and the load. Multiple iterations with `--runs` report mean plus or minus standard deviation, which is why the table columns read like 8521.08 plus or minus 69.61. Concurrency is supported with `--concurrency` to measure throughput under load. Results can be saved to file in Markdown, JSON or CSV, and granular time-series data for token generation can be saved when JSON output is used, through `--save-total-throughput-timeseries` and `--save-all-throughput-timeseries`. The tool downloads a book from Project Gutenberg to use as source text for prompts, and the stated reason is to ensure better benchmarking of speculative decoding and MTP models. A real text sample has a realistic token distribution, which a random one does not, and the effect of that is visible in the decode path. The project is MIT licensed, Python, requires 3.10 or newer, and the release history is v0.4.0 with results streaming support from 2026-07-03, v0.3.8 with diffusion models support from 2026-06-10, and v0.3.7 bugfixes from 2026-04-27. The last push was on 2026-07-10. ## The dependencies explain what kind of tool this is The dependency list is short and it explains the design. The core requirements are openai for the API client, tokenizers and transformers at 5.2.0 or newer for the tokenization, pydantic at 2.12.5 or newer for the configuration models, and tabulate and numpy for the output formatting. requests, asyncio and aiohttp are there for the HTTP work, with the async stack needed for the concurrency option. The transformers dependency is heavy for a benchmarking script, and it is there for the HuggingFace tokenizers, which the feature list names as the way the tool gets accurate token counts. That matters because the entire measurement grid is expressed in tokens, so a character-based estimate would shift every prompt size and make results incomparable. The packaging is Hatchling with hatch-vcs, so the version comes from the git tags and the version file is generated at build time, along with a build hash from the short commit hash. The console script is `llama-benchy = "llama_benchy.__main__:main"`. The repository layout is a src/ layout with the package under src/llama_benchy, plus tests/, docs/, scripts/, schemas/ and an examples/ directory holding a benchmark_visualization notebook. The schemas directory suggests the output is validated against a schema rather than just formatted, which fits a tool whose output is meant to be machine-readable. The one current limitation is stated without hedging: it evaluates against the /v1/chat/completions endpoint only. Anything speaking only the older completions API is out of scope, and that matters for older server deployments. ## Conclusion Adopt llama-benchy if you are choosing between inference servers and need comparable numbers that reflect the server you actually run, rather than a measurement taken through a C++ engine that behaves differently in production. Skip it if your workload uses the completions endpoint rather than chat completions, since the tool evaluates only /v1/chat/completions, or if you need a stable long-term benchmark series, because the version is pre-1.0 and the tool is still changing. Verify first that your server exposes /v1/chat/completions and a /models endpoint, that you can clear the server cache between runs if you want an uncached measurement, and that you read the token counts from the tool rather than from a character estimate. ## FAQ ### What does llama-benchy measure that other benchmark tools do not? It measures prompt processing and token generation speeds at different context depths, and can measure context prefill separately from prompt processing over existing cached context. It reports TTFR, estimated prompt processing time and end-to-end TTFT, and it supports running a command after each run so you can clear the server cache. ### How do I run llama-benchy without installing it? Use uvx against the PyPI release with `uvx llama-benchy --base-url --model `, or against the main branch with `uvx --from git+https://github.com/eugr/llama-benchy llama-benchy`. You can also clone the repository and use `uv run llama-benchy` with the same arguments. ### Which endpoint does llama-benchy support? It evaluates against the /v1/chat/completions endpoint only, which is listed as the current limitation. The model name can be auto-detected from the endpoint's /models endpoint when --model is not given. ## Sources - [eugr/llama-benchy on GitHub](https://github.com/eugr/llama-benchy) - [Issues](https://github.com/eugr/llama-benchy/issues) - [License: MIT](https://github.com/eugr/llama-benchy/blob/main/LICENSE) - [README](https://github.com/eugr/llama-benchy/blob/main/README.md) - [Releases](https://github.com/eugr/llama-benchy/releases) --- Hysen Labs editorial analysis, written from the project's own repository and release notes. Cite the canonical page: https://hysenlabs.com/projects/eugr-llama-benchy