GuideLLM: SLO-Aware Load Testing for OpenAI-Compatible Inference Servers
Evaluate and Enhance Your LLM Deployments for Real-World Inference Needs
At a glance
- What is it?
- GuideLLM is a Python benchmarking platform from the vLLM project that simulates production traffic against OpenAI-compatible and vLLM-native endpoints and reports full TTFT, ITL and end-to-end latency distributions. Its value is in SLO-driven sweeps and exportable reports, not in giving you a single headline throughput number.
- Who is it for?
- Adopt GuideLLM if you operate an OpenAI-compatible or vLLM-native inference endpoint and need latency distributions and reproducible load sweeps rather than a single throughput figure. Do not adopt it if you need a hosted-cloud or Ollama backend, or if your workload is embeddings and reranking, since the documented backend and endpoint lists do not cover those.
- Can I use it commercially?
- Yes. Apache-2.0 is a permissive licence: you can use, modify and sell software built on it, as long as you keep its copyright and licence notices.
- Is it still maintained?
- Yes. The repository received new commits within the last day.
- What is it written in?
- Mainly Python, according to GitHub's language statistics.
Answers come from the project's GitHub data, last synced on September 15, 2026, and from our analysis. They are not legal advice.
DEEP OPEN-SOURCE ANALYSIS
The gap GuideLLM targets: endpoint benchmarking versus model benchmarking
Most load generators measure an HTTP endpoint. They tell you requests per second and a mean latency, and they stop there. For an LLM deployment that is close to useless, because the number that decides whether users stay is the shape of the latency distribution: time to first token, inter-token latency, and how those two behave as concurrency climbs. GuideLLM is built around that distinction. The README states plainly that many tools benchmark endpoints rather than models and miss the details that matter for LLMs, and that GuideLLM focuses exclusively on LLM-specific workloads, measuring TTFT, ITL, output distributions and dataset-driven variation. The audience is engineering and ML teams who have a server running and need to answer capacity questions: how many concurrent requests before TTFT breaches a threshold, what output-length distribution the server actually produces, and where the safe operating range ends. It is not a training evaluation harness and it does not score answer quality. It measures the serving path.
How load is generated: profiles, backends and the sweep mechanism
The architecture visible in the README separates three concerns. First, a data source: HuggingFace datasets, local files, synthetic generation, or a custom source. Second, an execution profile that decides how requests are issued over time. The documented profiles are synchronous, concurrent, throughput, constant, Poisson and sweep. Third, a backend and endpoint pair. The listed backend is OpenAI-compatible, with vLLM-native servers also supported, and the supported endpoints are /completions, /chat/completions, /audio/translation and /audio/transcription. Data modalities span text, image, audio and video, which is why the audio endpoints appear at all. The sweep profile is the interesting one for capacity work: the README describes reproducible sweeps intended to identify safe operating ranges, which means the tool varies load across a run rather than holding one rate. Execution itself is described as multiprocessing, threading and async, which is what allows high-rate load generation without the client becoming the bottleneck. That claim is worth treating carefully. The README asserts scheduling accuracy beyond simple scripts, but it gives no reference hardware for that claim, so the client side is the first thing to watch when you push rates up.
Reports and metrics: what you actually get out
GuideLLM captures what the README calls complete latency and token-level statistics for SLO-driven evaluation, including full distributions for TTFT, ITL and end-to-end behaviour. Full distributions matter more than they sound. A p50 TTFT hides the tail that users complain about, and a mean ITL hides stalls. Output types are console, json, csv and html. The csv and json outputs are the ones that fit into regression tracking and dashboards, which the README explicitly names as a goal, describing standardized, exportable reports so results stay consistent across teams and workflows. The html and console outputs are for reading a single run. One design consequence worth naming: because reports are standardized and exportable, comparing two servers or two configurations is a matter of running the same profile against each and diffing the outputs. That only works if the dataset and profile are held fixed, which is a discipline the tool supports but does not enforce.
Getting it running: install and the command surface
GuideLLM is distributed on PyPI as guidellm and requires Python 3.10 through 3.13, per the README badges. The documented entry points are a CLI and a Python API, so the same workload can be scripted or run interactively. The README does not print a full command line in the material available here, so the exact flag names are not something I can state without guessing. What can be said from the repository layout is that configuration is expressed as a combination of backend, endpoint, data source and profile, and that the CLI and API expose the same set. The practical path is to install from PyPI, point the backend at your OpenAI-compatible base URL, choose a profile (concurrent or sweep to start), choose a dataset, and select an output format. The homepage at vllm-project.github.io/guidellm and the docs directory in the repository are where the concrete flag names live. Anyone who needs the exact invocation should read those rather than trust a second-hand summary, including this one.
Where GuideLLM is the wrong tool
The comparison table in the README is the most useful honesty in the project, because it shows what GuideLLM does not do. Its backend column lists OpenAI-compatible and vLLM-native. It does not list Ollama or hosted cloud providers, both of which appear as backends for other tools in that same table. So if your inference sits behind a proprietary hosted API with its own authentication and rate-limit semantics, GuideLLM is not the fit. The endpoint column is the second boundary: /completions, /chat/completions and the two audio routes. Embeddings and reranking do not appear, even though another tool in the table lists them. A team whose production traffic is mostly embedding lookups will get nothing from this. There is also a subtler failure mode. Because GuideLLM generates load from the client machine, a saturated client produces latency numbers that describe the client, not the server. The README's multiprocessing and async design is aimed at that problem, but nothing in the material states a maximum sustainable request rate or the hardware needed to reach it. Treat client saturation as the first hypothesis when results look implausible.
inference-perf and the difference in reporting philosophy
The closest comparison in the README table is inference-perf from kubernetes-sigs. Both are high-performance, both speak OpenAI-compatible endpoints, and both cover /completions and /chat/completions. The profiles overlap heavily: concurrent, constant, Poisson and sweep appear for both. The split is in metrics and output. The table marks inference-perf as lacking full metrics, and its output types are json and png. GuideLLM lists full metrics and adds console, csv and html. That difference is not cosmetic. A png is a picture for a human; a csv is a row you can commit to a repository and diff against last week's run. If your goal is a chart to paste into a slide, inference-perf covers it. If your goal is regression tracking where a change in TTFT distribution triggers a review, the csv and json outputs plus the standardized report format are the reason to pick GuideLLM. The other axis is data: inference-perf lists text only, while GuideLLM lists text, image, audio and video. Multimodal evaluation is where the two stop being interchangeable.
Maintenance, release cadence and the Apache-2.0 terms
The release history shows v0.7.1 in early July 2026, v0.7.2 in late July, and v0.7.3 at the end of July, with the last push to main in September 2026. That is a project shipping point releases on a roughly monthly cadence, which cuts both ways. You get fixes quickly; you also get interface churn. A benchmarking tool that changes its profile names or report schema between minor versions will break any script that parses its output, so pin the version in your environment and re-validate your parsing when you move. The licence is Apache-2.0, which permits commercial use, modification and redistribution provided you keep the licence and notice files and state significant changes. It also includes an explicit patent grant. That is a permissive arrangement, but it is not legal advice and it is not a substitute for reading the LICENSE file in the repository, which is the authoritative text. There is no separate commercial tier or open-core restriction mentioned in the material, so nothing here suggests the benchmarking path is gated.
Who should adopt it, and what to check before you do
The fit is narrow and clear. You run an OpenAI-compatible or vLLM-native inference server, you care about TTFT and ITL distributions rather than a single throughput number, and you want results in a format you can store and compare. Multimodal traffic, particularly audio, is a differentiator here since the supported endpoint list includes the audio translation and transcription routes. The misfit is equally clear: hosted cloud APIs, Ollama, embeddings and reranking workloads, and anyone who wants a quality score rather than a serving measurement. Before you build a workflow on it, run one sweep against a server whose behaviour you already know and check that the reported TTFT distribution matches what your own instrumentation sees. If the two disagree, the client machine or the dataset is the likely cause, and that is cheaper to discover on day one than after you have wired the csv output into a dashboard.
Editorial conclusion
Adopt GuideLLM if you operate an OpenAI-compatible or vLLM-native inference endpoint and need latency distributions and reproducible load sweeps rather than a single throughput figure. Do not adopt it if you need a hosted-cloud or Ollama backend, or if your workload is embeddings and reranking, since the documented backend and endpoint lists do not cover those. Before committing, verify three things on your own deployment: that the /audio/translation and /audio/transcription endpoints you intend to test are actually exposed by your server, that the dataset loader you plan to use (HuggingFace, file, synthetic or custom) is available in the version you install, and that the report output format you need (console, json, csv or html) is produced by the profile you select.
Community notes