GuideLLM: SLO-Aware Benchmarking for OpenAI-Compatible LLM Endpoints

What GuideLLM measures that a plain load test does not

A generic HTTP load generator can tell you that an endpoint returned 200 responses at some request rate. It cannot tell you how long the first token took, how evenly the remaining tokens arrived, or where the server stopped keeping its promises. GuideLLM is built around those LLM-specific signals. The README describes it as an "SLO-aware Benchmarking and Evaluation Platform for Optimizing Real-World LLM Inference", and the feature list names full distributions for TTFT (time to first token), ITL (inter-token latency) and end-to-end behaviour rather than averages alone.

The intended audience is engineering and ML teams running inference servers they control. The README frames the output as helping teams understand "system behavior, resource needs, and operational limits" and plan capacity as systems change. That is a deployment-tuning job, not a research job. If your question is whether model A writes better answers than model B, this is the wrong instrument. If your question is how many concurrent requests your vLLM server absorbs before inter-token latency crosses a threshold, GuideLLM is aimed squarely at you.

Traffic profiles, datasets and the OpenAI-compatible surface

The mechanism is a client that simulates end-to-end interactions against a server, generating workload patterns and then reporting on what came back. The README lists execution profiles as Synchronous, Concurrent, Throughput, Constant, Poisson and Sweep. The sweep profile is the interesting one for capacity work: it varies load across a range so you can see where latency degrades instead of testing a single rate. Poisson arrival is the closest of the set to organic traffic, while Constant pins the request rate and makes runs easier to compare.

On the input side, GuideLLM accepts real and synthetic datasets, including multimodal inputs. The comparison table names HuggingFace, files, synthetic generators and custom sources, with text, image, audio and video modalities. On the output side it reports to console and exports json, csv and html. The endpoints it drives are /completions, /chat/completions, /audio/translation and /audio/transcription, and the backend column says OpenAI-compatible. The README also states that it simulates interactions with "OpenAI-compatible and vLLM-native servers", so the server side is not restricted to one implementation.

That combination is what separates it from a shell script wrapped around curl. The dataset decides prompt and output length distributions, the profile decides arrival timing, and the report decides what you can compare later. Change any one of the three and you have a different experiment.

Installing GuideLLM and running a first sweep

The package is published on PyPI as guidellm, Python 3.10 to 3.13, under Apache-2.0. The repository also ships a Containerfile and a Containerfile.vllm, which is the path the related searches for a Docker setup point at. A plain install is the shortest route to a first run:

pip install guidellm

After that, point it at a running OpenAI-compatible server. The README does not print a complete CLI invocation, so the exact flags have to come from the documentation under docs/ and from guidellm --help rather than from this article. What the README does establish is the shape of a run: choose a backend endpoint, choose a profile such as sweep or Poisson, choose a data source such as a HuggingFace dataset or a synthetic generator, and choose an output type from console, json, csv or html. A sweep is the profile to start with, because a single fixed rate tells you whether the server survived that rate and nothing about headroom.

If you prefer containers, the repository layout shows Containerfile and Containerfile.vllm at the top level, so an image can be built from the checkout rather than pulled from a registry. The homepage at vllm-project.github.io/guidellm is the documentation entry point; the README itself is mostly positioning and a comparison table, and it does not walk through a full command.

Where GuideLLM stops being the right tool

The comparison table is honest about scope in one direction and silent in another. It lists Full Metrics as a GuideLLM column entry and marks several alternatives as lacking it, but every metric named in the README is a serving metric: TTFT, ITL, end-to-end latency, output distributions. There is no accuracy, no judge model, no scoring harness described. Teams that need answer quality alongside latency will have to pair it with something else.

The second limitation is the reporting surface. Output types are console, json, csv and html. The README says reports are "standardized, exportable" for dashboards, analysis and regression tracking, which means the tool hands you files and leaves the dashboard to you. Anyone expecting a live web UI will not find one described here. The related searches include a query about a GuideLLM UI; the documentation is the place to confirm whether one exists, because the README does not describe one.

Third, the numbers are only as meaningful as the dataset behind them. A synthetic generator with short prompts will produce a flattering saturation point that a production mix of long prompts and long outputs will not reproduce. The README presents the dataset choice as a first-class decision, and it is: the reported operating range describes the traffic you generated, not the traffic you will receive.

GuideLLM compared with inference-perf and genai-bench

The README's own table puts GuideLLM next to four tools, and the differences are concrete. inference-perf, from kubernetes-sigs, matches GuideLLM on CLI, high performance, OpenAI-compatible backends and the /completions and /chat/completions endpoints, and it supports Concurrent, Constant, Poisson and Sweep profiles. It differs on two axes: text is its only modality, and it exports json and png rather than json, csv and html. If you only serve text and you want charts straight out of the run, that trade is reasonable.

genai-bench, from sgl-project, covers more modalities than inference-perf (text, image, embedding, rerank) but the table marks it as neither high performance nor full metrics, with a single Concurrent profile and console, xlsx and png output. llm-perf, from ray-project, has no CLI and no API, text only, synthetic data only, Concurrent profile, json output. ollama-benchmark is narrower still: Ollama backends, /completions only, Synchronous profile. And vllm/benchmarks, inside the vLLM repository, offers no API, text only, and console-style output, though it does cover Synchronous, Throughput, Constant and Sweep profiles against both OpenAI-compatible and vLLM API backends.

The pattern is that GuideLLM is the widest of the set on profiles, modalities and output formats, at the cost of being a separate dependency with its own release cadence. Teams already deep in the vLLM repository may find the in-tree benchmarks sufficient for a quick check and reach for GuideLLM only when they need sweeps, multimodal data or exportable reports.

Maintenance, packaging and licence

The repository is not archived, and the last push was on 2026-09-10. Releases are frequent and dated: v0.7.1 on 2026-07-02, v0.7.2 on 2026-07-23, v0.7.3 on 2026-07-31. Versioning is derived at build time: setup.py reads git tags matching vX.Y.Z and computes the next version from a build type (release, candidate, nightly, alpha, dev), so installed versions carry suffixes like rc, a or dev depending on how the wheel was built. If you pin versions in CI, pin the released tags rather than whatever a nightly resolves to.

The dependency list in pyproject.toml is not small: click, culsans, datasets, faker, ftfy, httpx with HTTP/2, loguru, msgpack, numpy, protobuf and more, plus torch and torchcodec resolved from a CPU-only PyTorch index declared in the uv configuration. That is the real upgrade cost. A bump to GuideLLM can move datasets, numpy or httpx underneath you, and the uv sources override means torch comes from download.pytorch.org/whl/cpu rather than the default index. Reproducible runs are easier if you install into a virtual environment or use the provided Containerfile than if you install into a shared interpreter.

Licensing is Apache-2.0 per pyproject.toml and the LICENSE file at the repository root. That is permissive and generally compatible with commercial internal use, but this is a description of the declared licence, not legal advice. If you redistribute a modified GuideLLM or bundle it into a product, read the LICENSE text and your own policy.

Editorial conclusion

Adopt GuideLLM if you already run an OpenAI-compatible or vLLM-native endpoint and need latency distributions and saturation points rather than a single throughput number. Skip it if you want model quality scoring or a hosted dashboard, since it measures serving behaviour and exports static reports. Before trusting a run, verify that the dataset you point it at matches your production prompt and output lengths, because the reported limits only describe the workload you actually sent.

Frequently asked questions

Official sources

1. License: Apache-2.0
2. Project website
3. README
4. Releases
5. vllm-project/guidellm on GitHub