If you serve a model, three settings decide how many requests per second you get, and all three are set once and forgotten. Here is what each one does and when to change it.
Continuous batching
This is the big one, and it is the reason the inference servers exist at all. Without it, a server processes one request at a time and the GPU is idle for most of each request's duration, because a request is mostly waiting. With it, when a request finishes generating, the next one joins the running batch immediately.
The setting that controls the ceiling is the maximum number of sequences allowed in a batch, and the number that actually limits you is the GPU memory the KV cache needs. Which is the next section.
The KV cache budget
--gpu-memory-utilization is the fraction of the card the server may use, and the default is 0.9. The rest is the weights, and what is left is the KV cache — which is what determines how many requests can be in flight at once.
The other flag is --max-model-len, the context you promise to serve. It reserves KV space for it. A server configured for 32k reserves KV space for 32k tokens for every sequence it might run, so the effective concurrency is much lower than the flag suggests.
This is the setting people get wrong. Raising the context does not make the server better at serving; it makes the server serve fewer requests at once.
Quantisation
The weights in memory decide how many of them fit. At 8 bits, a 70B model needs about 70 GB; at 4 bits, about 35. That is the difference between a machine with two 80 GB cards and a machine with one.
The cost of quantisation is quality, and the cost of a bad choice is worse than the quality you lost. Running a quantised model and then serving a long context on it is where the arithmetic stops working: the KV cache is still in full precision by default, and it grows with the context.
The order to change them in
Memory first. If the model does not fit, nothing else matters. /tools/llm-vram-calculator does the sum.
Then the context. Set it to what your requests actually use, not to the model's maximum. This is free throughput.
Then the batch ceiling. Raise it until the KV cache is the constraint, and watch the latency as you do; there is a point where more concurrency makes every request slower.
Then quantisation. Last, because it is the one that costs quality, and because the first three are free.