Deploying a model service with vLLM

Last lesson ran a model on your own computer with Ollama, which is convenient for one person. If you want to turn a model into a service for hundreds of people at a company, or for your product's users, the problem is different: dozens of requests arrive at once, so how do you get through them as fast as possible on one or two GPUs?

vLLM is currently one of the most widely used open-source LLM serving frameworks. This lesson first uses an experiment to make the core server-side problem clear, then covers how vLLM solves it and how to use it.

vLLM is aimed mainly at Linux servers with NVIDIA GPUs, and my experiment machine (a Mac, with no NVIDIA GPU) can't run it. So this lesson's experiment uses Module 09's small GPT, and the vLLM usage section only lists commands from the official docs, without output I haven't produced myself.

One poem at a time, or many at once

Module 09, Lesson 6 explained that each generation step reads all of the model's parameters but computes only one character. On an LLM, that step's time is mainly spent reading parameters from GPU memory, and the compute units spend most of their time waiting for data.

So what if several requests are put together, the parameters are read once, and one character is computed for each request at the same time?

for batch in [1, 4, 16, 64]:
    for _ in range(N_POEMS // batch):
        model.generate(torch.full((batch, 1), NEWLINE), N_TOKENS)  # batch 首诗同时写

python batch_throughput.py

一共要写 64 首，每首 60 个字，用 KV 缓存
  一次写  1 首：总共  1.84 秒，每秒   2085 个字，每一首从开始到写完 0.03 秒
  一次写  4 首：总共  0.93 秒，每秒   4110 个字，每一首从开始到写完 0.06 秒
  一次写 16 首：总共  0.48 秒，每秒   8053 个字，每一首从开始到写完 0.12 秒
  一次写 64 首：总共  0.44 秒，每秒   8706 个字，每一首从开始到写完 0.44 秒

Writing the same 64 poems: one at a time takes 1.84 seconds; 16 at a time takes only 0.48 seconds, raising the characters written per second (throughput) to about 4 times.

But look at the last column. At 1 poem at a time, each poem is done in 0.03 seconds; at 64 at a time, every poem waits 0.44 seconds for its result. The bigger the batch, the higher the overall efficiency, but the longer each request waits (latency).

And from 16 to 64, throughput barely improves (8,053 to 8,706): the CPU's compute is maxed out. GPUs have far more compute, and gain far more from batching.

That's the core problem a model service has to solve: put as many requests together as possible, within an acceptable latency.

Two hard problems in real services

Our experiment is idealised: 64 poems starting at the same moment, each 60 characters long. Real requests aren't like that.

Requests vary in length and arrive at any time. One request needs 20 characters written, another 2,000. If they're in one batch and the next batch waits until the longest finishes, the short request is long done but still occupying a slot, and new requests just have to wait. vLLM's answer is continuous batching: after every generation step, it checks, removes finished requests from the batch immediately and adds new arrivals immediately. The requests in the batch keep changing, and the GPU stays fully loaded.

The KV cache takes a lot of space, and its size is unpredictable. As Module 09, Lesson 6 and last lesson calculated, the KV cache grows as the text gets longer, and how much each request will end up needing isn't known in advance. The traditional approach reserves one contiguous block of GPU memory per request for the maximum length, but most requests never use that much, so lots of memory is wasted and fewer requests can be handled at once.

PagedAttention

vLLM's core invention is called PagedAttention, from the 2023 paper "Efficient Memory Management for Large Language Model Serving with PagedAttention". It borrows the "paging" approach operating systems use to manage memory:

传统做法：每个请求预留一整块，按最大长度
  请求 A  [■■■■□□□□□□□□□□□□]   用了 4 格，空着 12 格
  请求 B  [■■■■■■□□□□□□□□□□]   用了 6 格，空着 10 格

分页：KV 缓存切成固定大小的小块，用到哪儿分配到哪儿
  块池    [A1][B1][A2][B2][B3][C1][空][空] ...
  请求 A 的块表：A1 → A2
  请求 B 的块表：B1 → B2 → B3

Each request's KV cache is cut into small fixed-size blocks that don't need to be stored contiguously; when one block fills, the next is allocated, so each request uses only what it actually needs. There's almost no waste, and the same GPU memory can serve far more requests at once.

When several requests share the same beginning (such as the same system prompt), these blocks can also be shared instead of stored separately for each. The "cache-hit input is cheaper" from Module 06, Lesson 4 is a similar mechanism on the server side.

The paper's abstract reports that at the same latency, vLLM's throughput was 2 to 4 times that of other systems at the time, with bigger gains for longer texts and larger models.

Installation and startup

The following commands are quoted from vLLM's official docs (as of September 2026) and need a Linux machine with an NVIDIA GPU. vLLM also supports AMD and Intel GPUs as well as some CPUs and other hardware, but installation differs; see the official docs.

uv venv --python 3.12 --seed
source .venv/bin/activate
uv pip install vllm --torch-backend=auto

Starting a service takes one command:

vllm serve Qwen/Qwen2.5-0.5B-Instruct

By default it serves an OpenAI-compatible API at http://localhost:8000. As with last lesson's Ollama, Part 1's code works by changing three environment variables:

export LLM_BASE_URL=http://localhost:8000/v1
export LLM_API_KEY=你设置的密钥
export LLM_MODEL=Qwen/Qwen2.5-0.5B-Instruct

When the service is for other people, always add key checking. Add --api-key at startup, or set the environment variable VLLM_API_KEY, and the server checks the key in the request header. As Module 06, Lesson 6 said, an API with no limits at all put on the internet means paying for other people.

Serving several LoRAs at once

Lessons 2 and 3 mentioned one of LoRA's benefits: one base model plus many small adapters. vLLM can serve several LoRAs at once in one service, with each request specifying which one to use:

vllm serve Qwen/Qwen2.5-0.5B-Instruct \
    --enable-lora \
    --lora-modules repobot=./.cache/repobot-lora

Put the LoRA's name, repobot, as the model name in a request and it uses the adapter fine-tuned in Lesson 3; put the original model's name and it uses the original model. Several customers each have their own LoRA while taking only one base model's worth of GPU memory.

Deploy yourself, or call an API

At this point, you can deploy models yourself. Whether you should is another question.

Deploying yourself means buying or renting GPUs, which cost money even when idle; handling scaling, monitoring, failures and upgrades yourself; and being limited to models your GPUs can hold.

Calling an API means paying by usage, with no idle cost, and access to the best models. As Module 01, Lesson 4 calculated, APIs like DeepSeek's are already very cheap.

Generally, a few situations make self-deployment worth considering: data mustn't leave your own servers; usage is high and steady enough that self-deployment costs less overall; or you need to use your own fine-tuned model. Before deciding, work out the costs of both options for real, following Module 06, Lesson 4.

Exercises

1. Add batch sizes 8 and 32 to batch_throughput.py, and find the batch size on your computer beyond which throughput stops improving noticeably.
2. Modify the experiment so poems in a batch vary in length (say, half write 20 characters and half 120), and the whole batch waits until the longest finishes. How much does throughput drop compared with all writing 60 characters? That's the problem continuous batching solves.
3. If you have access to a machine with an NVIDIA GPU (a cloud server, say), start vLLM with this lesson's commands and have Module 06's RepoBot v4 call it.

Self-check

1. Why does processing several requests together raise throughput? What's the cost?

Each generation step reads the model's parameters, and on an LLM most of the time goes into reading them. Processing several requests together computes one character for each request from a single read of the parameters, making full use of the compute units. The cost is longer latency per request: it's computed alongside others, and once the batch is big enough to max out compute, throughput stops improving too.

2. What problem does continuous batching solve?

Real requests vary in length and arrive at any time. If a batch has to wait for its longest request to finish before the next batch starts, finished short requests keep occupying slots and new requests have to wait. Continuous batching removes finished requests and adds new ones at every step, keeping the GPU fully loaded.

3. How does PagedAttention save GPU memory?

It cuts each request's KV cache into small fixed-size blocks allocated as needed, instead of reserving one contiguous block for the maximum length, so there's almost no waste. Blocks with the same prefix can also be shared between requests. The memory saved can be used to handle more requests at once.