# serverlessllm: the ten times claim comes from the cached case, and the multi-model case bottoms out at 5.93

> A shared-GPU serving system whose headline is a custom checkpoint format loaded with O_DIRECT I/O, measured on H100 hardware with NVMe storage. The quickest row in its own table is a reload of the same model, which the footnote says is not the multi-model scenario, while the two rows that represent it sit lower.

**ServerlessLLM/ServerlessLLM** — Serverless LLM Serving for Everyone.

- Repository: https://github.com/ServerlessLLM/ServerlessLLM
- Website: https://serverlessllm.github.io/
- Stars: 717 · Forks: 77
- Language: Python
- License: Apache-2.0
- Published: 2026-09-10 · Updated: 2026-09-10 · Language: en
- Canonical page: https://hysenlabs.com/projects/serverlessllm-serverlessllm

## The ten times figure is a reload of the same model, not the multi-model case

The tag line at the top of the document says load models ten times faster, and the performance section says six to ten times faster than SafeTensors. The table underneath is where those two numbers come apart. Two scenarios are reported per model: a random one and a cached one, and the footnote defines them. Random simulates serverless multi-model serving. Cached shows repeated loading of the same model. For a 32B Qwen model the random row goes from 20.6s to 3.2s, which is 6.40 times, and the cached row from 12.5s to 1.3s, which is 9.95 times. The same shape holds for the DeepSeek distill, at 5.93 times random and 8.58 times cached, and the fastest single row in the table for the small Llama model is 6.54 times on a random load. So the top-line number is the cached figure, the scenario that does not represent serving many models, and the scenario that does tops out at 6.54 times while dipping to 5.93 times on the largest model.

## Every measurement is an H100 with NVMe, and the AMD path is labelled experimental

The footnote under the table states the conditions: results obtained on NVIDIA H100 GPUs with NVMe SSD. There is no second machine, no consumer card, no network storage in the comparison, and no CPU. That matters because the mechanism is a disk read. The store writes a custom binary format optimised for sequential reads, bypasses the OS page cache with O_DIRECT I/O, holds a pinned memory pool for direct memory access transfers to the GPU, and loads with multiple threads in parallel. All four of those pay off differently depending on the device. The supported hardware list names NVIDIA compute capability 7.0 and above, naming V100, A100, H100 and RTX 3060, and AMD on ROCm 6.2 and above, naming MI100 and MI200 series, annotated as experimental. So the AMD path is offered without a performance claim attached to it, and the table gives you no way to estimate what a different card or a slower drive will do.

## The quick start pulls its compose file from the moving main branch

The ninety-second path is five commands:

```bash
# Download the docker-compose.yml file
curl -O https://raw.githubusercontent.com/ServerlessLLM/ServerlessLLM/main/examples/docker/docker-compose.yml

# Set model storage location
export MODEL_FOLDER=/path/to/models

# Launch cluster (head node + worker with GPU)
docker compose up -d

# Wait for the cluster to be ready
docker logs -f sllm_head
```

The later deploy step execs into that same head container and runs the CLI from a hardcoded interpreter path inside it:

```bash
docker exec sllm_head /opt/conda/envs/head/bin/sllm deploy --model Qwen/Qwen3-0.6B --backend transformers
```

Two things follow from the way this is written. A cluster built this way tracks the branch, not a version, so two people running the quick start a week apart can get different images. And the conda path inside the container is an assumption about the image layout, which is fine for the shipped image and wrong for anyone who rebuilds it with a different Python environment.

## The Docker-free path needs a store server, and the example ends on an undefined variable

For use without Docker, the loader is packaged separately and installed by its own pip command. The sequence is to save a model into the store format, start a store server with a storage path and a memory pool size given as a value with a unit, and then load the model from your own Python code. The import comes from the store package's transformers module, and the call takes a model name, a device map and a dtype. The mechanism is then listed: a custom binary format for sequential reads, direct I/O that skips the page cache, a pinned memory pool for transfers, and parallel multi-threaded loading. The example does not run as printed, because the final line calls generate with a keyword-expanded inputs variable that appears nowhere above it. Small fault, and the kind that costs a reader the ten minutes in which they wonder whether the loader is broken or they are.

## The file named pyproject.toml contains no project metadata at all

The packaging is split in a way worth knowing before you build anything. The pyproject file declares a build system using setuptools and wheel, and then contains nothing but tool configuration: a linter section with a line length of eighty, an isort section, and a codespell section that skips build and image paths. There is no project table, no name, no version, no dependencies and no console script. All of that lives in setup.py, which reads the README for its long description, pulls requirements out of separate files with a helper function, and uses find_packages. The result is a project where the file a modern tool looks at first has no answer in it. The linter configuration has its own oddity: the selected rule set is a single letter for import sorting, while the unfixable list names five other rules that are never selected, and one of them is also repeated in the ignore list.

## Every requirement is pinned exactly except the store, and a bandwidth tester is in the runtime list

The main requirements file pins nine packages to exact versions and leaves one floating. Pinned are the web framework, a GPU monitoring library, the OpenAI client, two Pydantic packages, Ray, requests, the speedtest command line tool, a stub package for requests, and the ASGI server with its standard extras. The one unpinned entry is the store package itself, which is the component the load speed depends on, so the version of the thing doing the work is the one thing your environment will resolve for itself. Two entries deserve a question. A bandwidth speed test tool has no business in a serving runtime unless it is used for the benchmark scripts, and a GPU monitoring library is the kind of dependency that should be optional. Separating those out is what the other requirement files are for, since the root listing also carries separate files for lint, test and worker environments.

## An Ubuntu 20.04 base adapted from another project's Dockerfile pins the whole stack

The container build starts from an NVIDIA CUDA base tagged with a specific CUDA version on Ubuntu 20.04, with a Python version argument alongside, and the file header says it was adapted from a vLLM Dockerfile at a fixed commit. That combination is deliberate rather than accidental: an older distribution paired with a specific CUDA runtime is how you get a reproducible image for driver compatibility, and pinning the upstream commit means the build recipe does not drift when that project changes. What it also means is that the Python version in your container is not something you choose, and that the project inherits another project's build decisions wholesale. The Python side of the repository also carries C++ formatting configuration and a manifest file, which is consistent with the store being a compiled component rather than pure Python, and the tree holds two packages, one for the serving cluster and one for the store.

## Conclusion

This is worth evaluating if you have several models to serve from one GPU and you are currently paying load time on every cold start, because that is the problem the storage format attacks and the direction of the improvement is real. Judge the size of the win from your own hardware, not from the table. Every number in it was taken on H100 GPUs with NVMe storage, the AMD path is marked experimental, and the highest figures come from the cached scenario rather than the multi-model one that motivates the system. If you are on consumer hardware with a slower disk, your cold-start savings will be smaller than 10x, and your warm-reload savings smaller still since that path barely touches the disk. Two operational notes to settle first. The quick start downloads its compose file from the moving main branch rather than from a release, so pin a tag if you want a reproducible cluster. And the last release is v0.8.0 dated 2025-11-03 while the last commit is 2026-09-04, which means you would be running unreleased code if you install from the branch.

## FAQ

### What does ServerlessLLM actually do?

It deploys multiple AI models on shared GPUs through three mechanisms: a custom checkpoint storage format loaded with O_DIRECT I/O, GPU multiplexing with fast switching, storage-aware scheduling and live migration, and unified inference with serverless LoRA fine-tuning so one base model can serve many adapters.

### How much faster is loading with ServerlessLLM?

The README claims six to ten times faster than the SafeTensors checkpoint loader, measured on H100 GPUs with NVMe SSD. In its own table the random-load rows, which simulate multi-model serving, run from 6.54 times down to 5.93 times, while the rows near ten times are cached reloads of the same model.

### What hardware does ServerlessLLM support?

NVIDIA GPUs with compute capability 7.0 and above, including V100, A100, H100 and RTX 3060, and AMD GPUs on ROCm 6.2 and above, such as MI100 and MI200 series, which are marked experimental. Windows is covered through a WSL example rather than a native path.

### Can I use the fast loader without Docker?

Yes. Install serverless-llm-store, save a model with `sllm-store save`, start the store server with a storage path and memory pool size, then import load_model from the store package's transformers module and call it with a model name, device map and dtype.

### Does ServerlessLLM expose an OpenAI-compatible API?

Yes, described as a drop-in replacement. The quick start queries a chat completions endpoint on port 8343 with curl, and a separate embeddings endpoint is provided for retrieval use cases alongside embedding models.

## Sources

- [License: Apache-2.0](https://github.com/ServerlessLLM/ServerlessLLM/blob/main/LICENSE)
- [Project website](https://serverlessllm.github.io/)
- [README](https://github.com/ServerlessLLM/ServerlessLLM/blob/main/README.md)
- [Releases](https://github.com/ServerlessLLM/ServerlessLLM/releases)
- [ServerlessLLM/ServerlessLLM on GitHub](https://github.com/ServerlessLLM/ServerlessLLM)

---

Hysen Labs editorial analysis, written from the project's own repository and release notes. Cite the canonical page: https://hysenlabs.com/projects/serverlessllm-serverlessllm