Llumnix (Ray edition): cross-instance scheduling for multi-instance vLLM, and why upstream now points elsewhere

The problem is the gap between one vLLM instance and a fleet of them

A single vLLM instance schedules its own requests well. Put sixteen of them behind a load balancer and the scheduling problem moves up a level, where the balancer can only see request counts or queue depths, not what each instance is actually holding in KV cache. Llumnix targets that layer. The README describes it as a cross-instance request scheduling layer built on top of inference engines such as vLLM, and the two stated goals are latency (time-to-first-token and time-between-tokens) and throughput. The audience is narrow and specific: teams already running multi-instance vLLM serving who want better dispatch decisions than round-robin, and researchers who want to prototype scheduling policies without rewriting an engine. The README's own framing of this repository is that it suits local deployments and quick prototyping of scheduling ideas, while the newer v1 architecture is aimed at production and cloud-native environments. That sentence is the most important one in the document, because it tells you what the maintainers think this code is for.

What actually moves: KV cache aware dispatch plus continuous rescheduling

The mechanism has two halves. The first is dispatch: the scheduler is described as dynamic, fine-grained and KV-cache-aware, meaning the placement decision accounts for cached state rather than only queue length. The second half is the unusual part. Llumnix reschedules requests continuously after they have been placed, using a KV cache migration mechanism the README says has near-zero overhead. That is what makes the rest possible: the same migration primitive is reused for load balancing, for de-fragmentation across instances, and for prefill-decode disaggregation, where prefill and decode run on different instances and the cache has to follow the request across the boundary. Most multi-instance serving stacks treat placement as a one-shot decision made at admission. Llumnix treats it as revisable, which is a meaningfully different design and also the source of most of its complexity. The README does not explain how the migration is implemented or what near-zero overhead is measured against, and that gap matters if you are trying to reason about tail behaviour under memory pressure.

Two entrypoints, and the one-line swap for existing vLLM deployments

Deployment is deliberately undramatic. If you already run multi-instance vLLM, the README says to replace the per-instance command python -m entrypoints.vllm.api_server with python -m llumnix.entrypoints.vllm.api_server, keeping flags such as --host and --port. The README states that during serving deployment Llumnix configures itself and acts as the scheduling layer over the multiple engine instances, so the surrounding orchestration does not change. The second entrypoint is serve, which the README describes as deploying Llumnix through the Ray job submission API, with details in docs/Quickstart.md. The api_server path is the compatible one, matching the default single-instance vLLM deployment method. Beyond the entrypoint command and the host and port flags, the README does not list the scheduler configuration keys in the material available here; they are in the docs tree, which also covers Supported_Models, Fault_Tolerance, Simulator and Prefill-decode_Disaggregation. If you need to know which knob selects a scheduling policy, that is a documentation lookup, not something you can infer from the README.

The v0 to v1 split is a maintenance fact, not a footnote

In March 2026 the maintainers refactored Llumnix into v1, described as more modular and cloud-native and better suited to production, and said future iteration would happen in the new repository. This repository is explicitly named as the Ray-based architecture, Llumnix v0, and positioned for local deployments and quick experimentation. Read that as a fork in the road rather than a deprecation notice: the code is not archived, the last push is recent, and the README still documents it fully. But the direction of travel is stated plainly. The only release listed is v0.1.0 from December 2024, and the README's own roadmap calls the project alpha, with architectural work on distributed serving and coordination, policy work on dispatching, migration and auto-scaling, and engineering work on testing and CI/CD all listed as planned. For anyone deciding where to invest, the practical consequence is that improvements to the scheduler, the migration path and the fault tolerance story are likely to land in v1 first. If you build on this branch, you should expect to port later or to maintain your changes yourself.

The published numbers come from one specific configuration

The performance section reports experiments with 16 Qwen2.5-7B instances on A10-24GB GPUs and 16 Llama2-13B instances on A800-80GB GPUs, driven by Poisson arrivals with ShareGPT input and output lengths. Against a simple round-robin scheduler, the README reports TTFT improvements up to 6.4x mean and 12.1x P99, plus 12% on P99 TBT, and average preemption stalls shorter by two orders of magnitude. Against a load balancer based on queue sizes, it reports up to 4.6x mean and 9.1x P99 TTFT and 15% on P99 TBT. These are the authors' measurements on that hardware and that trace, not a general claim, and the README does not give a reproduction script in the material available here. The honest reading is that the advantage over round-robin is largest exactly where round-robin is worst: heterogeneous cache state and bursty arrivals. If your traffic is uniform and your instances are identical, the ceiling on what a smarter scheduler can win is much lower than those figures suggest.

Where a cross-instance scheduler is the wrong layer

The clearest limitation is that Llumnix only helps once you have multiple instances and a reason to move work between them. On a single GPU, or on a handful of instances with steady traffic, the scheduling layer adds a coordination component and a migration path without much to schedule. The README's own positioning of this branch toward local deployments and prototyping cuts both ways: it is a good place to test a policy, and a questionable place to put your only control plane. There is also a dependency cost. The README notes an update to vLLM v0.6.3.post1 in January 2025, which tells you the integration tracks a specific engine version rather than floating freely; upgrading vLLM underneath a scheduling layer that inspects KV cache state is not a routine version bump. Finally, the fault tolerance documentation is referenced but not summarized in the README, so any claim about what happens when one instance dies should be read from docs/Fault_Tolerance.md directly rather than assumed.

The alternative is usually the load balancer you already have

The comparison the README itself draws is against a round-robin scheduler and against a queue-size-based load balancer, and the difference in approach is the whole point. Round-robin distributes requests evenly without looking at instance state. A queue-size balancer looks at one number per instance. Llumnix looks at cached state and keeps the option to move a request after admission. That extra information is what buys the reported TTFT reductions, and it is also what you pay for in coordination. If your workload has short prompts and uniform generation lengths, cache state correlates weakly with load and the queue-size balancer is a reasonable place to stop. If you are comparing against a full serving framework, the distinction to hold onto is that Llumnix is not an engine. It is a layer that assumes vLLM underneath, which is why the migration command in the README is a one-line substitution rather than a migration project.

Editorial conclusion

Adopt this repository if you want to experiment with cross-instance scheduling policies against a working vLLM deployment, and you accept that upstream has moved to a separate v1 codebase. Do not adopt it as a new production control plane without first checking the current state of the v1 repository, since the README frames this version as the prototyping branch. Before committing, verify three things in the docs: the exact vLLM version this branch pins, whether the migration mechanism is enabled by default in the api_server entrypoint, and what Fault_Tolerance.md actually promises when a single instance dies.

Official sources

1. Issues
2. License: Apache-2.0
3. llumnix-project/llumnix-ray on GitHub
4. README
5. Releases