LoRAX: Serving thousands of LoRA adapters on one GPU with dynamic loading

The problem: one GPU, many fine-tuned models

Serving fine-tuned LLMs traditionally means loading each model separately, which multiplies memory and cost. LoRAX addresses this by keeping a single base model in GPU memory and swapping lightweight LoRA adapters per request. The project claims this can serve thousands of fine-tuned models on one GPU, reducing serving cost without sacrificing throughput or latency. The intended user is someone who has many task-specific adapters, such as a team that fine-tunes a model for different customers or use cases, and wants to avoid spinning up a separate instance for each. The README frames it as a framework rather than a single tool, and the homepage points to LoRA Exchange, suggesting a broader ecosystem around sharing and serving adapters.

How LoRAX separates base model from adapter

LoRAX splits serving into two components: a base model that is shared across all requests, and an adapter that carries task-specific weights. The base model is a pretrained large language model like Llama, Mistral, or Qwen, loaded once. The adapter is a LoRA weight set that can be fetched from HuggingFace Hub, Predibase, or any local filesystem. When a request arrives, the server loads the requested adapter just-in-time, without blocking concurrent requests that use other adapters. The architecture relies on three mechanisms named in the README: heterogeneous continuous batching packs requests for different adapters into the same batch; adapter exchange scheduling prefetches and offloads adapters between GPU and CPU memory; and optimized inference kernels including flash-attention, paged attention, and SGMV handle the compute. The result is that the number of concurrent adapters should not degrade throughput, which is the core claim. The README does not describe the internal scheduling algorithm in detail, so the exact heuristics are not verifiable from this material.

Supported base models and adapters

The base model must be one of the supported architectures. The README explicitly lists Llama, CodeLlama, Mistral, Zephyr, and Qwen, and refers to a complete list in the documentation, which is not included here. Base models can run in fp16 or be quantized with bitsandbytes, GPT-Q, or AWQ. Adapters must be trained with the PEFT or Ludwig libraries, and any linear layer in the model can be adapted. This is a concrete constraint: if you have adapters trained with a different method, such as full fine-tuning or a non-PEFT library, LoRAX will not serve them. The README also mentions that adapters can be merged per request, which suggests a feature for combining multiple adapters into an ensemble, though the details are not in this excerpt. The separation means you must have a compatible base model for each set of adapters, so the savings depend on how many adapters share one base.

Getting LoRAX running: Docker first

The recommended path is a prebuilt Docker image, which avoids compiling custom CUDA kernels. The requirements are strict: an Nvidia GPU of Ampere generation or newer, CUDA 11.8 compatible drivers, Linux, and Docker. You must install the nvidia-container-toolkit, then run two system commands: sudo systemctl daemon-reload and sudo systemctl restart docker. The launch command is a single docker run that mounts a local volume for adapter data and exposes port 8080. For example, to serve Mistral-7B-Instruct-v0.1, you set model=mistralai/Mistral-7B-Instruct-v0.1 and run ghcr.io/predibase/lorax:main --model-id $model. After launch, you send a POST to /generate with JSON inputs and parameters. To use a LoRA adapter, add an adapter_id field in the parameters, pointing to a HuggingFace repo like vineetsharma/qlora-adapter-Mistral-7B-Instruct-v0.1-gsm8k. A Python client is available via pip install lorax-client, and the README shows a simple Client object with a generate method that accepts adapter_id. The REST API and Python client both support token streaming, but the excerpt does not show the streaming parameters.

Production features and their limits

The README lists production features: prebuilt Docker images, Helm charts for Kubernetes, Prometheus metrics, OpenTelemetry distributed tracing, an OpenAI-compatible API for multi-turn chat, per-request tenant isolation for private adapters, and structured output in JSON mode. These features make LoRAX look ready for deployment beyond a single test. However, the README does not provide any configuration examples for these features, so the operational details are not verifiable from this material. Tenant isolation is described as per-request, which implies that adapter access is controlled at the request level, but the mechanism is not specified. The OpenAI-compatible API is a notable advantage for teams that already use OpenAI clients, because it reduces integration work. The structured output feature is mentioned without detail, so it is unclear whether it supports constrained decoding or only JSON formatting. These gaps mean that production adoption requires consulting the full documentation, which is not included in this excerpt.

Maintenance and upgrade cost

The project is under active development, with a release lorax-0.4.0 in January 2025 and earlier v0.12.x releases in late 2024. The version numbering is inconsistent, with a jump from v0.12.1 to lorax-0.4.0, which suggests a possible rebranding or versioning change. The v0.12.0 release notes mention multi-LoRA prefix caching, fp8 kv cache, Mllama support, and function calling, indicating that the project evolves quickly. This means users must track releases to benefit from performance and feature improvements. The dependency on custom CUDA kernels means that upgrading the server may require rebuilding or pulling a new Docker image, and kernel compatibility with new GPU drivers is a risk. The project is not archived and had a push in May 2026, which suggests ongoing maintenance, but the material does not provide a changelog or migration guide. The Apache-2.0 license allows commercial use and modification, but it does not come with warranty or support obligations, so teams must rely on the community or their own engineering.

When LoRAX is the wrong tool

LoRAX is not for every serving scenario. If you serve a single fine-tuned model with no need for multiple adapters, the complexity of adapter scheduling and the requirement to keep a base model resident adds overhead without benefit. The hardware requirement of Ampere or newer GPUs excludes older data center GPUs like V100, which are still common. If your adapters are not LoRA-based, such as full fine-tuned weights or adapters from a non-PEFT library, LoRAX will not load them. The README also does not mention CPU inference or non-Nvidia GPUs, so the server is effectively tied to Nvidia hardware. Another limitation is that the base model must be one of the supported architectures, so if you have a custom or less common architecture, you may need to wait for support. The documentation does not provide benchmark numbers, so the claim that throughput stays constant with the number of concurrent adapters is unverified. In a scenario where you have many adapters but they are all used rarely, the adapter exchange scheduling might thrash if the working set exceeds CPU memory, though the README does not specify memory requirements for the CPU offload.

A real alternative: serving each adapter as a full model

The conventional alternative is to serve each fine-tuned model as a separate instance, for example using a standard inference server like vLLM or TGI, which the README does not mention but is a common approach in the ecosystem. The difference is architectural: instead of sharing a base model and swapping adapters, you load each model's full weights into memory, either on separate GPUs or sequentially. That approach is simpler to reason about, because there is no adapter scheduling or dynamic loading, and it works with any fine-tuning method, not just LoRA. The cost is higher memory usage per model, so you can serve far fewer models on the same hardware. If you only have a handful of fine-tuned models, the full-model approach may be more straightforward and avoids the risk of adapter loading latency. If you have hundreds or thousands of adapters, LoRAX's shared base model becomes attractive, but you must accept the constraint that all adapters must be LoRA-compatible with a supported base.

Editorial conclusion

Adopt LoRAX if you serve many fine-tuned LoRA adapters on shared GPU infrastructure and can meet the Ampere-or-newer, CUDA 11.8, Linux, and Docker requirements. Do not adopt it if your adapters are not LoRA-compatible, if you lack the required GPU generation, or if you need to serve a single model without the complexity of adapter swapping. Before committing, verify that your base model architecture is in the supported list and that your adapters were trained with PEFT or Ludwig, since other adapter formats are not covered. The project is Apache-2.0 licensed, so commercial use is unrestricted, but you must check the licenses of the base models and adapters you serve, as they are separate. Start with the prebuilt Docker image and test with a small set of adapters to measure real throughput, because the documentation does not provide benchmark numbers to confirm the stated scaling.

Official sources

1. License: Apache-2.0
2. predibase/lorax on GitHub
3. Project website
4. README
5. Releases