Paddler: a two-binary LLM load balancer built on llama.cpp slots
Open-source LLM/VLM load balancer and serving platform for self-hosting LLMs (and VLMs) at scale 🏓🦙 Alternative to projects like llm-d, Docker Model Runner, etc but with less moving parts and simple deployments built around ggml ecosystem. Runs on CPU and GPU.
At a glance
- What is it?
- Paddler is an Apache-2.0 Rust load balancer and serving platform for self-hosted LLMs and VLMs, built around a llama.cpp engine and its own slot implementation. It reduces a serving stack to two deployable processes, at the cost of tying you to the ggml model format.
- Who is it for?
- Adopt Paddler if your models are GGUF and you want a balancer, agent fleet and admin panel without a Kubernetes control plane or a separate inference server. Do not adopt it if you need vLLM or TensorRT-LLM backends, or if you cannot accept that the balancer's own slot implementation, not upstream llama.cpp, owns context and KV cache.
- Can I use it commercially?
- Yes. Apache-2.0 is a permissive licence: you can use, modify and sell software built on it, as long as you keep its copyright and licence notices.
- Is it still maintained?
- Yes. The repository last received commits 58 days ago.
- What is it written in?
- Mainly Rust, according to GitHub's language statistics.
Answers come from the project's GitHub data, last synced on September 15, 2026, and from our analysis. They are not legal advice.
DEEP OPEN-SOURCE ANALYSIS
The problem Paddler targets: inference without a control plane
Self-hosting an LLM for a product feature usually means assembling three things: an inference engine, a process that spreads requests across machines, and something to watch the fleet. The README frames Paddler as a response to that assembly work, describing it as an open-source LLM load balancer and serving platform that lets you run inference, deploy, and scale LLMs on your own infrastructure. The stated audience is product teams adding inference or embeddings to features, DevOps and LLMOps teams running models at scale, and organizations with privacy or compliance constraints in sectors like medical and financial work. Cost predictability is named explicitly: the README contrasts self-hosting with exposure to per-token pricing. That is the pitch. The interesting part is the shape of the answer, which is a single binary containing a balancer and an agent, rather than a scheduler plus a model server plus a metrics stack.
Balancer, agents and slots: the three-level request path
The README describes two deployable components. The balancer exposes an inference service for applications, a management service for internal coordination, and an optional web admin panel. Agents are described as usually deployed on separate instances, and they further distribute incoming requests to slots. Slots generate tokens and embeddings. So the path is application to balancer to agent to slot, with the management service carrying the control traffic between balancer and agents. The README states that Paddler uses a built-in llama.cpp engine for inference but has its own implementation of llama.cpp slots, which keep their own context and KV cache. That sentence is the architectural centre of the project. Paddler is not proxying to a llama.cpp server process; it embeds the engine and manages slot lifetime itself. The README also notes agents can be added dynamically, which the project presents as the hook for autoscaling tools, and that request buffering enables scaling from zero hosts. Dynamic model swapping is listed as a feature, which implies the slot layer can release and reload a model without restarting the agent.
Starting a balancer and an agent: the actual commands
Installation is a binary download from GitHub releases or a source build, with the README stating an MSRV of 1.88.0. The README gives the balancer command as `paddler balancer --inference-addr 127.0.0.1:8061 --management-addr 127.0.0.1:8060 --web-admin-panel-addr 127.0.0.1:8062`, noting that the web admin panel flag is optional. The agent command is `paddler agent --management-addr 127.0.0.1:8060 --slots 4`. Three ports are in play on the balancer side: inference, management, and the admin panel. The agent needs only the management address and a slot count. Running `paddler --help` lists the remaining commands, according to the README. What the README does not show is how a model is selected at agent startup, or whether model configuration happens through the admin panel, a config file, or flags. The admin panel screenshots are captioned as letting you add and update a model and customize the chat template and inference parameters, which suggests model setup is a runtime action through the panel rather than a command-line argument. Anyone planning an unattended deployment should confirm that path in the documentation before assuming a fully flag-driven boot.
Where the single-binary design stops helping
The built-in llama.cpp engine is the constraint that matters most. Paddler's inference path is ggml, so models must be in a format llama.cpp can load, and the README's own framing of the project as built around the ggml ecosystem makes that explicit. If your serving plan depends on a different runtime, Paddler is the wrong tool regardless of how clean the deployment story is. The second constraint is the custom slot implementation. Because Paddler writes its own slots with their own context and KV cache, its memory behaviour and batching semantics are Paddler's to maintain, not upstream llama.cpp's. That is a maintenance surface the project has taken on deliberately, and it is also why the README can claim dynamic model swapping and zero-host scaling, since those features need control over slot lifecycle that an external server would not expose. The trade-off is that upstream llama.cpp changes do not automatically arrive as improvements to Paddler's slot layer. The desktop application is labelled beta in the README, so the casual multi-laptop use case is the least settled part of the product.
Alternatives and the difference in moving parts
The README names llm-d and Docker Model Runner as comparable projects and characterises Paddler as having less moving parts and simple deployments. The comparison is worth taking at face value on the deployment axis. llm-d is a Kubernetes-native effort, which means a cluster, a scheduler and a set of custom resources before you serve a token; Paddler's equivalent is one balancer process and one agent process per host. Docker Model Runner keeps model distribution and execution inside the Docker toolchain, which suits teams already standardised on Docker Desktop and container workflows, while Paddler ships as a standalone binary you place on a machine. The difference is not capability so much as what you already operate. A team with a Kubernetes platform and GPU scheduling already solved will find llm-d's approach less foreign than a second, separate balancer. A team with three GPU workstations and no cluster will find the reverse. Paddler's desktop app is a third position: it targets office clusters and multi-laptop setups that neither of the other two address directly.
Licence, upgrades and the cost of tracking a young project
Paddler is Apache-2.0, which permits commercial use, modification and redistribution under the terms of that licence, including its patent grant and notice requirements. That is a permissive baseline, but the licence text governs, and this article is not legal advice. On upgrade cost, the release cadence visible in the repository is fast: v4.0.1, then v4.1.0-rc1, then v4.1.0 within roughly two weeks, with the 4.1 line labelled OpenCode support. A major version bump from 4.0 to 4.1 in that window suggests the project is still moving its interfaces. The README's answer on AI-generated code is relevant to maintenance expectations: it states that all code is human-reviewed and most is handcrafted, with AI assistance used so far for the HTTP client that connects to the core library and for an integration test harness. For an operator, the practical question is whether agent and balancer need to be upgraded in lockstep. The README does not state a compatibility policy between balancer and agent versions, so that is something to establish from the release notes before running a mixed-version fleet.
What to verify before you commit
The README describes a web admin panel for management, monitoring and testing, and observability metrics as a feature, but it does not name a metrics format or endpoint. If your monitoring stack expects Prometheus exposition, confirm the format before you plan dashboards around it. The same applies to embeddings: they are mentioned as a capability of slots and as a reason product teams adopt Paddler, but the README does not show an embeddings request example, so check the API documentation for the endpoint shape. Function calling, grammars and multimodal models each have a linked guide in the README's Starting out list, which is a reasonable sign that these paths are documented, though this article has not exercised them. Finally, the zero-host scaling claim rests on request buffering. Verify what happens to buffered requests when no agent is available and how long the balancer holds them, because that timeout is the difference between a slow first request and a failed one.
Editorial conclusion
Adopt Paddler if your models are GGUF and you want a balancer, agent fleet and admin panel without a Kubernetes control plane or a separate inference server. Do not adopt it if you need vLLM or TensorRT-LLM backends, or if you cannot accept that the balancer's own slot implementation, not upstream llama.cpp, owns context and KV cache. Before deploying, verify that your target GGUF model loads through the built-in engine, confirm the management and inference ports do not collide with existing services, and check whether the desktop app's beta status matters for your users.
Community notes