LoLLMs Hub: An Ollama-Compatible Proxy That Puts a Master Node in Front of Your Inference Servers

The problem LoLLMs Hub targets: many Ollama instances, no shared front door

Running one Ollama server on a laptop is a solved problem. Running four of them, spread over a Windows gaming PC with two GPUs, a Linux box and a cloud vLLM deployment, is not. Each instance has its own address, its own model list, and no shared notion of who is allowed to call it. The README frames the project as a way to unify "disparate AI backends into a single, managed API", and the feature list backs that up: user management, rate limiting, analytics, model pull and delete across servers, and TLS termination. The intended user is someone operating a small private cluster, not a hobbyist with one GPU. The topics list (agent, api-gateway, federation, load-balancer, user-management) points the same way. If your setup is one Ollama process on localhost, this project adds a network hop and a database you now have to maintain, and the README offers nothing that justifies that.

The master hub pattern is the actual architecture, and it is the part worth understanding

The README's mermaid diagram shows a client hitting a Master Node, which fans out to worker machines. On a worker, a Slave Hub sits in front of the local Ollama instances, or the worker exposes a raw Ollama service, or a cloud endpoint such as vLLM. Three reasons are given for inserting a slave layer: the slave spawns local processes on the worker, the master sees one remote server entry instead of dozens, and only the master needs public exposure while slaves stay on a private VPN or VLAN. That third point is the strongest argument in the whole document. It means the security boundary is the master, and the workers can be unreachable from the internet. The API compatibility claim is what makes chaining possible: the README states the hub's API is compatible with both Ollama and OpenAI, so a hub can be a client of another hub. Treat that claim as the load-bearing assumption of the design. If your client uses an Ollama endpoint the hub does not implement, the chain breaks at that call, and the README does not enumerate which endpoints are covered.

Routers and ensembles: two different orchestration features that are easy to confuse

Smart Routers and Ensembles solve different problems and the README presents them in sequence without much separation. A router is a virtual model name backed by decision rules evaluated top to bottom. Fast rules cover keywords, regex patterns, message length, image detection and specific users. Semantic rules hand classification to a small LLM when pattern matching is not enough. Three routing strategies are listed: priority with fallback, random, and least loaded, the last described as routing to the backend with the lowest active request count. Ensembles are a mixture-of-experts flow: several agents run in parallel over the same query, and a master model synthesizes their outputs. The contract example in the README shows three agents producing clause, case-law and compliance findings, then a synthesizer combining them. The cost difference matters. A router picks one backend per request. An ensemble multiplies your token spend by the number of agents plus the synthesizer, and it multiplies latency by the slowest parallel branch. The vision router example (image goes to a VLM, its description plus the original prompt goes to a text model) is a two-call chain, so even the simplest router can double latency on image requests.

Getting it running: two scripts and a setup wizard

The README gives a short install path. Clone the repository, then run the platform script:

git clone https://github.com/ParisNeo/lollms_hub.git cd lollms_hub

On Windows, the instruction is to double-click run_windows.bat. On macOS or Linux, the README gives:

chmod +x run.sh ./run.sh

The first run walks through a setup wizard and creates admin credentials. Stopping the hub is closing the terminal or pressing Ctrl+C, which tells you it runs in the foreground by default. TLS is configured through the UI at Settings, HTTPS/SSL, with two options: uploading key.pem and cert.pem, or supplying full file paths for certificates already on the server, as with Certbot. The README states a server restart is required for those changes to take effect. That is the entire documented configuration surface. There is no config file format, no environment variable list and no reverse-proxy example in the supplied material, so if you need to run this under systemd or behind nginx, you are working it out from the source.

Where this is the wrong tool

The documentation is thin in places that matter for production. There is no stated request throughput, no memory or disk footprint, no database requirement, and no description of what happens to in-flight requests when a backend dies mid-generation. The rate-limiting feature is mentioned as a topic and appears on the dashboard as "live rate-limit queues", but the README never defines the algorithm, the defaults or the config keys. The version history is the bigger warning. Releases are listed as v17.1.0 (described as the last stable version before a full revamp), then v8.0.0, then v9.0.0, which means the numbering restarted at some point. Anyone pinning a version needs to read the release notes rather than assume v9 is newer than v17 in any meaningful sense. The last push timestamp is 2026-04-23, so the project is active, but activity is not the same as a stable interface. If you need a documented, versioned API contract with deprecation windows, this repository does not currently offer one.

Alternatives and the difference in approach

LiteLLM Proxy is the closest comparison for the OpenAI-compatible side. It presents one OpenAI-shaped endpoint over many providers, and its configuration is file-driven, with a YAML config describing models and routing. LoLLMs Hub instead centers on a web UI for administration: adding servers, pulling models, creating users and building routers happen through the dashboard, and the README's screenshots walk through exactly that. The practical difference is where your configuration lives. With a file-driven proxy, your setup is reviewable in a pull request and reproducible across environments. With LoLLMs Hub as documented, the state lives in the hub's own storage and is edited by clicking. For a single operator managing a home lab, the UI is faster. For a team that needs the gateway config in version control, it is a gap the README does not address. On the Ollama side specifically, the master/slave chaining has no direct equivalent in the LiteLLM documentation, so if you want a private VLAN of workers behind one public master, that is LoLLMs Hub's distinct contribution.

Maintenance, licensing and what to verify before you commit

The licence is Apache-2.0, which permits commercial use and modification and includes a patent grant, but it also means you carry the obligation to preserve notices and state changes in modified files. That is a general property of the licence, not legal advice; check it against your own distribution model. Maintenance cost here is not the Python code, it is the topology. Every worker you add is another machine whose Ollama version, model files and network reachability you now track, and the master is a single point of failure for the whole cluster unless you run more than one. The README does not describe high availability for the master. Upgrade cost is the other item: with the version numbering reset visible in the release list, plan to read release notes per jump rather than assume a patch-level upgrade. Verify the compatibility claim first with your actual client, since the entire chaining design rests on the hub speaking Ollama and OpenAI well enough to be both a server and a client.

Editorial conclusion

Adopt LoLLMs Hub if you already run several Ollama instances across more than one machine and want one authenticated endpoint plus per-user usage tracking without writing your own gateway. Do not adopt it if you have a single Ollama box on localhost, or if you need a published API contract and stability guarantees, since the version history (v17.1.0 described as the last stable version before a full revamp, then v8.0.0 and v9.0.0) shows the numbering is not a reliable compatibility signal. Before deploying, verify two things against the running instance: that the Ollama and OpenAI compatibility endpoints in the README actually behave as drop-in replacements at your client's version, and that your TLS certificate paths survive a restart through Settings, since the README states a restart is required to apply them.

Official sources

1. Issues
2. License: Apache-2.0
3. ParisNeo/lollms_hub on GitHub
4. README
5. Releases