Olla puts one routing layer in front of mismatched inference servers

A proxy that sits in front of whatever inference server you run

Olla is a proxy and load balancer written in Go for local inference infrastructure. It routes requests across backends, fails over when one stops answering, and presents a single view of the models available behind it.

The problem it solves appears once you run more than one inference server. A workstation with a local runtime, a shared machine with a faster one, and perhaps a third with different model weights: each speaks its own dialect, each has its own address, and every client has to know which model lives where. Clients end up carrying routing logic that belongs in one place.

The badge row and topic list name the backends it targets, and the breadth is the point: the widely used local runtimes alongside the high-throughput serving engines, across three GPU vendors. Olla's value is proportional to how heterogeneous your setup is, which makes it a tool for people who accumulated inference capacity rather than provisioned it.

The audience is anyone self-hosting models across more than one machine or more than one serving stack, and wanting clients to stop caring which.

What the dependency list tells you about the design

The module file is short and every entry is informative, which is unusual enough to be worth reading as documentation.

An expression language library is present, which strongly implies routing rules are configurable expressions rather than fixed strategies. That is the difference between a load balancer you configure and one you modify.

A lock-free concurrent map library and a pair of fast JSON packages appear together, and they describe the same concern from two directions. A proxy in front of streaming model output touches JSON on every chunk and shares state across many concurrent requests, so the standard library's reflection-based JSON and mutex-guarded maps are exactly where a naive implementation spends its time. Choosing replacements for both says performance was measured rather than assumed.

A time-based cache library fits model discovery, where backend capabilities need refreshing periodically rather than on every request. A cross-origin package indicates browsers are expected to call it directly. A terminal formatting library and a terminal detection package suggest the output is written for a person watching a console.

Go itself is the right choice here. A proxy is concurrent connection handling with tight latency requirements and long-lived streams, and the runtime floor of a recent Go release means the modern concurrency and networking improvements are available.

Running it, and the container image

Distribution covers both a published container image and platform binaries attached to releases. The composition file in the repository describes the expected deployment shape, naming the image, the container and the single port it listens on.

  olla:
    image: ghcr.io/thushan/olla:latest
    container_name: olla
    ports:
      - "40114:40114"

That file references the moving tag, and pinning an exact version instead is the better habit for something sitting in a request path, since release notes publish a version-specific image for exactly that purpose.

Two environment variables carry the configuration.

    environment:
      - OLLA_CONFIG_FILE=/config/config.yaml
      - OLLA_LOG_LEVEL=info

The configuration file is mounted read-only from the host and the service listens on a single port, with a restart policy and a health check defined. Mounting configuration read-only is a small thing that prevents a container writing to a file you version.

The container itself is worth a look, because it was built with some care. It starts from a minimal base, installs only certificates, timezone data and a fetch tool, and creates an unprivileged user that the application runs as. Running a proxy as a non-root user inside the container is the baseline expectation and is skipped often enough to be worth noting when it is not.

There is a comment in that file explaining why runtime files are copied individually rather than wholesale: the same file is built from two different contexts, one from the repository root and one synthesised by the release tool, so naming each file keeps both builds honest. Explaining a non-obvious choice at the point of the choice is the kind of comment that survives usefully.

Version 0.0.29, and what that number should mean to you

The most recent release is 0.0.29, published on 2026-08-10, with the last push on 2026-08-21.

Read that leading pair of zeros carefully. A project at 0.0.x is telling you the interface, the configuration format and the behaviour are all subject to change, and there are twenty-nine of them, so change has been frequent. For a component sitting in the request path of everything your clients do, that is the central risk: an upgrade that alters routing behaviour changes which backend answers, and the symptom is a latency or quality difference rather than an error.

Against that, the engineering signals are good. The repository carries a linter configuration, a dedicated security scanner configuration, a release automation configuration, a continuous integration workflow, a code review tool configuration, an examples directory and a documentation directory. A separate security scanner configuration in a Go project is a deliberate addition rather than a default, and for a network proxy it is the right one to add.

The practical posture is to pin an exact image tag, read release notes before moving, and keep a configuration file you control in version control rather than relying on defaults that may shift.

What this does not do

Olla routes and balances. It does not run models, so every backend it fronts is something you already operate, and it adds a hop rather than capacity. If your problem is that inference is slow because you have one machine, this does not help.

It is also infrastructure you now maintain. A proxy in the request path is a new thing that can fail, and failover protects against a backend failing rather than against the proxy itself. Nothing in what is published describes running more than one instance, so the proxy is a candidate single point of failure in a setup built for resilience.

The heterogeneity that makes it valuable also bounds it. Backends differ in more than their addresses, since the serving engines expose different capabilities, and a unified view necessarily presents the common subset or leaks the differences. Anyone depending on a backend-specific feature should check how it survives the proxy.

Finally, the project is one maintainer's work by the look of the repository, at a pre-release version, in a category where the alternative is often a general purpose proxy that many people already run. That is not a reason to avoid it and it is a reason to pin versions.

A general purpose reverse proxy is the alternative

Most teams reaching this problem already run a general reverse proxy or an ingress controller, and could point it at their inference servers with a routing rule.

The difference in approach is how much the proxy understands. A general proxy routes by path, host and header, balances by connection count or round robin, and health-checks by fetching a URL. It knows nothing about models, so it cannot route a request for a particular model to the only backend that has it loaded, and its health check cannot tell a server that is up from one that is up but has no capacity.

Olla's argument is that a proxy for this workload should understand the workload: discovering which models each backend serves, presenting them as one catalogue, and routing on that knowledge. The expression library in its dependencies suggests those rules are yours to write.

Take the general proxy when your backends are interchangeable, when you already operate one and adding a rule costs nothing, or when you are unwilling to put a 0.0.x component in the request path. Take Olla when backends differ in which models they hold, when clients would otherwise need to know your topology, or when you want routing decisions expressed as rules rather than encoded in each client.

Apache terms and what to verify first

Olla is Apache-2.0 licensed, which includes an express patent grant and permits commercial use, and is the permissive choice most likely to clear an internal review without discussion.

The repository shows the habits of a project run with discipline: linting, security scanning, release automation, continuous integration, examples and documentation, together with instruction files for coding agents at the root, so the maintainer is building it with the kind of tooling it serves.

The verification order before putting this in front of anything that matters is short and specific. Pin the exact image tag rather than the moving one. Mount your own configuration read-only, as the composition file does, and keep it in version control. Then exercise a failover deliberately, by stopping one backend while traffic flows, because automatic failover is the feature you are adopting this for and it is the one that only proves itself when a backend goes away.

Editorial conclusion

Olla fits a setup where inference capacity accumulated across several machines and serving stacks, and where clients should stop knowing which backend holds which model. Skip it when your backends are interchangeable and a reverse proxy you already run would do, and think hard before putting a 0.0.x component in the request path of everything. Pin the exact image tag rather than the moving one, mount your own configuration read-only as the composition file does, and test a failover deliberately by stopping a backend while traffic flows, since that behaviour is the reason to adopt it and only proves itself when a backend disappears.

Frequently asked questions

Official sources

1. License: Apache-2.0
2. Project website
3. README
4. Releases
5. thushan/olla on GitHub