OrcaRouter Lite: a self-hosted LLM router that falls back to a hosted account

The gap OrcaRouter Lite is trying to fill

Most teams that call more than one model provider end up writing the same glue: a mapping from a logical request to a concrete model name, a retry path when one provider returns a 429 or a 500, and some way to know afterwards which model actually answered. OrcaRouter Lite packages that glue as a server. You run it, point your existing OpenAI SDK at http://localhost:8000/v1, and send model="auto" instead of a hardcoded model name. The README describes the routing rule as picking the cheapest model in your configured providers that meets the request's capability requirements, where those requirements are tools, vision and JSON mode.

The audience is narrower than the tagline suggests. The project describes itself as the open-source single-workspace edition of OrcaRouter, and the README is explicit that the hosted product is the place to go for more advanced routing. So this is aimed at a team running one workspace for one product, not at a platform group that needs to hand out isolated routers to many internal customers. The README's comparison table frames the positioning against LiteLLM (a library, not a server), OpenRouter (closed-source hosted), and Ollama (local-only). The sentence the project uses for itself is self-hosted server with a managed fallback.

What happens between an inbound request and a provider call

The architecture visible in the README is a translation layer in front of one routing pipeline. Three inbound protocols are accepted: the OpenAI chat completions shape, the Anthropic wire format, and the Gemini format via the google-genai SDK. Requests are translated at the edge, and from there model="auto", the cross-provider prompt cache, the routing strategies and the analytics dashboard all behave the same regardless of which protocol the client spoke. That detail matters more than it first appears. It means a Claude Code session and a Python script using the OpenAI SDK can share a cache and land in the same analytics view, because they converge before the routing decision is made.

The routing decision itself is capability-driven rather than rule-driven. The README contrasts this with manual routing rules and with if x: cost optimization branches in application code. For a vision request, the example shows an image_url content block and states the request routes to the cheapest vision-capable model the configured keys cover. The resolved choice is reported back through the x-orca-resolved-model response header, which is the mechanism that makes the abstraction auditable at the call site: you can log or display which model was used without querying the dashboard. Streaming follows the OpenAI SSE convention with data: framing and a terminal [DONE] sentinel, so an SDK that already streams from OpenAI does not need changes.

Getting it running: two paths and the keys that separate them

Path A is the self-hosted route. The README gives these commands: clone the repository, cd into it, copy .env.example to .env, add at least one provider key such as OPENAI_API_KEY=sk-..., then run docker compose up. The startup log is documented as printing a generated key in the form sk-orca-abc123..., and that key is what clients send as the bearer token against http://localhost:8000/v1. The dashboard is served at the root path, http://localhost:8000/, and the README notes it is available on Path A only. It also lists providers, routing, analytics and keys as dashboard areas.

Path B skips the clone entirely. You register at the hosted site, copy an sk-orca-* key, and use https://api.orcarouter.ai/v1 as the base URL. The README states an account is required and that hosted handles routing, billing and the long tail of providers, billed per-token on your OrcaRouter account. The two paths are not exclusive. Setting ORCAROUTER_API_KEY in the .env file of a self-hosted instance adds hosted as one more provider in the routing chain, which the README frames as try-before-you-buy, local logging, and failover when local providers fail. The README does not document which configuration keys control routing strategy selection or the prompt cache; only the environment variables above appear in the material.

The Anthropic and Gemini endpoints are the least obvious feature

The README includes integration guides for Claude Code and the google-genai SDK, and the setup for the first is a base URL without the /v1 suffix: export ANTHROPIC_BASE_URL=http://localhost:8000 alongside ANTHROPIC_API_KEY. That is a deliberate detail. Pointing Claude Code at a router that only speaks OpenAI's format would require a translation shim; here the Anthropic format is a first-class inbound protocol. The Gemini example uses HttpOptions(base_url="http://localhost:8000") with the google-genai client.

What this buys you is a single place where spend and routing policy live, even when different parts of an organisation have standardised on different SDKs. The trade-off is that three inbound protocols mean three surfaces where a client-side quirk can appear, and the README does not describe how protocol-specific fields that have no equivalent in the internal representation are handled. If your application depends on a provider-specific parameter that only exists in one of the three formats, that is a question for the integration guides rather than the README.

Where Lite stops being the right tool

The single-workspace constraint is the sharpest boundary. If you need per-team keys with independent budgets and isolated logs, the README's own framing points you at the hosted product instead, and the Lite edition does not advertise that capability. A second limitation is the dependency direction. Setting ORCAROUTER_API_KEY makes an external account part of your routing chain. For the failover use case that is the point, but it also means a deployment that was self-hosted for data-residency reasons now has one provider path that leaves your infrastructure. Whether that is acceptable depends on which requests can be routed there, and the README does not describe a policy for excluding specific requests from the hosted provider.

A third gap is documentation depth around the routing decision. The README states that model="auto" picks the cheapest capable model, and it names tools, vision and JSON mode as the capability inputs, but it does not explain how capability metadata is maintained per model, how ties are broken, or what happens when no configured provider satisfies the request. Those are exactly the cases where a router's behaviour matters most, and the material available does not answer them. Treat the absence as a genuine unknown rather than assuming a sensible default.

How it differs from LiteLLM in practice

LiteLLM appears in the project's own comparison table as a library, and that is the real distinction. A library is embedded in your process: you import it, you control the call path, and the routing logic runs wherever your application runs. OrcaRouter Lite is a server you deploy separately, which means your application talks HTTP to it and the router can be upgraded, restarted or reconfigured without redeploying the application. The README also lists a built-in dashboard as something Lite has and the LiteLLM library does not, and it lists model="auto" as a feature the library lacks.

That difference cuts both ways. A separate server is another process to run, monitor and secure, and the README's claim of no Postgres and no Redis required keeps that footprint small but does not make it zero. In exchange you get a stable endpoint that multiple applications and multiple SDKs can share, plus the header that tells each caller which model answered. If your routing needs are confined to a single Python service and you want the decision logic inside your own test suite, a library is the simpler shape. If you have several callers in several languages, the server shape is the one that avoids reimplementing the same policy three times.

Version, maintenance and licence

The repository shows a single release, v0.1.0, dated 2026-05-11, with the most recent push to main on 2026-09-04. That is a young project by any measure: one tagged release, and the README badge claims 403 passing tests, which is a statement from the project rather than an independent measurement. The README also documents twelve translated versions of the file, which is a maintenance commitment in itself, since every change to the English text creates eleven potential stale translations.

The licence is MIT, which permits commercial use, modification and redistribution provided the copyright notice and permission notice are included. That is permissive enough for embedding in a product. Two things to check before relying on it: whether the hosted service's terms interact with your use of the ORCAROUTER_API_KEY path, since that path routes traffic through a third party, and whether the generated sk-orca-* key printed at first startup is persisted somewhere you can rotate. The README shows the key being printed but does not describe rotation. On upgrades, a project at v0.1.0 with no documented migration notes means you should pin a specific commit or image tag rather than tracking main, and read the release notes before moving.

Editorial conclusion

OrcaRouter Lite is a reasonable fit for a team that already pays for provider keys and wants one OpenAI-compatible endpoint in front of them, plus a dashboard and a model="auto" mode that picks the cheapest capable model. It is the wrong tool if you need multi-tenant isolation, because the README states this is a single-workspace edition, or if you want routing decisions you can fully inspect, because the capability matching behind model="auto" is not described in the material. Before adopting, verify three things: that the providers you actually use are covered by the model catalog, that the resolved model reported in the x-orca-resolved-model response header matches what you expect for your tool, vision and JSON-mode requests, and whether your deployment needs to survive the loss of the hosted fallback, since Path A with ORCAROUTER_API_KEY set makes an external account part of your routing chain.

Official sources

1. Continuum-AI-Corp/OrcaRouter-Lite on GitHub
2. License: MIT
3. Project website
4. README
5. Releases