LLM Gateway: A Self-Hostable Proxy That Puts OpenAI, Anthropic and Vertex Behind One Endpoint

The problem is key sprawl, not inference

Every team that calls more than one model provider ends up rebuilding the same small layer. Provider SDKs differ in authentication headers, retry semantics and response envelopes. Spend is scattered across billing dashboards that do not share a schema. When someone asks which model produced a slow response last Tuesday, the answer lives in application logs rather than anywhere queryable. LLM Gateway targets that layer specifically. The README describes it as middleware between your applications and various LLM providers, with four stated jobs: route requests, manage provider API keys in one place, track token usage and cost, and analyze performance metrics. It is aimed at platform or backend engineers who own the integration surface, not at end users. The repository layout confirms the scope: apps/gateway handles routing, apps/api is a Hono backend, apps/ui is a Next.js dashboard, and packages/models holds model and provider definitions. There is also apps/playground, described as a consumer AI chat app, and apps/airside, a self-serve provider portal. Those last two are worth noting because they signal the project ships more than a proxy, which affects how much of the codebase you inherit when you self-host.

One OpenAI-shaped endpoint in front of several providers

The mechanism the README makes explicit is format translation. The feature list claims a unified API interface compatible with the OpenAI API format, and the usage example posts to /v1/chat/completions with a model field and a messages array, exactly the shape an OpenAI client already sends. That is the migration path: existing code keeps its request body and swaps the base URL and bearer token. What the README does not spell out is the routing decision itself. There is no documented rule for how a model string maps to a provider, beyond the existence of packages/models as the place where model and provider definitions live. In practice that means the mapping is data, not code, and you should read that package before assuming a given model name resolves the way you expect. The analytics side is similarly described at the level of intent: track requests, tokens used, response times and costs, and compare models on performance and cost-effectiveness. The README states these capabilities but does not give the metric definitions, the aggregation window or the storage schema. The folder structure suggests Drizzle ORM migrations under packages/db, so the ledger is a relational store you can query directly, but the exact tables are not in the supplied material. Treat the dashboard as the documented interface and the database as an undocumented one.

Running the unified container, and the volume trap

The README gives two self-hosted paths. The scripted one sets two secrets and calls a wrapper: export LLM_GATEWAY_SECRET and GATEWAY_API_KEY_HASH_SECRET, each generated with openssl rand -base64 32, then run ./scripts/run-unified-container.sh. The manual path creates two named volumes, llmgateway_postgres and llmgateway_redis, and starts ghcr.io/theopenco/llmgateway-unified:latest with seven published ports: 3002, 3003, 3005, 3006, 3007, 4001 and 4002. Two environment variables are passed in that example, AUTH_SECRET and GATEWAY_API_KEY_HASH_SECRET. The README carries an explicit warning here, and it is the most concrete operational detail in the document: do not bind-mount a host directory to /var/lib/postgresql/data, because PostgreSQL initialization inside the container needs to set permissions on that directory and that can fail depending on the host filesystem and ownership. Use Docker-managed volumes instead. That is a real constraint, not a stylistic preference, and it rules out the common pattern of keeping Postgres data in a visible host folder for easy backup. If your backup tooling assumes a bind mount, you will need to change it to a volume-aware approach before this runs reliably. For development the README prescribes pnpm i && pnpm run setup, which installs dependencies, starts Docker services, syncs the database schema and seeds initial data, followed by pnpm dev and pnpm build. WSL2 users are told to run Docker Desktop with WSL integration enabled. The hosted option is also documented: create an account at llmgateway.io and get an API key, then call https://api.llmgateway.io/v1/chat/completions.

The licence split is the real architectural boundary

The README states a dual licence. Core functionality is AGPLv3. Commercial features in the ee/ directory require an Enterprise licence, and multi-organization administration requires a white-label licence. The enterprise list is given as advanced billing and subscription management, extended data retention (unlimited versus 30 days), custom provider key configurations, team and organization management, and priority support, with the README adding and more to be defined. Two things follow. First, the 30-day retention figure is a property of the open-source deployment, so if your compliance posture requires keeping prompt and response metadata for a year, that is a commercial conversation, not a configuration flag. Second, the phrase and more to be defined means the enterprise boundary is not frozen. A feature you build against today could move behind ee/ in a later release, and the release cadence visible in the material is weekly (v1.14.0, v1.15.0 and v1.16.0 land seven days apart), so the surface changes often. The AGPLv3 half also matters for how you deploy: if you modify the core and expose it to users over a network, the licence's source-availability terms apply to your modified version. I am not a lawyer and this is not legal advice; the point is that the licence choice is a deployment decision you should settle before writing integration code, not after. Note also that the repository metadata reports the licence as NOASSERTION, which means the automated classifier could not resolve the SPDX identifier from the files, even though the README names AGPLv3 and points at LICENSE and ee/LICENSE. Read both files yourself.

Where a single gateway becomes the wrong shape

A gateway is a hop, and hops fail. The README does not describe health checking, failover between providers, or what happens to an in-flight request when an upstream provider returns a 429. The topics list mentions guardrails and rate-limiting, so those concerns are in scope for the project, but the supplied material does not document their semantics. That gap matters most in the failure mode this architecture invites: if the gateway is down, every application that was pointed at it is down, including the ones that only ever called one provider and did not need routing at all. The single-container deployment concentrates this further. Seven ports and two datastores in one image is convenient to start and awkward to scale selectively; you cannot give the gateway more replicas without also thinking about the Postgres and Redis volumes it carries. The other case where this is the wrong tool is a team with exactly one provider and no cost-attribution requirement. The gateway adds a network hop, a database, a Redis instance and a licence question, and returns analytics nobody has asked for yet. The README's own framing, middleware between your applications and various LLM providers, sets the bar: the value appears when there is more than one provider to reconcile, or when the usage ledger is itself a deliverable.

Compared with calling the provider SDKs directly

The alternative most teams actually weigh is not another gateway but the vendor SDKs themselves: the OpenAI client, the Anthropic client, and the Google Vertex client, each configured in application code with its own key and its own retry policy. That approach has no extra hop and no extra datastore, and it fails in a specific, predictable way: every application reimplements provider selection, and there is no single place where token spend is aggregated. LLM Gateway's difference is that it moves that logic out of the application and behind an HTTP boundary that speaks the OpenAI format, so a service that only knows how to call /v1/chat/completions can be pointed at Anthropic without a code change. The trade is that provider-specific parameters and response fields have to survive the translation, and the README does not document how faithfully they do. A second alternative is the hosted tier of LLM Gateway itself, which the README presents as the no-setup option. That choice is about data residency and control rather than features: self-hosting gives you the Postgres volume, the hosted version gives you someone else's operations. The README does not compare the two on limits, pricing or retention, so that decision needs information from outside this document.

Upgrade cost and what to check before you commit

Weekly releases with a dual-licence boundary produce a specific maintenance profile. You are not just pulling a new image; you are re-reading ee/LICENSE to see whether the line moved, and re-checking packages/db migrations against your Postgres volume, because a schema sync is part of the documented setup flow. The unified image bundles Postgres and Redis, so an upgrade means the datastores move with the application, which is simpler than coordinating separate upgrades and less flexible when you want to pin one component. Budget for that review cadence rather than assuming a drop-in image swap. The concrete things to verify first are all inspectable in the repository: whether the providers you depend on are defined in packages/models, what the AGPLv3 and ee/LICENSE files actually say, and which of the seven published ports your environment will expose. Start with the README's own smoke test, a POST to /v1/chat/completions with model set to gpt-4o, against your self-hosted instance rather than the hosted one, so you learn whether routing and key storage work before you migrate any production traffic.

Editorial conclusion

Adopt LLM Gateway if you already call two or more providers and want one OpenAI-shaped endpoint plus a usage ledger you control, and if AGPLv3 obligations are acceptable for the code you deploy. Do not adopt it if you need unlimited retention of request logs or multi-organization administration without a commercial agreement, because the README places both in the ee/ directory under a separate enterprise licence. Before committing, verify three things against your own deployment: that every provider you need is present in packages/models, that the ports 3002, 3003, 3005, 3006, 3007, 4001 and 4002 fit your network policy, and that the model string you send (the README example uses gpt-4o) resolves to a provider key you have already stored in the gateway.

Official sources

1. Issues
2. Project website
3. README
4. Releases
5. theopenco/llmgateway on GitHub