CoAI.Dev: A Multi-Tenant LLM Gateway With Billing, Channel Routing and Model Caching

The gap CoAI.Dev is trying to fill: gateway plus storefront in one binary

Most teams that sell or meter LLM access end up assembling two things: a proxy that normalizes provider APIs, and a separate application that handles accounts, quotas and payments. The README frames CoAI.Dev as a replacement for that combination, describing it as a one-stop business solution and comparing it directly to Next Web plus One API. That comparison is the clearest statement of intent in the material. The target user is not an individual developer calling an API. It is an operator who wants to run a site where users sign in, spend credits, pick models from a curated market, and where the operator can see channel health and per-request cost from an admin dashboard. The feature list backs this up: user management, subscription management, gift codes, redemption codes, price setting, SMTP configuration, custom site name and logo. Those are commerce features, not developer-tool features. If you are building an internal tool for a dozen engineers, most of this surface is dead weight you will still have to deploy and patch.

Channel priority, weight and retry: the routing mechanism worth understanding

The part of the system that carries the most engineering weight is channel management. According to the README, the project uses a self-written channel algorithm supporting multi-channel management, a priority setting that determines call order, and a weight setting that controls load-balancing probability among channels at the same priority. Retry on failure, model redirection, upstream hiding and channel status management are listed alongside those. Read together, that describes a two-level scheduler: priority groups define an ordered fallback chain, and weights distribute traffic inside each group. Model redirection is the piece that matters operationally. It means a request for one model name can be served by a different upstream model, which is how you keep a catalog stable when a provider deprecates something or when you want to route a cheap tier to a cheaper backend. Upstream hiding matters for a different reason: it lets the operator present a unified model list without exposing which vendor actually served the request. The README does not publish the retry policy details, so how many attempts a failed request gets, and whether a retry can double-bill, are things you have to check in the source or in the documentation before trusting the routing layer with paid traffic.

Billing models and the cache that decides whether a request costs anything

CoAI.Dev documents two billing modes: subscription and what the README calls elastic billing. Elastic billing is configurable per request, per token, or not at all, and supports anonymous calls plus a minimum request points check. That last item is a floor: a request that would otherwise cost almost nothing can still be charged a minimum. The model cache changes the economics again. The README states that when caching is enabled, a request with the same parameter hash that has been seen before returns the cached result directly, and that a cache hit is not billed. Cache enablement, cache duration and the number of stored results are configurable. This is a real design decision with a real consequence. A cache hit is free to the user and free of upstream cost to the operator, which is good for margin on repeated questions. It also means identical prompts return identical answers without touching the provider, so any workflow that depends on sampling variety will behave differently once caching is on. The README does not say how the parameter hash is computed or whether user identity is part of it. If it is not, one tenant's cached answer can be served to another, and that is the first thing to verify if you plan to enable caching on multi-tenant traffic.

Getting it running: deployment paths and the external services you still need

The repository points to a deployment guide at coai.dev/docs/deploy rather than embedding install steps in the README, so the exact command sequence is not in the supplied material and should be taken from that guide. What the README does establish is the dependency shape. File parsing is described as out of the box for all models, covering PDF, Docx, Pptx, Excel and image formats, with OCR support, but the README directs you to a separate project, CoAI.Dev Blob Service, for the storage side, listing S3, R2 and MinIO as supported targets and noting that blob service supports Vercel and Docker one-click deployment. Web search is built on SearXNG, the open-source meta search engine, with Google, Bing, DuckDuckGo, Yahoo, Wikipedia, Arxiv and Qwant named as sources, plus safe search mode, content truncation and image proxying. So a full deployment is not one container. It is the CoAI application, a blob service, and a SearXNG instance, plus whatever object storage you point the blob service at. Admin configuration covers SMTP for email, custom site name and logo, and a model market where the operator writes model descriptions and tags. The README also mentions quick upstream synchronization, which lets channel settings, model market and price settings be pulled from an upstream site and then modified.

Where the open repository ends and the commercial edition begins

This is the limitation that matters most for an adoption decision, and the README is unusually direct about it. A section titled CoAI.Dev Business lists features in the paid edition: TTS and STT, a plugin marketplace, a RAG knowledge base, more payment providers, more billing models, advanced order management, additional authentication methods including SMS and OAuth login, and model monitoring with channel health detection and automatic channel switching on fault alarms. The truncated README cuts off mid-sentence on that last point, so the full split is not visible here. The practical reading is that the open repository gives you the gateway, the chat interface, the admin console, the billing primitives and the channel routing, while the operational safety features you would want for production monitoring sit behind the commercial license. The README also marks a group of features as discontinued, including an AI project generator, batch article generation and an AI card function described as deprecated. Anyone evaluating the project from an older tutorial should check which of those still exist in v4.0.0. The release cadence is worth noting too: v3.10 shipped in March 2024, v3.11.1 in January 2025, and v4.0.0 in October 2025. That is a slow, infrequent release rhythm for a project with this much surface area, and it means upgrade notes deserve a careful read before you move between major versions.

How it differs from running a plain OpenAI-compatible proxy

A minimal proxy such as a single-purpose API forwarder does one thing: it accepts OpenAI-format requests and forwards them to a backend, usually with key rotation. CoAI.Dev accepts the same request format, and the README explicitly frames it as an OpenAI API distribution and proxy system, but it puts an account system, a credit ledger, a model catalog and a channel scheduler in front of the forward. The difference shows up in what you can answer. With a plain proxy you can tell whether a request succeeded. With CoAI.Dev you can, per the README, set prices, sell subscriptions, issue gift codes and redemption codes in batches, restrict one gift code per user, and bill per token or per request with a minimum floor. The trade is operational weight. A plain proxy is a small process you can read in an afternoon. CoAI.Dev is a TypeScript application with a React front end, a database, an admin console and two optional external services, and the README's own deployment guide is a separate document. If your requirement is only to hide provider keys from a client application, the full platform is the wrong shape. If your requirement is to run a metered service with multiple tenants and a public model catalog, the proxy alone leaves you building exactly the billing and account layer this project already ships.

Licence, upgrade cost and what to verify before you commit

The project is Apache-2.0, which the README describes as business-friendly for commercial secondary development and distribution, while noting that the licence terms still apply and that illegal use is not permitted. That is a permissive licence, and it is the reason a company can fork this and ship a modified version without publishing its changes. It says nothing about the commercial edition, which is a separate arrangement and not covered by the repository licence. On maintenance cost, the honest assessment from the material is that the operational burden is in the dependencies, not the application. SearXNG is a separate service with its own update cycle. The blob service is a separate repository with its own deployment targets. Object storage has to be provisioned and paid for. SMTP has to be configured for account email. Each of those is a thing that can break independently of a CoAI upgrade, and a major version bump like v3.11 to v4.0.0 lands on top of all of them. The specific things to confirm before adopting: whether the parameter hash behind the model cache includes tenant identity, how retries interact with billing, which features in the README belong to the commercial edition, and whether the deployment guide at coai.dev/docs/deploy matches the environment you actually run. Those four answers decide whether this is a platform you can operate or a demo you can only show.

Editorial conclusion

Adopt CoAI.Dev if you are standing up a paid or internal LLM service where one gateway must front many providers, track spend per user, and hand out subscription or redemption-code access without you writing the billing layer. Skip it if you only need a thin OpenAI-compatible proxy, or if you cannot operate SearXNG and the separate blob service, because file parsing and web search depend on those components. Before committing, verify the v4.0.0 deployment guide against your target environment, confirm which of the listed features are in the open repository rather than the commercial edition, and check how the model cache interacts with your privacy requirements, since a cache hit returns a stored result under the same request parameter hash.

Official sources

1. coaidev/coai on GitHub
2. License: Apache-2.0
3. Project website
4. README
5. Releases