LLMIO: a Go gateway for weighted LLM routing and per-request cost tracking

The problem LLMIO targets: many providers, one client config

Once a team uses more than one model vendor, every client starts carrying vendor-specific configuration. Claude Code expects an Anthropic-shaped endpoint, Codex expects an OpenAI-shaped one, and Gemini CLI expects Google's native paths. LLMIO's answer is to expose all three shapes from a single process and let the gateway decide which upstream key actually serves the request. The README lists openclaw, claude code, codex, gemini cli, cherry studio and open webui as the clients it is aimed at.

The second problem is cost visibility. When the same logical model is served by several providers at different prices, the question "what did this request cost and which key paid for it" is not answerable from the client side. LLMIO records each request with a TraceID, latency breakdown (proxy, first-chunk, completion), TPS, token usage split into input, cached and output, and a per-request cost computed from configurable per-million-token prices in CNY or USD. That accounting is the feature most gateways of this size treat as an add-on.

The audience is therefore narrow and specific: a platform or infrastructure engineer who already holds keys from two or more vendors, wants to spread load across them, and is willing to run one more container. It is not aimed at application developers who just want an SDK wrapper, and the repository does not present it as a hosted service.

How LLMIO routes a request: adapters, balancers and SQLite

The repository layout tells most of the story. handler/ holds REST handlers, service/ holds business logic, providers/ holds provider adapters, balancers/ holds the scheduling strategies, models/ holds GORM models and database initialisation, and middleware/ holds auth, rate limiting and streaming. main.go is the HTTP server entry and route table. A request therefore flows: middleware authenticates it, a handler parses the provider-shaped body, service/ resolves the logical model to a concrete provider association, balancers/ picks one, providers/ translates and forwards it, and the result is written back to the client while a log record is persisted.

The README names two scheduling strategies in balancers/: random by weight, and priority by weight. It also states that routing can be filtered by tool calling, structured output and multimodal capability, so an association that cannot serve a tool-calling request is not selected for one. Weights and per-token prices are configured per association in the admin UI, which the README describes as letting you configure multiple providers per model.

Persistence is deliberately minimal. The project uses a pure Go SQLite driver (github.com/glebarez/sqlite in go.mod) writing to db/llmio.db, and the README calls this "ready to use out of the box". There is a DB_VACUUM environment variable that runs a SQLite VACUUM on startup to reclaim space, which is an admission that the file grows with request logs and needs occasional compaction.

Installing LLMIO with Docker Compose and sending a first request

The README recommends Docker Compose. The compose file in the repository mounts ./db into the container and sets three environment variables: GIN_MODE to release, TOKEN to your secret, and TZ. TOKEN is required for public access because it guards both the console login and the API routes.

services:
  llmio:
    image: atopos31/llmio:latest
    ports:
      - 7070:7070
    volumes:
      - ./db:/app/db
    environment:
      - GIN_MODE=release
      - TOKEN=<YOUR_TOKEN>
      - TZ=Asia/Shanghai

Start it with the command the README gives:

 docker compose up -d

After startup the admin UI is served on the same port as the API. The README states the Web UI is at http://localhost:7070/ and that the SQLite file is created at ./db/llmio.db in the working directory. If you prefer a single container, the README also gives a docker run form with the same port, volume and environment variables.

Before the gateway can serve anything you have to add providers and model associations in the console; the README does not document a seed file or an import path for that configuration, so it is a manual step. Once an association exists, a request looks like any OpenAI call, with the token in the Authorization header:

curl -H "Authorization: Bearer YOUR_TOKEN" http://localhost:7070/openai/v1/models

The same token works across route families, but the header differs. Anthropic-shaped routes read x-api-key and Gemini native routes read x-goog-api-key. For CLI tools the README suggests exporting OPENAI_API_KEY, ANTHROPIC_API_KEY and GEMINI_API_KEY and pointing the tool at the gateway. It also notes that the /v1/* paths exist only for compatibility and that the provider-specific routes are preferred.

Where LLMIO is the wrong tool

The clearest limitation is the storage layer. Everything, configuration and request logs, lives in one SQLite file. There is no documented Postgres or MySQL option, no read replica, and no retention policy in the README beyond the DB_VACUUM flag. A team that needs to query historical spend from a warehouse, enforce per-tenant isolation, or keep logs for compliance review will find this design in the way rather than helpful. SQLite also serialises writes, so a high request rate combined with full IO logging is a plausible bottleneck; the README mentions "optional full IO logging" without stating a size limit or a sampling mechanism.

The second boundary is operational. The README documents rate-limit fallback and provider connectivity checks for fault isolation, but it does not document retry semantics, timeout budgets or what happens to an in-flight streaming response when the selected provider fails mid-stream. If your requirement is exactly-once delivery or deterministic failover with a documented SLA, the README does not give you the contract to build on.

Finally, consider the scope of the project. The last push was on 2026-08-16, and the release history shows v0.8.13 on the same day, v0.8.12 on 2026-05-29 and v0.8.11 on 2026-05-20. That is a pre-1.0 version number, and the README does not describe a migration path between releases or a compatibility guarantee for the SQLite schema. Treat upgrades as something to test against a copy of db/llmio.db rather than something to assume.

LLMIO compared with a full API management platform

APIPark is the closest thing in the related searches to a direct alternative, and the difference is scope rather than features. APIPark is positioned as an API management and developer-portal product: it handles API publication, consumer onboarding, keys and quotas as first-class objects. LLMIO does none of that. It has one TOKEN guarding the console and the API, and its unit of configuration is the model association (provider, weight, capability filters, per-token price), not the API consumer.

GPT-Load, the other name in the search list, is closer in spirit: a self-hosted relay that sits between clients and upstream providers. The distinction LLMIO makes is the combination of three things in one binary: multi-vendor protocol adapters (OpenAI, Anthropic, Gemini native), weighted selection across providers serving the same model, and per-request cost computed from prices you configure per association. A plain relay typically forwards and logs; it does not price the request or filter candidates by capability.

The honest summary is that LLMIO trades breadth for a small operational surface. One Go binary, one SQLite file, one port. If you need a portal, quotas per customer or a relational database, the other two categories fit better, and you should not expect LLMIO to grow into them based on what the README describes.

Licence, maintenance and the real cost of upgrading

LLMIO is MIT licensed, which permits commercial use, modification and redistribution provided the copyright notice and permission notice are retained. That is a permissive licence with no copyleft obligation on your own code. It says nothing about the upstream providers you route to, and their terms of service govern how you may use the keys you configure, not this project's licence. Nothing here is legal advice; if you redistribute a modified binary, read the LICENSE file in the repository rather than this summary.

The maintenance picture from the release history is a burst pattern rather than a steady cadence: v0.8.11 on 2026-05-20, v0.8.12 on 2026-05-29, then v0.8.13 on 2026-08-16, with the last push on the same day as the newest release. There is no published deprecation policy, and the README does not describe schema migrations. The practical upgrade procedure the repository supports is: stop the container, copy db/llmio.db somewhere safe, pull the new image, start it, and check the logs and the console before pointing production clients back at it. The DB_VACUUM variable is the only maintenance hook the README documents; setting it to true on startup reclaims space, which matters if you keep full IO logging enabled. Budget for that copy-and-verify step on every version bump, because nothing in the repository promises the old file will be read by the new binary.

Editorial conclusion

Adopt LLMIO if you run several providers for the same model and want weighted routing plus per-request cost and token logs without operating a database cluster. Skip it if you need Postgres, multi-tenant isolation or an audit trail you can query with SQL, because the README documents only a single SQLite file at ./db/llmio.db. Before rollout, set TOKEN, confirm the container writes to the mounted ./db volume, and check that your client's auth header matches the route family you call: Authorization: Bearer for /openai, x-api-key for /anthropic, x-goog-api-key for /gemini.

Frequently asked questions

Official sources

1. atopos31/llmio on GitHub
2. Issues
3. License: MIT
4. README
5. Releases