LLMIO: a Go gateway for weighted LLM routing and per-request cost logs

What problem LLMIO solves, and for whom

Anyone wiring an LLM feature into a product eventually accumulates the same mess: one key for OpenAI, another for Anthropic, a Gemini key somewhere else, each with its own base URL, its own auth header and its own response shape. LLMIO collapses that into a single service. It exposes OpenAI Chat Completions, OpenAI Responses, Anthropic Messages and Gemini native generateContent behind one process, and it accepts the auth header each family expects, so an existing client library keeps working after you point it at the gateway.

The audience is narrow but real. The README names the clients it was built around: openclaw, claude code, codex, gemini cli, cherry studio and open webui. These are tools that expect a provider-shaped endpoint and an API key. If you are the person who runs those tools locally or on a small server and wants to switch models without editing config files in five places, LLMIO targets you directly. The admin UI, the token and cost columns in the log view, and the model association screen all assume a human operator who needs to see what happened, not a platform team building an internal API product.

How the routing and logging actually work

The repository layout tells most of the story. main.go holds the HTTP server and routes, handler/ holds the REST handlers, service/ holds business logic and load balancing, middleware/ holds auth, rate limiting and streaming middleware, providers/ holds the per-vendor adapters, and balancers/ holds the weight and scheduling strategies. The README states there are two strategies in balancers/: random by weight and priority by weight. That is the core scheduling decision, and it is deliberately small. Routing can also be filtered by capability, since the README says you can route based on tool calling, structured output and multimodal support. So a request for a model can be pinned to associations that actually support the feature it needs.

Persistence is a single SQLite database at db/llmio.db, opened through github.com/glebarez/sqlite and GORM, both visible in go.mod. There is no external database in the dependency list. Every request is recorded with a TraceID, a latency breakdown (proxy, first-chunk and completion time), TPS, token usage split into input, cached and output, and an optional full input/output log. Cost is computed from per-million-token prices you configure per association, in CNY or USD, and shown in the log detail view.

Session tracking is the one feature worth calling out separately, because it is easy to miss. You pass session_id in any request body, which the README notes works through extra_body in the OpenAI SDK, and the gateway tags the log row with it. You can then filter by session_id in the admin UI or query GET /api/logs?session_id=. That turns a pile of per-request rows into something you can read as a conversation.

Installing LLMIO with Docker Compose and making a first call

The README recommends Docker Compose. The compose file in the repository mounts ./db into /app/db, publishes port 7070, and passes the token through the TOKEN environment variable. Replace the placeholder before starting, because TOKEN is described as required for public access.

services:
  llmio:
    image: atopos31/llmio:latest
    ports:
      - 7070:7070
    volumes:
      - ./db:/app/db
    environment:
      - GIN_MODE=release
      - TOKEN=<YOUR_TOKEN>
      - TZ=Asia/Shanghai

Start it, then open the admin UI on port 7070 to add providers, models and associations.

TOKEN=<YOUR_TOKEN> docker compose up -d

If you would rather not use Compose, the README gives the equivalent single command, with the same port, volume and environment variables.

docker run -d \
  --name llmio \
  -p 7070:7070 \
  -v $(pwd)/db:/app/db \
  -e GIN_MODE=release \
  -e TOKEN=<YOUR_TOKEN> \
  -e TZ=Asia/Shanghai \
  atopos31/llmio:latest

There is also a local run path: download the release package for your OS and arch (the README says version above 0.5.13), extract it, and start the binary. The service creates ./db/llmio.db in the current directory.

GIN_MODE=release TOKEN=<YOUR_TOKEN> ./llmio

Once it is up, the first real check is listing models through the OpenAI-compatible route. The README shows this exact curl with a Bearer header, and you should get a model list back rather than a 401.

curl -H "Authorization: Bearer YOUR_TOKEN" http://localhost:7070/openai/v1/models

For claude code or codex, point the client at the gateway by exporting the provider key variables the README lists. Note that the same token is used for all three in this example.

export OPENAI_API_KEY=<YOUR_TOKEN>
export ANTHROPIC_API_KEY=<YOUR_TOKEN>
export GEMINI_API_KEY=<YOUR_TOKEN>

The single-node SQLite design is the main limitation

A pure Go SQLite file is a reasonable default for a personal or small-team gateway. It is also the constraint that decides where LLMIO does not belong. Config, associations, request logs and the cost columns all live in db/llmio.db on one host. Two containers sharing that file over a network filesystem is not something the README describes, and the dependency list contains no coordination layer for it. If you need several gateway instances behind a load balancer with a shared view of logs and pricing, this project does not offer that, and nothing in the repository suggests it is planned.

Log growth is the second-order problem. Every request writes a row with token counts, latency fields and optional full input/output payloads. The README exposes exactly one maintenance knob for this: DB_VACUUM, which runs SQLite VACUUM on startup when set to true. There is no documented retention policy, no scheduled pruning and no log rotation described. Full IO logging in particular will inflate the file quickly, and the README does not document a per-association or per-model switch for it. If you enable full logging on a busy gateway, plan for the VACUUM flag and for manual inspection of the file size.

The README also does not document rollback, downgrade, or a migration procedure between releases. Releases exist (v0.8.13, v0.8.12, v0.8.11), and the Dockerfile builds with GORM, which runs schema migration at startup, but the project's stated upgrade story is effectively: pull the new image and start it. Treat that as an unknown until you have tested it against a copy of your database.

LLMIO compared with a hosted LLM gateway

The obvious alternative class is a hosted gateway, where the vendor runs the routing layer and you get a dashboard, a shared log store and no server to patch. The difference in approach is where state lives. With LLMIO, the routing table, the weights, the per-million-token prices and every log row sit in a SQLite file you own, on a machine you own. That means no request metadata leaves your infrastructure, and it also means you are the one backing up db/llmio.db, watching its size, and restarting the process after an upgrade.

A second difference is the auth model. LLMIO authenticates callers with a single TOKEN value, used as a Bearer token, an x-api-key, or an x-goog-api-key depending on the route. That is simple to operate and simple to leak. A hosted gateway typically gives you per-key issuance, per-key quotas and revocation. If several people or services share your LLMIO instance, they share one secret, and the README documents no per-caller key management.

The trade is therefore not features against features. It is operational ownership against operational convenience. LLMIO wins if you were already going to run a small server, want the cost numbers computed from your own price table, and prefer that the request log never leaves the box.

Maintenance, release cadence and the MIT licence

The last push to the repository was on 2026-08-16, and the most recent release, v0.8.13, was tagged the same day. Before that, v0.8.12 landed on 2026-05-29 and v0.8.11 on 2026-05-20, so the gap between the May releases and the August release is roughly three months. That is a real cadence, not a dead project, but it is also not a project with a stated support window. The README does not describe a release policy, a deprecation process, or which versions receive fixes.

Upgrade cost is dominated by the schema question. The Dockerfile builds a single static binary with CGO_ENABLED=0, and GORM is in the dependency list, so the startup path includes model migration. The README does not document what happens when a migration meets an older database, and it does not document a downgrade path. Before moving a production instance from one release to the next, copy db/llmio.db and test the new binary against the copy.

The licence is MIT, which is permissive and imposes no copyleft obligation on your own code. The repository's LICENSE file is the authoritative text; nothing here is legal advice, and if you redistribute the binary or embed it in a product, read that file rather than this paragraph. The Go module path is github.com/atopos31/llmio, and go.mod pins Go 1.26.6, so building from source requires a toolchain at least that new.

Editorial conclusion

Adopt LLMIO if you already run several provider keys and want one endpoint, weight-based scheduling and a costed log per request, and you are comfortable with a single Go binary plus a SQLite file under ./db as the whole persistence story. Do not adopt it if you need multi-node coordination, a documented upgrade or rollback procedure, or a hosted control plane; the README describes neither clustering nor rollback. Verify first that your client can send the auth header the route expects (Bearer, x-api-key or x-goog-api-key), and confirm whether the provider-specific routes under /openai, /anthropic and /gemini cover the features you depend on, since /v1/* is kept only for compatibility.

Frequently asked questions

Official sources

1. atopos31/llmio on GitHub
2. Issues
3. License: MIT
4. README
5. Releases