llmaz: a Kubernetes inference platform for large language models

What llmaz actually solves for a Kubernetes team

Serving an open model on Kubernetes is not hard once. It is hard the fifth time. Each backend (vLLM, Text-Generation-Inference, SGLang, llama.cpp, TensorRT-LLM) has its own container image, its own flags, its own way of pointing at model weights, and its own readiness behaviour. Every new model means another hand-written Deployment, Service and resource request that drifts from the previous one. llmaz is an operator that collapses that work into two custom resources: an OpenModel that describes where the weights come from and what hardware they need, and a Playground that says how many replicas to run. The README describes the project as an inference platform for large language models on Kubernetes, and the repository topics list huggingface, modelscope, ollama, llamacpp, sglang, vllm and text-generation-inference, so the intended audience is platform engineers who already have a cluster and want the model-serving layer to be declarative rather than scripted. It is not aimed at someone who wants to call a hosted API, and it is not a training tool: the roadmap puts model training and fine tuning in the long term.

How the OpenModel and Playground resources fit together

The split between the two resources is the design decision worth understanding. An OpenModel is the model-side declaration. In the README example it carries familyName: opt, a source block with modelHub.modelID: facebook/opt-125m, and an inferenceConfig.flavors list where each flavor sets limits such as nvidia.com/gpu: 1. That flavors list is the hardware contract: it is where you say which GPU type or count a given model should be scheduled against. A Playground is the serving-side declaration. It carries replicas and a modelClaim that names the OpenModel. Separating them means the same model definition can back more than one Playground, and a Playground can be resized without editing the model. The operator then watches both resources and reconciles the underlying workload. The README states that by default llmaz creates a ClusterIP service named like <service>-lb for load balancing, which is what you port-forward to for a local test. Model loading is handled by the platform: the README says llmaz supports HuggingFace, ModelScope and object stores and will automatically handle the model loading, requiring no effort from users. The pyproject.toml in the repository lists huggingface-hub, modelscope and omnistore as Python dependencies, which matches that claim about where weights are fetched from. Distributed inference is built on LWS (sigs.k8s.io/lws v0.6.2 in go.mod), and the README describes multi-host and homogeneous xPyD support from day 0, with heterogeneous xPyD as future work. The project is written in Go and built on controller-runtime, so anyone debugging it will be reading Go controller code.

Installing llmaz and serving facebook/opt-125m

The README does not inline the install commands. It points at the installation guide in the repository, at site/content/en/docs/getting-started/installation.md, and that is the page to follow. What the README does give is the deployment path once the operator is running. You apply a Model and a Playground, and the operator does the rest. If the weights need a Hugging Face token, the README says to create the secret first:

kubectl create secret generic modelhub-secret --from-literal=HF_TOKEN=<your token>

Then apply an OpenModel. This is the README's toy example for facebook/opt-125m, with a flavor that requests one NVIDIA GPU:

apiVersion: llmaz.io/v1alpha1
kind: OpenModel
metadata:
  name: opt-125m
spec:
  familyName: opt
  source:
    modelHub:
      modelID: facebook/opt-125m
  inferenceConfig:
    flavors:
      - name: default # Configure GPU type
        limits:
          nvidia.com/gpu: 1

Next, a Playground that claims that model and runs one replica:

apiVersion: inference.llmaz.io/v1alpha1
kind: Playground
metadata:
  name: opt-125m
spec:
  replicas: 1
  modelClaim:
    modelName: opt-125m

To verify, port-forward the load-balancing service the operator created. The README gives this command, and the service name follows the <service>-lb convention:

kubectl port-forward svc/opt-125m-lb 8080:8080

Two curl calls confirm the service is live. The first lists registered models, and the second sends a completion request using the same JSON body shape as the OpenAI API:

curl http://localhost:8080/v1/models

curl http://localhost:8080/v1/completions \
-H "Content-Type: application/json" \
-d '{
    "model": "opt-125m",
    "prompt": "San Francisco is a",
    "max_tokens": 10,
    "temperature": 0
}'

If you have no GPU, the README points at docs/examples/llamacpp/README.md for a CPU path. Note the API group difference: the OpenModel uses llmaz.io/v1alpha1 and the Playground uses inference.llmaz.io/v1alpha1. Mixing those up is an easy first mistake.

Alpha API, version skew, and the cases where llmaz is the wrong tool

The README carries an alpha stability badge and states plainly that llmaz is alpha now, so the API may change before graduating to Beta. Treat every field in the OpenModel and Playground specs as movable. The most recent release listed is v0.1.4 from 2025-06-10, with v0.1.3 and v0.1.2 in April 2025, so the surface being stabilized is still small. The last push to the default branch was on 2026-08-27, which tells you work continues, but the release cadence is not the same thing as API stability. Two concrete limits follow. First, the README documents no rollback or downgrade procedure for the CRDs, so if a spec field changes between versions you are on your own for migrating existing objects. Second, the README describes heterogeneous xPyD as something to be implemented in the future, so if you need to split prefill and decode across mixed hardware today, the documented answer is homogeneous multi-host only. llmaz is also the wrong tool when the problem is one model and one Deployment. A hand-written vLLM Deployment with a Service in front of it is a few dozen lines, has no operator to upgrade, and no CRD to reconcile. The operator earns its place when you are managing several models, several backends, or several teams on shared hardware. If you are not on Kubernetes at all, nothing here applies.

llmaz compared with KServe-style model servers

The closest familiar point of reference is KServe, which the related searches surface as KFServing. Both are Kubernetes control planes for model serving, and the difference is in what they assume about the model. KServe grew out of a general model-serving stack with predictor, transformer and explainer stages, and its InferenceService covers everything from scikit-learn to PyTorch to LLMs. llmaz starts from the LLM case and works outward. Its resources are named for LLM concepts (OpenModel, Playground, flavors for GPU selection, modelClaim for binding), and its integrations are LLM-specific: vLLM, Text-Generation-Inference, SGLang, llama.cpp, TensorRT-LLM for backends, and Open WebUI for a built-in chat interface. The README also lists an Envoy AI Gateway integration for token-based rate limiting and model routing, and HPA scaling driven by LLM-based metrics, which are concerns that only appear once you are running generative models in production. That focus is a real trade-off. If your cluster serves a mix of classical models and LLMs, KServe covers both under one API and llmaz does not. If your world is entirely LLMs and you want the GPU flavor, backend and model-hub handling to be first-class fields rather than parts of a generic predictor spec, llmaz's narrower model is easier to reason about.

Maintenance cost, upgrades and the Apache-2.0 licence

Running llmaz means running a Kubernetes operator, and that has a cost beyond the initial install. The repository ships a chart/ directory and an index.yaml, so a Helm-based install is part of the project layout, and the Dockerfile builds a single manager binary on a distroless static base running as user 65532:65532. Upgrades therefore mean replacing that manager image and, when the CRDs change in an alpha project, checking that your existing OpenModel and Playground objects still validate. The README does not document a CRD conversion or migration story, so budget for reading release notes before each bump. The Go module requires Go 1.24.0 with toolchain go1.24.4 and depends on k8s.io modules at v0.33.5 and controller-runtime v0.21.0, which means the operator tracks recent Kubernetes client libraries; older clusters should be checked against those versions before you start. The project is licensed Apache-2.0, which is a permissive licence that permits commercial use and modification, and the pyproject.toml repeats the same identifier for the Python side. This is a description of the licence text the repository declares, not legal advice; if you redistribute llmaz or embed it in a product, read the LICENSE file and your own counsel's guidance on attribution and notice requirements.

Editorial conclusion

Adopt llmaz if you already run Kubernetes, want to serve several open models through one control plane, and can accept an alpha API plus manual upgrades of a Go operator. Do not adopt it if a single vLLM Deployment already meets your needs, or if you have no cluster to run it on. Before committing, read the installation page in the repository, confirm which backend images your cluster can pull, and check whether the Playground spec still matches the examples in docs/examples.

Frequently asked questions

Official sources

1. InftyAI/llmaz on GitHub
2. License: Apache-2.0
3. Project website
4. README
5. Releases