AIBrix: a Kubernetes control plane for LLM inference, installed with three kubectl applies

Eight named components, not a serving engine

AIBrix is presented as a Kubernetes control plane for production LLM inference: you bring the engine (vLLM, SGLang, or TensorRT-LLM) and AIBrix routes requests, scales on inference metrics, and manages models, LoRA adapters, and KV cache. The repository sits under the vllm-project organisation. What it ships is a named set of components. High-density LoRA management covers lightweight low-rank adaptations. An LLM gateway and routing layer directs traffic across models and replicas. An autoscaler scales replicas on engine metrics such as queue depth, KV cache, and tokens. A unified AI runtime works as a sidecar for metric standardization, model downloading, and management. Distributed inference spreads work across nodes, a distributed KV cache enables high-capacity cross-engine reuse, and heterogeneous GPU serving, marked experimental, mixes GPU types while still targeting cost and SLO.

Nightly from the tree, or three pinned manifests for v0.7.0

Two install paths are given, and they differ in more than wording. The nightly path clones the repository and applies three directories: dependencies, CRDs, then components.

# Local Testing
git clone https://github.com/vllm-project/aibrix.git
cd aibrix

# Install nightly aibrix dependencies
kubectl apply -k config/dependency --server-side

# Install nightly AIBrix CRDs (separate from the operator so uninstalls don't wipe user CRs)
kubectl apply -k config/crd --server-side

# Install nightly aibrix components
kubectl apply -k config/default

The stable path pins a release and skips the clone entirely. The README snippet now sets `VERSION=v0.8.0`, the latest release, and says to change `VERSION` to install a different release; the same three applies for v0.7.0 look like this:

# Install component dependencies
kubectl apply -f "https://github.com/vllm-project/aibrix/releases/download/v0.7.0/aibrix-dependency-v0.7.0.yaml" --server-side

# Install AIBrix CRDs (separate from the operator so uninstalls don't wipe user CRs)
kubectl apply -f "https://github.com/vllm-project/aibrix/releases/download/v0.7.0/aibrix-core-crds-v0.7.0.yaml" --server-side

# Install aibrix components
kubectl apply -f "https://github.com/vllm-project/aibrix/releases/download/v0.7.0/aibrix-core-v0.7.0.yaml"

In both paths the first two applies carry `--server-side` and the last one does not. The release path carries a comment on the CRD step, that CRDs ship separately so uninstalling the operator does not delete user resources, and that comment is the most operationally useful line in the file. The three paths are kustomize directories under the top-level `config/`, which sits in the tree next to `deployment/` and `dist/`, so a nightly install follows the source while a pinned install does not depend on your clone at all.

The Makefile names four images and tags them by commit hash

The Makefile is where the shape of the deployment becomes concrete. Four images make up the set: controller-manager, gateway-plugins, runtime, and metadata-service, published under the aibrix registry namespace with a nightly tag. The default image tag is not a version number. IMAGE_TAG falls back to the output of `git rev-parse HEAD`, so a locally built image is identifiable by its commit. Two variables decide what goes inside those images: DOCKERFILE_PATH points at build/container, and KVCACHED_RUNTIME_BASE_IMAGE defaults to ghcr.io/ovg-project/kvcached-vllm:kvcached-v0.1.6-vllm-v0.30.0, which ties the runtime image to a KV caching base image maintained elsewhere. For local testing, ENVTEST_K8S_VERSION is fixed at 1.29.0, so the envtest assets match one Kubernetes minor version. The container tool defaults to docker, and the file notes podman as the substitution. One leftover sits in the file too, an IMG variable carrying a TODO to remove it once every target uses IMAGE_TAG.

go.mod pins Go 1.22.5 with a 1.22.6 toolchain directive

The module is `github.com/vllm-project/aibrix`, and the toolchain is pinned twice: `go 1.22.5` as the language directive, then `toolchain go1.22.6`, which makes a build fetch a specific patch release instead of using whatever Go is on the machine. The dependency list is where the component boundaries show up. `github.com/envoyproxy/go-control-plane v0.12.0` and `github.com/grpc-ecosystem/grpc-gateway/v2 v2.20.0` sit under the gateway side, which is consistent with gateway-plugins being a separate image. `github.com/redis/go-redis/v9 v9.6.1` and the in-memory `github.com/alicebob/miniredis/v2 v2.37.0` cover Redis-backed work and its test double. `github.com/ray-project/kuberay/ray-operator v1.2.1` and `github.com/kubewharf/godel-scheduler-api` cover external schedulers. `github.com/openai/openai-go/v3 v3.32.0`, `github.com/pkoukk/tiktoken-go`, and `github.com/pkoukk/tiktoken-go-loader` handle client calls and tokenization, while metrics come from `github.com/prometheus/client_golang v1.20.2` and traces from the OpenTelemetry modules at v1.28.0. The remainder reads like a service rather than a library: `github.com/bytedance/sonic v1.15.2` and `github.com/tidwall/gjson` for JSON handling, `github.com/vmihailenco/msgpack/v5`, `github.com/pebbe/zmq4 v1.4.0`, `github.com/coreos/go-oidc/v3 v3.11.0` alongside `github.com/open-policy-agent/cert-controller v0.12.0`, and `github.com/onsi/ginkgo/v2 v2.20.1` with `github.com/onsi/gomega v1.35.1` and `github.com/stretchr/testify v1.10.0` for tests.

samples/ is the API surface the README does not index

Twenty directories sit under samples/, and together they are the closest thing to an API reference in the repository: adapter, ai-gateway-integration, autoscaling, batch, deepseek-r1, disaggregation, distributed, external-replica-router, heterogeneous, jev, kvcache, modelclaim, modelwarmup, multimodality, orchestration, quickstart, semantic-router, vllm-ascend, vllm_omni, and volcano-engine. Read them against the component list and the mapping is direct. kvcache is the distributed KV cache, autoscaling is the app-tailored autoscaler, ai-gateway-integration and semantic-router are the routing side, heterogeneous and vllm-ascend are the mixed-GPU story. One names a stack rather than a feature: volcano-engine. The Ray operator in go.mod matches the README note that KubeRay is included in the dependency install and is only required for Ray-based multi-node inference. samples/quickstart/ is the one to open first if you want the smallest example, and samples/deepseek-r1/ pairs with samples/disaggregation/ if you are chasing that model.

Empty architecture section, and no upgrade or rollback path

Two gaps decide how far the repository alone gets you. The README Architecture section is short: requests hit an OpenAI-compatible gateway and go straight to an engine pod, while Kubernetes controllers handle model lifecycle, warmup, multi-node topology, autoscaling, and observability. The fuller picture lives in the architecture guide on the documentation site. The second gap is operational: the README quick start does not describe upgrading between v0.x releases, undoing a failed apply, or removing an installation. The install comments hint at why that matters. Both paths apply CRDs as a step separate from the operator, and the stated reason is that CRDs ship separately so uninstalling the operator does not delete user resources. Read that as a warning rather than a nicety: an operator that owns its CRDs can take custom resources with it when it is deleted. With no rollback recipe in the quick start, taking a snapshot before each apply is a sensible precaution.

Minor versions every few months, Apache 2.0 throughout

Releases run on a fairly steady minor-version clock. v0.1.0 landed on 2024-11-13, v0.2.0 on 2025-02-19 and v0.2.1 on 2025-03-09, v0.3.0 on 2025-05-21, v0.4.0 on 2025-08-05, v0.5.0 on 2025-11-10, v0.6.0 on 2026-03-03, v0.7.0 on 2026-06-18, and v0.8.0 on 2026-10-13. The last push to main was on 2026-11-02, so the tree runs ahead of the newest tag and the nightly configuration under `config/` is where that difference shows up. Licensing is Apache 2.0, and issues go to the repository's own tracker.

Editorial conclusion

AIBrix fits a team already running vLLM, SGLang or TensorRT-LLM on Kubernetes that needs routing, autoscaling or KV cache reuse as separate components rather than one serving engine, and that is willing to read the documentation site rather than the README. It does not fit anyone choosing an inference stack from a single file, because the README architecture section is two sentences pointing to the architecture guide and the quick start has no upgrade or rollback recipe. Verify four things first: that your Go toolchain resolves 1.22.6 through the toolchain directive, that you apply `config/crd` separately from the operator so an uninstall cannot wipe user CRs, that the container tool is docker or that you set CONTAINER_TOOL yourself, and that the release manifest you pinned is v0.8.0 or another tagged release rather than the moving nightly configuration under `config/`.

Frequently asked questions

Official sources

1. Official README
2. Project repository
3. Release notes