# KAITO: a Kubernetes operator that guesses how many GPUs your model needs > The project puts LLM inference, fine-tuning and retrieval behind three custom resources, and leans on Karpenter to buy the hardware on demand. **kaito-project/kaito** — Kubernetes AI Toolchain Operator - Repository: https://github.com/kaito-project/kaito - Website: https://kaito-project.github.io/kaito/docs/ - Stars: 1,023 · Forks: 187 - Language: Go - License: NOASSERTION - Published: 2026-10-07 · Updated: 2026-10-07 · Language: en - Canonical page: https://hysenlabs.com/projects/kaito-project-kaito ## Three custom resources instead of one deployment manifest KAITO describes itself as an operator suite for LLM inference, fine-tuning and retrieval augmented generation in a Kubernetes cluster. The design follows the standard custom resource pattern, but the CRDs are split along functional lines rather than merged into one. `Workspace` is the base unit. You hand it a GPU instance type and a HuggingFace model ID, and the controller estimates memory, works out how many GPUs that implies, asks for nodes, and configures the inference engine. `InferenceSet` sits above it and manages replica counts for the same model, reacting to a KEDA autoscaler that reads vLLM metrics. `InferencePool` creates the Gateway API Inference Extension objects, including an EPP endpoint picker that enables KVCache-aware routing. A separate `RAGEngine` operator covers the retrieval side, with its own components for the LLM endpoint, an embedding service and a vector database. ## The memory estimate is the whole point The README is unusually direct about what the project removes from the user's side of the API. Detailed deployment parameters are gone. Pipeline parallelism, data parallelism, tensor parallelism and maximum model length are not fields you fill in; the controller supplies preset values for them based on the model and the hardware you named. If you have ever hand-tuned those four knobs against a specific GPU count, this is the part of KAITO that would make you want it. Two details make that promise more credible. First, node provisioning goes through Karpenter's `NodePool` APIs rather than a bespoke mechanism, so KAITO does not own the whole hardware lifecycle. Second, the README says GPU nodes' built-in local NVMe is used as model storage, so no extra storage system is needed for inference. The `go.mod` file shows how heavy the Kubernetes side of this is, with `sigs.k8s.io/karpenter v1.10.0`, `sigs.k8s.io/gateway-api-inference-extension v1.3.1` and `sigs.k8s.io/controller-runtime v0.22.4` as direct dependencies, alongside both the AWS and Azure Karpenter providers. ## What v0.11.0 broke, and what v0.12.0 added The release history is worth reading because it documents the project moving underneath any existing deployment. v0.11.0, published on 2026-07-01, removed the legacy preset system outright. Workloads still referencing preset names had to migrate to `modelName` in the model catalog first. The same release tightened node estimation by switching to vLLM's native calculation for default `max-model-len` and GPU count, with the release note warning that deployments relying on looser estimates may need to set the value explicitly or request more nodes. v0.11.0 also promoted `InferenceSet` to `v1beta1` on by default and shipped a ProductionStack as an optional Helm subchart. v0.12.0 followed on 2026-08-27 with DeepSeek V4 Flash and Pro onboarding, NVIDIA MIG support for bring-your-own nodes, an accelerator partition mode, and real node-provisioning and inference failure reasons surfaced in Workspace status. It also expanded streaming output guardrails to redact secrets, sensitive data and invisible text. That last item is the one worth pausing on: the operator is now filtering what the model streams back to callers. ## One inference engine, and a CPU escape hatch The limitation that matters most is stated once and not repeated: only the vLLM engine is supported. LoRA adapters are supported, and KVCache offloading is on by default. Anything built on a different serving stack, such as TGI or a proprietary endpoint, is out of scope for the controller's tuning logic. There are two documented ways around that. A separate AIKit project handles CPU models, documented separately from the main operator. And there is a path for custom models hosted on HuggingFace using the HuggingFace runtime, documented under its own custom model page. The fine-tuning story sits in the same family. `examples/` contains `fine-tuning/`, `inference/`, `RAG/`, `aws/`, `custom-model-integration/` and `gateway-api-inference-extension/`, so the repository itself is the best index of what shape a working setup takes. ## Retrieval built on LlamaIndex and reciprocal rank fusion The RAGEngine CRD describes a whole retrieval service rather than just a database. Vector database support covers a built-in FAISS in-memory store as the default, with Qdrant and Milvus as persistent alternatives. Embedding can be local or remote. The interesting piece is `RAGService`, which leans on LlamaIndex for orchestration. It exposes `/index` for ingesting documents, `/v1/chat/completion` to intercept LLM calls and append retrieved context automatically, and `/retrieve` for integration with MCP servers. That last endpoint uses Reciprocal Rank Fusion to combine BM25 sparse results with dense vector results, which is a pragmatic default for teams that have neither tuned their embeddings nor indexed enough documents to justify it. The `/index` and `/retrieve` shapes mean this is designed to sit behind an existing chat product rather than to be the product itself. ## Where the README ends and the site begins The README is an architecture overview with a table of what is new, not an installation guide. Every operational step is a link: installation through Helm, installation through Terraform under `terraform/`, a quick start, autoscaling with KEDA, BYO models, CPU models, and a RAG document for the service APIs. The documentation site is where the substance lives, which is a normal arrangement for an operator of this size but worth stating when you are evaluating it. The repository itself is a better source for engineering decisions. The tree has `benchmarks/`, `test/`, `hack/`, `presets/`, `charts/`, `config/`, `plugins/`, a `Tiltfile` for local development, and a `Makefile` pinned to `VERSION ?= v0.12.0` with separate `e2e.test` and `rage2e.test` binaries. One oddity worth noting: the repository carries a `LICENSE` file in its tree, but the GitHub license field reads `NOASSERTION`, so the recognised license identifier is not settled by the API metadata alone. Check the file itself before assuming terms. The last commit on record is dated 2026-09-18, which is within months of this writing, so the project is actively moving. ## Conclusion KAITO is worth a look if you run GPU clusters on Kubernetes and would rather describe a model than hand-tune tensor parallelism, pipeline parallelism and context length. The memory estimate, the node count and the scheduling parameters are the controller's job, and that is exactly the work that is tedious and easy to get wrong by hand. The costs are real: only vLLM is supported, the recent releases show the CRD surface still moving, and the operational detail that decides whether an inference stack works lives in the documentation site rather than the README. Start from the quick start guide with a small model on hardware you already have, and treat the model catalog as the thing you will actually need to understand. ## FAQ ### What is KAITO and what problem does it solve? KAITO is a Kubernetes operator suite that automates deploying LLM inference, fine-tuning and retrieval services in a cluster. It removes detailed deployment parameters from the API, so you give it a GPU instance type and a HuggingFace model ID and the controller handles parallelism settings, memory estimation and node counts. ### Which inference engines does KAITO support? Only vLLM, according to the project documentation. LoRA adapters are supported and KVCache offloading is enabled by default. Custom models are supported through the HuggingFace runtime path, and CPU models are handled by the separate AIKit operator rather than by KAITO itself. ### How does KAITO get GPU nodes when there are not enough? It integrates with Karpenter. The controller estimates the GPU memory requirement from the instance type and model metadata, then triggers node auto-provisioning through Karpenter NodePool APIs. Any provisioner implementing the Karpenter core APIs should work, not only the one used in the project's own end to end tests. ### What are the three main CRDs in KAITO? Workspace manages a single inference or tuning workload. InferenceSet manages replica counts for the same model and drives autoscaling through KEDA reading vLLM metrics. InferencePool creates Gateway API Inference Extension resources, including an endpoint picker for KVCache-aware routing. RAGEngine is a separate operator for retrieval services. ## Sources - [Issues](https://github.com/kaito-project/kaito/issues) - [kaito-project/kaito on GitHub](https://github.com/kaito-project/kaito) - [Project website](https://kaito-project.github.io/kaito/docs/) - [README](https://github.com/kaito-project/kaito/blob/main/README.md) - [Releases](https://github.com/kaito-project/kaito/releases) --- Hysen Labs editorial analysis, written from the project's own repository and release notes. Cite the canonical page: https://hysenlabs.com/projects/kaito-project-kaito