AIKit: Turning LLM Inference and Fine-Tuning into OCI Artifacts

The problem AIKit solves: model distribution, not model quality

Serving an open model locally is not hard. Getting the same model onto five machines, a CI runner and a Kubernetes cluster, with a known provenance record, is the annoying part. AIKit's answer is to treat the model, its runtime and its configuration as one container image or one OCI artifact, so the registry you already use becomes the distribution channel. The README states the project offers three capabilities: inference through LocalAI, fine-tuning through Unsloth, and OCI packaging that supports the CNCF ModelPack specification plus generic artifact packaging. That third item is the actual differentiator. The audience is engineers who already have Docker or Podman and a registry, and who would rather pull an image than manage Python environments, weight directories and CUDA drivers by hand. The README is explicit that no GPU, internet access or additional tools are needed beyond Docker or Podman for the basic path, which sets the entry bar at a single command.

What happens between docker run and the OpenAI-compatible endpoint

The runtime layer is LocalAI, not code written by the kaito-project. The README describes LocalAI as a drop-in replacement REST API that is OpenAI API compatible, which is why AIKit can advertise compatibility with clients such as Kubectl AI and Chatbot-UI without writing per-client adapters. The documented flow is short: the container starts an HTTP server on port 8080, exposes a chat UI at /chat, and serves /v1/chat/completions. Model selection happens at image level. Each pre-made image is tagged with a model and size, for example ghcr.io/kaito-project/aikit/llama3.1:8b, and the request body names the model as llama-3.1-8b-instruct. The README also notes support for multiple models within a single image, GGUF and GGML formats, multi-modal models, image generation, and CPU instruction-set selection at startup. The build side is the part that is genuinely AIKit's own: a declarative configuration for inference and fine-tuning, compiled into images with a chiseled Ubuntu base, which the README ties to a smaller attack surface. Packaging produces OCI artifacts that any OCI-compliant registry can hold, so the same artifact can be pulled by a Kubernetes node or inspected locally.

Running it: one command, one curl, no GPU

The README's quick start is a single command: docker run -d --rm -p 8080:8080 ghcr.io/kaito-project/aikit/llama3.1:8b. The web UI is then at http://localhost:8080/chat. The API example is a curl against http://localhost:8080/v1/chat/completions with a JSON body containing a model field and a messages array. The README prints an example response where the assistant content is a one-sentence description of Kubernetes, and labels the output as illustrative rather than a benchmark. Pre-made CPU images listed in the README include llama3.2:1b, llama3.2:3b, llama3.1:8b, llama3.3:70b, mixtral:8x7b and phi4:14b, each with a model name such as llama-3.2-1b-instruct or phi-4-14b-instruct. The README states that AMD64 and ARM64 are both supported and that Docker pulls the matching image automatically. For anything outside that list, the documented path is to create your own image and host it in a registry of your choice. GPU inference is documented separately, with NVIDIA CUDA and AMD ROCm named as supported backends, and Kubernetes deployment has its own documentation page.

The air-gapped claim has an asterisk worth reading

The README's air-gapped bullet is the most carefully worded line in the file. It says air-gapped inference is supported with self-hosted or local registries when model content and dependencies are baked or mirrored ahead of time, and then adds that runner images which download models at startup are not air-gapped by default. That distinction matters more than the feature list suggests. If your chosen image fetches weights on first boot, a disconnected cluster will fail at container start, not at build time, and the failure will look like a network problem rather than a packaging decision. The mitigation is on the packaging side: bake the model into the image or mirror it into a local registry before the cluster needs it. Anyone evaluating AIKit for a restricted environment should confirm which of their images are baked and which are pullers, because the README does not enumerate that per image.

Fine-tuning is real but it is not the CPU path

Fine-tuning is listed as one of the three main capabilities and is backed by Unsloth, which the README describes as fast and memory efficient. There is a dedicated fine-tune documentation page and a separate declarative spec for fine-tuning, distinct from the inference spec. What the README does not do is give a fine-tuning command in the quick start. The quick start is inference only, and the no-GPU promise in the features list is attached to the general workflow rather than to training. Unsloth is a GPU-oriented training library, so the reasonable reading is that the fine-tuning path expects an NVIDIA GPU even though the inference path does not. That is a gap in the material rather than a contradiction, but it means a team planning to fine-tune should treat the fine-tune docs as the source of truth and budget for GPU capacity separately from the laptop demo.

Where AIKit is the wrong tool

AIKit's pre-made catalog is thin. The README lists roughly a handful of families: Llama 3.2, 3.1 and 3.3, Mixtral, Phi 4, Gemma 2 and a truncated entry after that. If your requirement is to run whatever model was published last week, you are building your own image, and at that point the value you get from AIKit is the packaging and serving layer, not the model selection. Second, the container-per-model pattern is a poor fit for high-throughput serving. A single LocalAI process in a single container is a different shape from a multi-process inference server with continuous batching, and the README makes no throughput claims. Third, the licence column in the model table is a reminder that packaging convenience does not change model terms: the Llama entries point at Meta's Llama licence, Mixtral at Apache 2.0, Phi 4 at MIT. Shipping a Llama image to your users is a licensing decision, not a Docker decision. Fourth, the README does not state what happens to a running container when you want to swap models; the documented unit of change appears to be the image, which means a restart rather than a hot reload.

How it differs from Ollama and from vLLM

Ollama is the closest comparison for the local developer experience, and the difference is in the distribution unit. Ollama manages models through its own CLI and its own model store, so a model is something you pull with ollama pull and reference by name. AIKit makes the model an OCI image or artifact, which means your existing registry, your existing image signing, your existing SBOM tooling and your existing Kubernetes pull secrets all apply. The README lists SBOMs, provenance attestations and signed images as supply chain features, and that list is the clearest statement of intent: this is a container-native distribution story. vLLM sits at the other end. It is an inference server aimed at GPU throughput, and AIKit's README makes no comparable performance claims, because it is not trying to. If your problem is tokens per second per GPU, AIKit is not the answer. If your problem is getting a known model onto a cluster with a verifiable provenance record, the OCI framing is the reason to pick it.

Maintenance, versioning and the MIT licence

AIKit is MIT licensed, which is permissive and places few obligations on redistribution of the tool itself. It does not relicense the models. The README's model table carries a separate licence column, and those terms travel with the weights regardless of how you package them. The repository is not archived and the release cadence visible in the material is roughly three releases between December 2025 and August 2026: v0.20.3, then v0.21.0, then v0.22.1. That is a project still on a zero-major version, which in practice means the CLI surface can move. Anyone pinning AIKit in a build pipeline should pin the CLI version and the image tag together, and read the release notes between those tags rather than assuming a v0.20 command still works in v0.22. The upgrade cost is mostly in the build configuration, since the runtime images are versioned separately on the registry. Because AIKit sits on top of LocalAI, Unsloth and llama.cpp, its maintenance burden is partly inherited: a LocalAI or llama.cpp change can affect behaviour without any AIKit release.

Editorial conclusion

Adopt AIKit if you already run Kubernetes or an OCI registry and want model distribution to use the same pull, sign and SBOM machinery as your other images, or if you want a CPU-only OpenAI-compatible endpoint from a single docker run. Do not adopt it if you need day-zero coverage of new model releases, a large curated model catalog, or a fine-tuning workflow without an NVIDIA GPU. Before committing, verify three things: which pre-made image tag matches your model and licence, whether the runner image downloads weights at startup (which the README says is not air-gapped by default), and which version of the aikit CLI the fine-tune documentation targets, since the CLI has moved from v0.20.3 to v0.22.1 in under a year.

Official sources

1. kaito-project/aikit on GitHub
2. License: MIT
3. Project website
4. README
5. Releases