OME: A Kubernetes Operator That Puts Models, Runtimes and GPUs Behind Custom Resources
Open Model Engine (OME) — Kubernetes operator for LLM serving, GPU scheduling, and model lifecycle management. Works with SGLang, vLLM, TensorRT-LLM, and Triton
At a glance
- What is it?
- OME (Open Model Engine) is an Apache-2.0 Kubernetes operator that turns LLM serving into a set of CRDs: BaseModel, ServingRuntime, InferenceService, AcceleratorClass and BenchmarkJob. This review covers the mechanism, the install path, and where the operator model stops being the right answer.
- Who is it for?
- Adopt OME if you already run Kubernetes 1.28 or newer and want model definitions, runtime choice and GPU placement expressed as CRDs rather than as a pile of per-team Helm charts. Do not adopt it if you serve one model on one node and never intend to scale past that, or if you cannot take on the CRD and controller upgrade path.
- Can I use it commercially?
- Yes. Apache-2.0 is a permissive licence: you can use, modify and sell software built on it, as long as you keep its copyright and licence notices.
- Is it still maintained?
- Yes. The repository received new commits within the last day.
- What is it written in?
- Mainly Go, according to GitHub's language statistics.
Answers come from the project's GitHub data, last synced on September 15, 2026, and from our analysis. They are not legal advice.
DEEP OPEN-SOURCE ANALYSIS
The Problem OME Targets: Model Serving Drift Across Teams
Most Kubernetes LLM deployments start as a Deployment manifest with a container image tag, a GPU resource request, and a mounted volume. That works until a second model appears, then a third, and each one arrives with its own runtime, its own quantization format and its own idea of how many GPUs it needs. The operator's answer is to make the model itself an object in the cluster. BaseModel and ClusterBaseModel describe where weights come from and what they contain. ServingRuntime and ClusterServingRuntime describe how a given engine is launched. InferenceService is the binding between the two. The README states that model parsing extracts architecture, parameter count and capabilities directly from model files, so the cluster holds a description of the artifact rather than a filename convention. The audience is platform teams running more than a handful of models, particularly teams that need prefill-decode disaggregation or multi-node inference and do not want to hand-write that topology per service. A single-model team gets little from this. The operator's value is proportional to the number of models and the number of distinct hardware profiles you are juggling.
How the Controller Chain Turns a BaseModel Into a Running Endpoint
The architecture section lists the resources and then states what the controller does with them in sequence: download and parse the model, select the optimal runtime configuration, match the model to appropriate accelerators, and generate Kubernetes objects. The interesting step is the second one. Runtime selection is described as weighted scoring across architecture, format, quantization, parameter size and framework compatibility. That is a matching problem, not a lookup. It means a model whose format is SafeTensors and whose size fits a single accelerator can land on one runtime configuration, while the same model quantized and split across nodes can land on another, without the user editing a runtime spec. AcceleratorClass supplies the other half: GPU capabilities, discovery patterns and cost information, with selection policies named as BestFit, Cheapest and MostCapable. Cost as a first-class field in a scheduling resource is a deliberate choice, and it only pays off if the cost numbers in your AcceleratorClass objects reflect reality. The README also notes integration with Kueue for gang scheduling, LeaderWorkerSet for multi-node deployments, KEDA for custom-metrics autoscaling and the Gateway API plus the Gateway API Inference Extension for routing. Those are separate projects with their own CRDs, so OME is a coordinator over an existing stack rather than a replacement for it.
Installing OME and What the Helm Charts Actually Split Into
The README requires Kubernetes 1.28 or newer. Installation from the OCI registry is the recommended path and uses two charts, which is the detail worth noting: CRDs ship separately from the controller and its resources. The commands given are helm upgrade --install ome-crd oci://ghcr.io/moirai-internal/charts/ome-crd --namespace ome --create-namespace followed by helm upgrade --install ome oci://ghcr.io/moirai-internal/charts/ome-resources --namespace ome. Installing from source follows the same split against local paths: helm install ome-crd charts/ome-crd and helm install ome charts/ome-resources. A third chart, ome-serving, is optional and deploys pre-configured ClusterBaseModels, ClusterServingRuntimes and InferenceServices. That third chart is the one to read carefully before applying, because it decides what your cluster starts serving. The registry host in the documented commands is ghcr.io/moirai-internal, which does not match the ome-projects organization name on the repository. The README gives it as the install path, so treat that as the documented source of truth and verify pull access from your own network before you plan a rollout around it. The README does not show a values.yaml, so chart-level configuration is not documented in the material available here.
Where the Operator Model Costs You: CRDs, Upgrades and Runtime Coverage
The largest commitment is the CRD surface. BaseModel, FineTunedWeight, ServingRuntime, InferenceService, AcceleratorClass and BenchmarkJob are all cluster-scoped or namespaced API objects with a v1beta1 version. Beta means the schema can still move. Upgrading OME is therefore not just a controller image bump: the ome-crd chart has to be applied before the ome-resources chart, and any field that changed between v1.2.0, v1.2.1 and v1.2.2 has to be reconciled in stored objects. The release cadence visible in the material is roughly three weeks between v1.2.0 in mid-July and v1.2.2 in early August, which is frequent enough that pinning chart versions matters. Runtime coverage is the second constraint. The README calls SGLang first-class and lists cache-aware load balancing, multi-node deployment, prefill-decode disaggregation and multi-LoRA adapter serving under it; vLLM is described as supported for high-throughput inference with no comparable feature list. The repository description also names TensorRT-LLM and Triton, but the feature section does not describe what OME does differently for them. If your stack is built on TensorRT-LLM, the material here does not tell you how deep that integration goes, and that is a gap to close before you commit. The third constraint is the AcceleratorClass requirement. Hardware-aware scheduling depends on someone writing those resources with accurate capabilities and costs. A cluster with no AcceleratorClass objects gives the scheduler nothing to score against.
The Alternative: Plain Deployments Plus KServe or Raw vLLM Manifests
The obvious comparison is not another operator with the same scope; it is the combination most teams use today, which is KServe for the serving abstraction plus hand-written manifests for the rest. KServe gives you an InferenceService-style resource and a predictor abstraction, but it does not attempt weighted runtime selection across model formats, and it does not ship an AcceleratorClass concept with cost-aware placement policies. The difference in approach is where the decision lives. With KServe and raw manifests, a human decides which runtime serves which model and encodes that decision in YAML. With OME, the controller makes that decision from parsed model metadata and a scoring function, and the human writes the runtime definitions and accelerator classes that constrain it. That is a real trade: you get consistency across many models, and you give up the ability to read a single manifest and know exactly what will run. If your team already has a working vLLM Deployment per model and the count is stable, OME adds a controller, six CRDs and a beta API lifecycle in exchange for automation you may not need. If the count is growing and the runtime choices are inconsistent between teams, the automation is the point.
BenchmarkJob and the Measurement Loop
BenchmarkJob is the resource that closes the loop on runtime selection. The README describes it as configurable traffic patterns, concurrent load testing and result storage, used for systematic comparison across models and service configurations. This matters because weighted scoring is only as good as the weights, and the weights are only as good as the measurements behind them. A team that changes a quantization format or moves a model to a different AcceleratorClass can run the same BenchmarkJob shape against both configurations and compare stored results rather than arguing from intuition. The limitation is that the README does not document the BenchmarkJob schema, the traffic pattern options or where results are stored. You will need the API reference at ome-projects.github.io/ome/docs/reference/ome.v1beta1/ to write one. Treat BenchmarkJob as a capability to investigate rather than a turnkey harness, and be aware that it measures what it is pointed at, which means a badly chosen traffic pattern produces a confident number that means nothing.
Licence, Maintenance Load and What to Check First
OME is Apache-2.0, which permits commercial use, modification and redistribution, and includes an explicit patent grant. It does not impose copyleft obligations on your own code. If you fork and redistribute a modified operator you must preserve the licence and notices, and the licence text itself is the authority rather than this summary. Maintenance cost has three parts. First, the CRD upgrade path: the ome-crd chart must be applied before ome-resources on every version bump, and with a beta API you should expect schema movement between minor releases. Second, the dependency surface: Kueue, LeaderWorkerSet, KEDA and the Gateway API Inference Extension are separate projects with their own release cycles, and a version skew between any of them and OME is a failure mode you own. Third, the AcceleratorClass content: capabilities and cost figures are data you maintain, and stale cost data silently degrades placement decisions rather than raising an error. The web console lives in a separate repository, ome-projects/ome-console, so it versions independently of the operator. Before adopting, verify four things: that your cluster is on Kubernetes 1.28 or newer, that the OCI charts under ghcr.io/moirai-internal pull from your network, that the v1beta1 API reference matches the chart version you intend to run, and that the runtime you actually use (SGLang, vLLM, TensorRT-LLM or Triton) has documented behaviour in OME beyond being listed as supported.
Editorial conclusion
Adopt OME if you already run Kubernetes 1.28 or newer and want model definitions, runtime choice and GPU placement expressed as CRDs rather than as a pile of per-team Helm charts. Do not adopt it if you serve one model on one node and never intend to scale past that, or if you cannot take on the CRD and controller upgrade path. Before committing, verify the v1beta1 API reference against your cluster version, confirm the OCI chart path ghcr.io/moirai-internal/charts/ome-crd resolves from your network, and check whether your accelerator is described by an AcceleratorClass you can write yourself.
Community notes