# Kubeflow MPI Operator: running allreduce training jobs on Kubernetes

> The MPI Operator is a Kubernetes controller that turns an MPIJob custom resource into a launcher pod running mpirun plus a set of worker pods, which is how Kubeflow runs distributed training and HPC workloads. The launcher and worker split, and the Volcano scheduler dependency, are the two details that shape what you can run.

**kubeflow/mpi-operator** — Kubernetes Operator for MPI-based applications (distributed training, HPC, etc.)

- Repository: https://github.com/kubeflow/mpi-operator
- Website: https://www.kubeflow.org/docs/components/training/mpi/
- Stars: 540 · Forks: 240
- Language: Go
- License: Apache-2.0
- Published: 2026-10-22 · Updated: 2026-10-22 · Language: en
- Canonical page: https://hysenlabs.com/projects/kubeflow-mpi-operator

## What the operator does: an MPIJob becomes a launcher and workers

The MPI Operator makes it easy to run allreduce-style distributed training on Kubernetes, and the mechanism is worth understanding because the shape of the resource drives everything else.

An MPIJob custom resource describes a launcher and a set of workers, and the operator reconciles that into pods. The launcher is the process that runs the actual mpirun command across the worker set, and the workers are the pods that hold the devices. Here is the shape of the spec, taken from the sample output in the documentation:

```yaml
spec:
  runPolicy:
    cleanPodPolicy: Running
  mpiReplicaSpecs:
    Launcher:
      replicas: 1
      template:
        spec:
          containers:
          - command:
            - mpirun
            - --allow-run-as-root
            - -np
            - "2"
```

Two fields carry most of the meaning. mpiReplicaSpecs holds the launcher and worker templates, and the worker template in the same sample requests a GPU limit of nvidia.com/gpu: 2 with slotsPerWorker set to 2, which is the device count each worker contributes to the allreduce operation.

The alternative to the operator is running mpirun yourself inside a pod, which means solving the problem the operator exists to solve. In an allreduce job every rank has to start roughly together, because the collective will not complete until all participants are present. A pod that starts the ranks sequentially will hang on the first collective, so you need something that holds the job until the whole set is scheduled, and then starts them together. That is gang scheduling, and it is the reason the operator exists rather than a plain Job.

The topics list confirms the intended users: distributed computing, horovod, pytorch, tensorflow, apache-mxnet, HPC and Kubernetes. The sample workload is a TensorFlow benchmark, and the topic list also names the frameworks people actually run under it.

## Installing it, and the two ways the manifests differ

The installation is two kubectl commands that differ only in which branch they point at.

For the latest development version:

```shell
kubectl apply --server-side -f https://raw.githubusercontent.com/kubeflow/mpi-operator/master/deploy/v2beta1/mpi-operator.yaml
```

And for the release version, which pins the tag:

```shell
kubectl apply --server-side -f https://raw.githubusercontent.com/kubeflow/mpi-operator/v0.8.2/deploy/v2beta1/mpi-operator.yaml
```

Both use server-side apply, which matters when you are layering multiple manifests onto the same cluster, because client-side apply can conflict with fields another manifest already owns.

The alternative is to follow the Kubeflow getting started guide and deploy the whole of Kubeflow, since the operator is one component of a larger stack.

The history is worth one sentence, because it explains the version numbers you will meet in the wild. An alpha version of MPI support was introduced with Kubeflow 0.2.0, and you must be using a version of Kubeflow newer than 0.2.0.

To check that the custom resource definition landed, you list the CRDs:

```
kubectl get crd
```

And the output should include mpijobs.kubeflow.org:

```
NAME                                       AGE
...
mpijobs.kubeflow.org                       4d
...
```

If the definition is missing, the fallback is to build the manifests with kustomize yourself:

```bash
git clone https://github.com/kubeflow/mpi-operator
cd mpi-operator
kustomize build manifests/overlays/kubeflow | kubectl apply -f -
```

Since Kubernetes v1.14, kustomize became a subcommand of kubectl, so on a current cluster you can skip the build step and apply the overlay directly:

```bash
kubectl apply -k manifests/overlays/kubeflow
```

Or pipe the rendered output yourself:

```bash
kubectl kustomize manifests/overlays/kubeflow | kubectl apply -f -
```

The project is Apache-2.0 licensed and written in Go, with the last push on 2026-10-19 and the recent releases v0.8.2 on 2026-07-03, v0.8.0 on 2026-02-17 and v0.7.0 on 2025-10-30.

## Running a job and reading its status

Creating a job is applying an MPIJob manifest, and the repository ships a TensorFlow benchmark example to start from.

The example is inspected with cat and applied with kubectl apply:

```
kubectl apply -f examples/v2beta1/tensorflow-benchmarks/tensorflow-benchmarks.yaml
```

The point of the example is that you copy it and change it, and the parameters worth understanding are the ones in the launcher command. The mpirun flags in the sample are the whole story of how the job is wired: -np with 2 is the number of processes, -bind-to none and -map-by slot distribute the ranks across the slots that each worker contributes, and the -x flags export environment variables into the launched program, in this case NCCL_DEBUG=INFO, LD_LIBRARY_PATH and PATH. The -mca settings select the pml ob1 and btl openib components, which are the Open MPI memory and point-to-point layers, and --allow-run-as-root is there because a containerised job usually runs as root.

Monitoring has two parts. The pods appear with the requested number of GPUs, and the job status is in the status section of the resource:

```
kubectl get -o yaml mpijobs tensorflow-benchmarks
```

The status output in the documentation is worth reading for its structure rather than its values. The conditions array carries the lifecycle as separate entries with a reason and a message: MPIJobCreated, MPIJobRunning and MPIJobSucceeded, each with its own transition timestamps, and a completionTime once the job finishes. The replicaStatuses block shows the launcher succeeded with 1 and the worker entry as an empty object.

The log is on the launcher pod, which is where mpirun's output goes, and you find it with a label selector:

```
PODNAME=$(kubectl get pods -l training.kubeflow.org/job-name=tensorflow-benchmarks,training.kubeflow.org/job-role=launcher -o name)
kubectl logs -f ${PODNAME}
```

The labels training.kubeflow.org/job-name and training.kubeflow.org/job-role are what make this work, and the role label is how you select the launcher specifically rather than any pod of the job.

## Volcano, the scheduler plugin, and what the dependencies say

The go.mod is more informative about the design than the feature list is, because it names what the operator expects to find in your cluster.

The Kubernetes libraries are all at 0.35.6, covering api, apimachinery, apiserver, client-go and code-generator, with klog at 2.140.0 and kube-openapi and utils pinned to specific pseudo-versions. The controller runtime is sigs.k8s.io/controller-runtime at 0.23.3, which is the standard way to write a Kubernetes controller in Go, and sigs.k8s.io/kind at 0.31.0 is the API type generation tool.

The two dependencies that shape what you can run are scheduler-plugins at 0.34.7 and volcano.sh/apis at 1.14.0. The comment in the Makefile explains the first: the go.mod go version is used as a single source of truth for the scheduler-plugins version, and the Volcano scheduler version is derived from the same mechanism. So the operator leans on Volcano for gang scheduling, and the versions of the scheduling components are pinned by the go directive rather than maintained separately.

The rest of the direct dependencies are the usual controller toolkit: ginkgo and gomega for the test suite, go-cmp for comparisons, prometheus client_golang for the metrics the operator exposes, golangci-lint as a tool dependency, and the x/crypto, x/time and x/tools modules. The Makefile pins the toolchain versions around it, with a Helm version of v3.11.2, a kubectl version of v1.35.0 chosen because it supports -k for kustomization, and an envtest Kubernetes version of 1.35.0, which is the kubebuilder test environment the controller tests run against.

The metrics and the Prometheus dependency matter more than they sound. A training job operator is a component that sits in a cluster for months, and a controller that can be scraped is one you can alert on. The documentation has an Exposed Metrics section, and the fact that the table in the visible text is empty is a reminder that the metrics list is generated or truncated rather than absent.

The repository layout also carries the things you would expect in a project of this maturity: a proposals/ directory for design documents, a ROADMAP.md, a RELEASE.md, an ADOPTERS.md, an sdk/ directory, a hack/ directory for build scripts, and a third_party/ directory with a licenses file that the Dockerfile copies into the image.

## The container image, and the platforms it is built for

The Dockerfile is a two-stage build with a distroless runtime, and the details tell you how the image is put together.

The build stage is golang:1.26, the source is added to the GOPATH-style path, and the operator binary is produced with a make target whose name takes the controller version as a suffix, so make RELEASE_VERSION=... mpi-operator.v2. A symlink then points at the versioned binary in the output directory, which is how the runtime stage copies a stable path.

The runtime stage is gcr.io/distroless/base-debian12:latest, the controller version is set as an environment variable, the built binaries are copied into /opt, and the third_party library license file is copied alongside them as license.txt. The entrypoint is /opt/mpi-operator with a default command of --help, which is a good sign, because a container that starts by printing its usage does not silently crash on a bad flag.

The Makefile shows the platform matrix, and it is not uniform. The default PLATFORMS is linux/amd64, linux/arm64 and linux/ppc64le, so the ppc64le build is for the high-performance computing clusters that need it. The Intel-specific image is built for linux/amd64 only, and the MPICH-specific image covers linux/amd64 and linux/arm64.

That split is the clearest statement of the project scope. An MPI distribution is not just a library, it is a specific implementation with its own launcher semantics and its own vendor libraries, so building separate images for the Intel and MPICH stacks rather than trying to support both in one binary is a reasonable decision.

The registry defaults to docker.io/mpioperator and the image name to mpi-operator, and there is a Docker Hub badge for the pull count. The build also defines an SSH port variable for the image builder, which is a kubebuilder asset requirement rather than something the operator itself listens on.

## Conclusion

Adopt the MPI Operator if you are running allreduce style distributed training and want the launcher and worker lifecycle managed as a Kubernetes resource rather than a batch script. Do not adopt it expecting general job scheduling, since it is scoped to MPI style workloads and its gang scheduling comes from Volcano. Before you deploy, check that your cluster has the Volcano scheduler installed, since the operator depends on it, and confirm the api version you are applying matches your Kubeflow install, because the v2beta1 manifests are a different shape from the v1alpha2 shown in older status output.

## FAQ

### What does the Kubeflow MPI Operator do?

It makes it easy to run allreduce-style distributed training on Kubernetes, by turning an MPIJob custom resource into a launcher pod running mpirun plus a set of worker pods, so that the ranks start together rather than sequentially and the collective can complete.

### How do I install the MPI Operator?

Apply the manifests with kubectl apply --server-side pointing at the master branch for the development version or at a release tag such as v0.8.2 for the stable one, and check with kubectl get crd that mpijobs.kubeflow.org appears. If it does not, apply the kustomize overlay with kubectl apply -k manifests/overlays/kubeflow.

### Which scheduler does the MPI Operator depend on?

Volcano. The go.mod depends on volcano.sh/apis and on sigs.k8s.io/scheduler-plugins, and the Makefile derives the scheduler-plugins and Volcano scheduler versions from the go.mod go version as a single source of truth.

### How do I see the logs of a running MPI Job?

From the launcher pod, since that is where mpirun runs. Select it with the labels training.kubeflow.org/job-name and training.kubeflow.org/job-role=launcher, then run kubectl logs -f on the pod name that selector returns.

### Which platforms is the mpi-operator image built for?

The default set is linux/amd64, linux/arm64 and linux/ppc64le, with the Intel-specific image built for linux/amd64 only and the MPICH-specific image for linux/amd64 and linux/arm64. The runtime image is distroless base-debian12.

## Sources

- [kubeflow/mpi-operator on GitHub](https://github.com/kubeflow/mpi-operator)
- [License: Apache-2.0](https://github.com/kubeflow/mpi-operator/blob/master/LICENSE)
- [Project website](https://www.kubeflow.org/docs/components/training/mpi/)
- [README](https://github.com/kubeflow/mpi-operator/blob/master/README.md)
- [Releases](https://github.com/kubeflow/mpi-operator/releases)

---

Hysen Labs editorial analysis, written from the project's own repository and release notes. Cite the canonical page: https://hysenlabs.com/projects/kubeflow-mpi-operator