Kubeflow Pipelines: A Kubernetes-Native ML Workflow Engine With an Argo Deadline
Machine Learning Pipelines for Kubeflow
At a glance
- What is it?
- Kubeflow Pipelines compiles Python-defined ML workflows into Argo Workflows on Kubernetes, with a MySQL 8 backing store. The current release line is 2.17.x, and the project has already announced that Argo 3.x support ends at KFP 3.0.
- Who is it for?
- Adopt Kubeflow Pipelines if your training and batch jobs already run on Kubernetes and you want pipeline definitions expressed in Python rather than YAML. Do not adopt it if you have no Kubernetes cluster to attach it to, since the standalone install still requires Argo Workflows and MySQL 8, and the documented path for KFP 3.0 requires an Argo 4.x controller first.
- Can I use it commercially?
- Yes. Apache-2.0 is a permissive licence: you can use, modify and sell software built on it, as long as you keep its copyright and licence notices.
- Is it still maintained?
- Yes. The repository received new commits within the last day.
- What is it written in?
- Mainly Go, according to GitHub's language statistics.
Answers come from the project's GitHub data, last synced on September 15, 2026, and from our analysis. They are not legal advice.
DEEP OPEN-SOURCE ANALYSIS
The gap Kubeflow Pipelines fills: Python-defined ML DAGs on Kubernetes
A data scientist who has written a training script and wants it to run on a schedule, on a cluster, with the outputs of one step feeding the next, faces a choice between writing Kubernetes manifests by hand or adopting an orchestrator. Kubeflow Pipelines targets the second option. Its stated goals in the README are end to end orchestration of ML pipelines, easy experimentation through managed trials, and re-use of components and pipelines so that teams do not rebuild each pipeline from scratch. The audience is an ML platform team that already operates Kubernetes and wants pipeline definitions authored in the Kubeflow Pipelines SDK rather than in raw YAML. The repository is written primarily in Go and carries an Apache-2.0 licence, so the control plane is a Go service even though most users touch only the Python SDK.
How a pipeline actually runs: the SDK compiles, Argo orchestrates, MySQL stores
The README states plainly that Kubeflow Pipelines uses Argo Workflows by default under the hood to orchestrate Kubernetes resources. That single sentence defines the data flow. A user writes a pipeline with the Python SDK, the SDK compiles that pipeline into an Argo Workflow representation, and the Argo controller then schedules the resulting Kubernetes resources. Kubeflow Pipelines is therefore not a scheduler of its own; it is a layer that turns Python pipeline definitions into Argo artefacts and tracks their runs. The persistence layer is MySQL, pinned at version 8 in the compatibility matrix. The execution layer changed in version 1.8: the README notes that the Docker container runtime was deprecated on Kubernetes 1.20 and that Kubeflow Pipelines switched to the Emissary Executor by default. Emissary is described as container runtime agnostic, which means the executor does not depend on a Docker daemon being present on the node. The architecture document is referenced at docs/sdk/Architecture.md rather than reproduced in the README, so anyone evaluating the internal service boundaries should read that file directly.
Installing it: platform bundle or standalone, and the dependency matrix you must satisfy
There are two documented installation routes. The first is to install Kubeflow Pipelines as part of the Kubeflow Platform. The second is to deploy it as a standalone service, and the README links an operator guide for that path. Either way the dependency matrix constrains what you can run against: Argo Workflows at v3.7 or v4.1, and MySQL at v8. Those are the only versions listed, so a cluster running an older Argo controller is outside the supported set even if it happens to work. For developers building from source, the repository includes an optional just command runner at the repo root. The README gives these examples: running just with no arguments lists available recipes, just backend-test runs the backend tests, and just backend-images builds the backend images. The README is explicit that every recipe is a thin wrapper around an existing make target, for example make -C backend/src/v2 test, and that there is deliberately no generic just build or just test recipe. Heavy or Docker-building flows are exposed only through explicitly named recipes such as backend-images. That is a small but sensible design decision: the wrapper cannot silently diverge from the make targets it calls, and a developer who prefers make loses nothing.
The Argo 3.x deprecation is the real constraint on your upgrade path
The most consequential item in the README is the KFP 3.0 compatibility notice. Argo Workflows 3.x remains supported for KFP 2.x, but it is marked deprecated and will not be supported by Kubeflow Pipelines 3.0. Before upgrading to KFP 3.0, operators must upgrade their Argo Workflows controller to a supported 4.x release listed in the matrix. The README points to a KFP 3.0 compatibility tracking issue for the removal and migration work. This is a limitation with a deadline attached, and it is not a small one. Because the Argo controller is the component that actually schedules the pipeline's Kubernetes resources, a cluster pinned to Argo 3.x can stay on the 2.x line but cannot move to 3.0 without first replacing the controller. Teams that treat Kubeflow Pipelines as a self-contained service and forget that Argo sits underneath will discover this at the worst possible moment. The version cadence visible in the release list, with 2.17.0 in July 2026 and 2.17.1 and 2.17.2 following in August and September, suggests the 2.x line is still receiving patch releases, so there is no immediate pressure. The pressure arrives when 3.0 lands.
Where Kubeflow Pipelines is the wrong tool
The dependency set is the clearest disqualifier. If a team does not operate Kubernetes, or does not want to operate Argo Workflows and a MySQL 8 instance alongside their pipeline tooling, the standalone install is not a lightweight option. The README does not describe a single-binary or embedded mode; the service is designed to sit on a cluster with those dependencies present. A second case is a team whose workflows are not containerised. Because execution runs through the Emissary executor against Kubernetes pods, the unit of work is a container image. A pipeline made of shell steps that expect a shared filesystem on one host does not map onto this model without being rewritten. A third case is a small team with a handful of jobs that run fine on a single machine. The orchestration layer here exists to coordinate many containerised steps across a cluster; adopting it for three sequential scripts adds Argo and MySQL to the operational surface for no scheduling benefit. None of this is a defect in the project. It is a mismatch between the problem the project was built for and the problem the team actually has.
How it differs from Airflow and from plain Argo Workflows
The closest comparison is Apache Airflow, and the difference is where the DAG lives. Airflow defines workflows in Python too, but its scheduler is its own component and its execution model centres on workers that run tasks, with Kubernetes available as one executor among several. Kubeflow Pipelines inverts that: the README states that Argo Workflows orchestrates Kubernetes resources by default, so Kubernetes is not an option, it is the substrate, and the pipeline is compiled into an Argo artefact rather than held in a scheduler's own database. The practical consequence is that a Kubeflow Pipelines deployment inherits Argo's controller versioning, which is exactly what the KFP 3.0 notice is about. The comparison against using Argo Workflows directly is sharper. Argo gives you the workflow engine and the Kubernetes resource model, but the README's stated goals for Kubeflow Pipelines include easy re-use of components and pipelines and management of experiments and trials. That layer, the component and experiment abstraction plus the Python SDK that produces it, is what you get on top of Argo. If your team is comfortable writing Argo Workflow YAML and has no need for reusable components or trial tracking, the extra layer buys you less than it costs.
Maintenance, licensing and what to check before you commit
Kubeflow Pipelines is Apache-2.0, which permits commercial use and modification; the licence text governs, and this is not legal advice. Maintenance cost is concentrated in the dependency chain rather than in the project's own releases. You are responsible for keeping Argo Workflows and MySQL 8 at supported versions, and the Argo 3.x deprecation means that responsibility has a scheduled consequence at KFP 3.0. On the project side, the release list shows patch releases through September 2026 and the last push to the default branch is dated the same month, so the 2.x line is active. Two documentation notes are worth flagging. The README links a DeepWiki page for AI-generated repository documentation but carries an explicit warning that it is AI generated and may not have completely accurate information, so it should not be treated as a specification. The README also links a community meeting held every other Wednesday and a Slack channel, which are the practical escalation paths when the documentation is thin. The architecture details live in docs/sdk/Architecture.md and the API specification is published separately, so an evaluation that stops at the README will miss the service boundaries entirely.
Editorial conclusion
Adopt Kubeflow Pipelines if your training and batch jobs already run on Kubernetes and you want pipeline definitions expressed in Python rather than YAML. Do not adopt it if you have no Kubernetes cluster to attach it to, since the standalone install still requires Argo Workflows and MySQL 8, and the documented path for KFP 3.0 requires an Argo 4.x controller first. Before committing, verify which Argo version your cluster runs, confirm the MySQL 8 instance you intend to use, and read the KFP 3.0 compatibility tracking issue linked from the README.
Community notes