Argo Workflows: Container-Native Workflow Engine for Kubernetes

What Argo Workflows Is and Who Uses It

Argo Workflows is a workflow engine for Kubernetes. It is implemented as a Kubernetes Custom Resource Definition (CRD), which means workflows are defined as Kubernetes manifests and managed by the Kubernetes API. Each step in a workflow is a container, not a function or a script running in a shared environment. The container-per-step model means that every step has an independent runtime, its own resource limits, and its own image, without sharing a process space with any other step.

The project is a graduated project of the Cloud Native Computing Foundation (CNCF), licensed under Apache-2.0. Roughly 200 organizations are listed in the USERS.md file as official adopters.

The primary use cases documented in the README are machine learning pipelines, data and batch processing, infrastructure automation, and CI/CD. The common thread is any workload that benefits from running multiple containers in a defined order with conditional logic, parallel branches, and dependency tracking across steps.

Workflows as Kubernetes Resources: CRDs and the Execution Model

Deploying Argo Workflows installs CRDs for the Workflow, WorkflowTemplate, CronWorkflow, and related resource types. Once installed, a workflow is submitted by applying a YAML manifest to the cluster, and the Argo Workflows controller picks it up and begins scheduling the steps as Kubernetes pods.

The workflow controller watches for Workflow resources and creates pods for each step according to the execution graph. An argoexec sidecar container in each pod handles artifact collection and parameter output, communicating results back to the controller so downstream steps can consume them. This is the mechanism for passing outputs between steps: artifacts (files) are stored in a configured backend (S3, GCS, Azure Blob, Artifactory, and others are all supported), and parameters are short values passed through the Kubernetes API.

Workflow templates can be stored in the cluster for reuse. A CronWorkflow resource triggers a workflow on a schedule. The server component provides a REST and gRPC API for submitting and monitoring workflows, and a web UI for visualizing the execution graph and viewing logs.

DAG Workflows, Step Sequences, and Parallel Execution

Argo Workflows supports two workflow declaration models. The steps model defines a sequence of stages, where each stage can contain multiple parallel steps. The DAG model expresses workflows as a directed acyclic graph, with each task declaring its dependencies explicitly.

Both models support loops (running a step once per item in a list), conditionals (skipping steps based on parameter values), retries with configurable backoff, and timeouts at both the step and workflow level. Daemoned steps can run continuously in the background while other steps proceed. Exit hooks run after a workflow completes or fails, enabling notification and cleanup actions.

Parallelism limits can be set at the workflow and namespace level to prevent a single workflow from consuming all cluster resources. Pod scheduling can use Kubernetes affinity, tolerations, and node selector rules, which is relevant for ML workloads that need GPU nodes.

SDKs for defining workflows in code rather than YAML are available: Hera (Python), Juno (TypeScript), Java, and Go are listed in the README. Hera is noted specifically as the path for Python users who want to avoid writing raw YAML.

Installing Argo Workflows and Running a First Workflow

Argo Workflows is installed into a Kubernetes cluster via kubectl manifests or a Helm chart. The Helm chart is published to Artifact Hub as argo/argo-workflows and tracked by a badge in the README. The quickstart documentation at argo-workflows.readthedocs.io/en/latest/quick-start/ provides the install commands for a minimal setup, including namespace creation and the controller/server deployment.

After installation, the Argo CLI (available at the releases page) can be used to submit and monitor workflows. The CLI provides the argo submit, argo list, argo get, and argo logs commands for workflow management.

The repository's examples/ directory contains dozens of example workflow definitions covering artifact passing, conditionals, loops, DAG patterns, CI pipelines, and coinflip (random branching) workflows. The README links to killercoda.com/argoproj/course/argo-workflows/ for an interactive training environment that runs without a local Kubernetes cluster.

Artifacts, Single Sign-On, and Enterprise Integration

The artifact system in Argo Workflows is configurable per workflow and per step. Supported backends include S3 (and S3-compatible stores), Google Cloud Storage, Azure Blob Storage, Alibaba Cloud OSS, Artifactory, HDFS (via the colinmarc/hdfs dependency), raw (inline), Git, and plugin-provided backends. Artifact garbage collection is documented as a supported feature, allowing old workflow artifacts to be cleaned up automatically.

The server component supports Single Sign-On via OAuth2 and OIDC, which is relevant for teams that need role-based access to the workflow UI and API. Webhook triggering allows external events to submit workflows. Prometheus metrics are available out of the box and as custom metrics defined in the workflow spec itself.

Windows containers are documented as supported, which is relevant for CI/CD workloads that must build or test Windows binaries.

Where Argo Workflows Creates Operational Burden

Argo Workflows requires a running Kubernetes cluster. There is no standalone deployment option: the workflow controller, server, and executor are all Kubernetes-native components. Teams that do not operate Kubernetes cannot use Argo Workflows without first adopting it.

The executor model means every step creates a pod. For workflows with many short-lived steps, the pod scheduling overhead accumulates. For ML experimentation workloads that require running hundreds of fast hyperparameter-tuning steps, the Kubernetes pod startup latency may be significant.

Upgrade management is documented in the repository changelog. The project uses semantic versioning and maintains separate changelogs for the v2 and v3 series (CHANGELOG-2-x-x.md and CHANGELOG-3-x-x.md). Reviewing release notes before upgrading is necessary because CRD changes require migration steps.

Argo Workflows versus Apache Airflow

Apache Airflow is a workflow scheduler that defines workflows as Python DAGs, runs tasks in a shared executor environment, and does not require Kubernetes for basic operation. The practical comparison is that Airflow is better suited for teams with existing Python data engineering workflows on non-Kubernetes infrastructure, while Argo Workflows is designed for teams that need container isolation per task and are already operating Kubernetes.

The container-per-step model in Argo Workflows means each step can use a different image, language, and runtime. An Airflow task runs in a shared Python environment, which introduces dependency conflicts for heterogeneous workloads. Argo Workflows avoids this at the cost of pod startup overhead.

A blog post linked in the README (Argo Workflows vs Apache Airflow at bit.ly/30YNIvT) compares the two in depth. The Argo project also integrates with Kubeflow Pipelines, Netflix Metaflow, and Kedro, which are Python-native ML orchestration layers that can submit Argo Workflows underneath.

Editorial conclusion

Argo Workflows is the right tool for teams that already run Kubernetes and need to orchestrate multi-step container workloads for ML pipelines, data processing, or CI/CD. It is the wrong tool for teams that do not operate Kubernetes, since it has no standalone deployment path; Airflow or Temporal serve those environments instead. Before adoption, verify that your Kubernetes cluster version and networking policies are compatible with the Argo Workflows controller and executor, and review the artifact backend configuration you will need for storing workflow outputs.

Frequently asked questions

Official sources

1. argoproj/argo-workflows on GitHub
2. License: Apache-2.0
3. Project website
4. README
5. Releases