# Armada adds queues, fairness and gang scheduling across Kubernetes clusters

> Armada is an open source batch job meta-scheduler that puts a control plane above a fleet of Kubernetes clusters, adding a prioritised queue with fair-share and rate limiting, gang scheduling, preemption and Prometheus metrics, with a Lookout web interface for operators. It suits organisations running many batch jobs whose demand exceeds one cluster, and it adds a scheduler layer of its own.

**armadaproject/armada** — A multi-cluster batch queuing system for high-throughput workloads on Kubernetes.

- Repository: https://github.com/armadaproject/armada
- Website: https://armadaproject.io
- Stars: 640 · Forks: 181
- Language: Go
- License: Apache-2.0
- Published: 2026-10-21 · Updated: 2026-10-21 · Language: en
- Canonical page: https://hysenlabs.com/projects/armadaproject-armada

## Kubernetes was built for services, Armada for batch

The README frames the whole project in one line. Kubernetes was built for services, Armada was built for batch. The pitch on the site is a single API across any number of clusters for millions of jobs, described as the open source batch job meta-scheduler that makes Kubernetes work at scale. The argument is that when job volume exceeds what a single cluster can handle, you need a control plane sitting above the fleet, routing jobs intelligently, fairly and at scale. That control plane is the product. The four problems Armada names are all scheduling problems rather than execution problems. Kubernetes has no concept of ordering, so jobs compete for resources with no fairness guarantees. Each cluster is an island, so there is no multi-cluster coordination. Distributed jobs that need all their workers to start together, the MPI, PyTorch and Spark shape, are not scheduled as a unit. And one team can starve everyone else, because there is no fairness across teams. Armada's answer in each case is a layer of scheduling policy that Kubernetes does not have.

## Fair-share, preemption and gang scheduling are the mechanisms

The feature table is short and each row maps to one of those gaps. Multi-cluster scheduling gives one API across unlimited Kubernetes clusters. Fair-share queuing is described as dominant resource fairness across teams and queues, which is the mechanism that stops one team from permanently dominating shared infrastructure. Gang scheduling gives atomic startup for distributed workloads, so a distributed job is either fully scheduled or held in queue rather than starting half its workers and deadlocking. Preemption means urgent jobs bump lower-priority work automatically. Prometheus metrics provide full observability into queue health and cluster utilisation. The Lookout UI is a web interface for monitoring jobs, queues and clusters. And enterprise-ready is listed as secure and highly available with OIDC authentication support. The README claims the gang scheduling implementation is battle-tested at scale with deep fairness and preemption integration, and it reports that Armada has run in production at G-Research since 2020 processing millions of batch jobs per day across tens of thousands of nodes. That is a vendor statement rather than a measurement you can reproduce, but it is the reason the project exists in this form.

## Gang scheduling is the piece that changes the failure mode

Of the mechanisms, gang scheduling is the one with the most visible effect on how your jobs behave, and it is worth separating from the rest. A distributed training job needs every worker up before any of them can do useful work. On plain Kubernetes, the pods are scheduled independently, so a job with sixteen workers can land eight workers on one cluster and the rest on another, or land eight workers and wait indefinitely for a ninth while holding resources the whole time. The failure is not a crash, it is a stall that consumes quota. Armada's gang scheduling makes the set of workers a scheduling unit: either the whole set is placed, or the whole set stays in the queue. For MPI, PyTorch and Spark workloads that is the difference between a job that starts and a job that hangs. The README's use case list names the same shape from several angles, with ML and AI training described as distributed GPU training with gang scheduling across clusters, and SLURM migration described as familiar scheduling semantics, queues, priorities and preemption, on Kubernetes-native infrastructure. The gang property is what makes the migration story plausible rather than aspirational.

## Getting a local cluster going with the operator

The fastest local route goes through the separate Armada Operator repository, and it is three commands:

```bash
git clone https://github.com/armadaproject/armada-operator.git
cd armada-operator
make kind-all
```

The target is a Kind cluster, so the whole stack runs inside Kubernetes on your own machine. The CLI is installed as a side effect of that target, and to install it on its own, or on a machine without the full setup, the README gives a script and a release page:

```bash
# download via script
scripts/get-armadactl.sh

# or grab the binary from the release page
https://github.com/armadaproject/armada/releases/latest
```

There is also a Helm chart published to Artifact Hub under the gresearch repository, which is what a production install would use, and a full quickstart guide on the project site. The important structural point about the local setup is that the developer path and the operator path are different, and the developer path is the one that tells you what the system is made of.

## Local development runs services as host processes

Armada runs locally through Goreman, with the dependencies, Redis, Postgres and Pulsar, running in containers while the Armada components themselves run as host processes built from source, so iteration is fast and debuggers attach directly. The commands are driven by mage targets:

```bash
mage kind                  # one-time: create local Kubernetes cluster
export KUBECONFIG=.kube/external/config

mage dev:up auth           # with OIDC via Keycloak
mage dev:up fake-executor  # no Kubernetes cluster needed
mage dev:down              # stop dependency containers
```

Three variants of dev:up are worth noting. The default runs with no auth at all, which is convenient and also means you are not exercising the authentication path you will need in production. The auth variant brings up OIDC through Keycloak, which is the closest local approximation to a real deployment. And the fake-executor variant removes the need for a Kubernetes cluster entirely, which tells you the executor interface is separable from the scheduler, and gives you a way to work on queueing behaviour without the cluster in the way. The repository layout backs this up, with magefiles/ at the top level, a _local/ directory, a plugins/ directory, an e2e/ directory and a testsuite/ directory alongside internal/, pkg/, cmd/ and client/.

## The dependency list shows the shape of the system

The go.mod is a good map of what Armada is made of, and several entries are load-bearing. The Apache Pulsar client at 0.21.0 sits at the top of the list, and Pulsar is one of the three dependencies named in the local development section, which means the queue and the event stream between components run on it. Postgres and Redis complete the set, with jackc/pgx and pgxpoolprometheus for the database layer and patrickmn/go-cache for caching. Authentication uses coreos/go-oidc at 3.21.0, matching the OIDC support listed in the feature table, and the API surface is a gRPC service with the grpc-gateway providing the HTTP/JSON layer on top, plus go-openapi for the specification and validation. Prometheus client_golang is there for the metrics, and hashicorp/go-memdb and go-lru suggest an in-memory index layer for fast lookups on hot paths. The module targets go 1.27.1, which sets the floor for anyone building from source, and the build uses Mage, with magefile/mage as a dependency.

## Where a second scheduler is the wrong answer

The honest case against Armada is that it adds a scheduler between you and Kubernetes, and if your problem is throughput rather than fairness, that is the wrong place to intervene. The README's own framing implies the limit. If your jobs fit in one cluster, Kubernetes Jobs plus a decent priority class may be enough, and Armada's queue, fair-share, preemption and cross-cluster routing would all be machinery you operate without benefit. The cost side is concrete. Pulsar, Postgres and Redis are now three stateful services to run in addition to Kubernetes, and Lookout is another component in front of the operator. The multi-cluster story also assumes the clusters are not independently managed, since a single API routing to many clusters has to own the decision, and any cluster you keep outside that API is outside the fairness policy. Where Armada earns its keep is the case it names, where job volume exceeds a single cluster and where several teams share the same infrastructure and need neither to starve the other. The use case list points the same way, with quantitative finance and HPC, ML training, multi-tenant platform engineering, SLURM migration and CI/CD at scale.

## Conclusion

Adopt Armada when your batch job volume has outgrown a single Kubernetes cluster, when you need priority classes, fair-share and preemption across teams, and when you can operate Pulsar, Postgres and Redis alongside it. Do not adopt it for a workload that fits comfortably in one cluster with ordinary Kubernetes Jobs, because the scheduler layer is real complexity you take on for no gain there. Verify first that the workload is genuinely batch, that your teams need fairness between them rather than just more throughput, that the Lookout interface and Prometheus metrics fit your monitoring stack, and that the OIDC path is configured before you expose the API to more than one team.

## FAQ

### What problem does Armada solve that Kubernetes does not?

Kubernetes has no job queue, so jobs compete for resources with no fairness guarantees, and each cluster is an island with no multi-cluster coordination. Armada adds a prioritised queue with fair-share and rate limiting, routes jobs across clusters from a single API, and adds gang scheduling so distributed jobs either start all their workers or stay in queue.

### How do I run Armada locally?

Clone the armada-operator repository and run make kind-all, which creates a Kind cluster and installs armadactl as well. For development on the components themselves, Armada runs through Goreman with Redis, Postgres and Pulsar in containers and the components as host processes, driven by mage dev:up, with an auth variant for OIDC via Keycloak and a fake-executor variant that needs no Kubernetes cluster.

### What is gang scheduling in Armada and when does it matter?

Gang scheduling gives atomic startup for distributed workloads, so a job that needs all its workers to start together is either fully scheduled or held in queue. The README names MPI, PyTorch and Spark as the workloads it targets, and describes the effect as avoiding jobs that start partially and stall while holding resources.

### How is Armada deployed in production?

The README documents a Helm chart published to Artifact Hub under the gresearch repository, and a full quickstart guide at armadaproject.io/getting-started. For the components themselves, the dependencies are Redis, Postgres and Pulsar, with Prometheus metrics for queue health and cluster utilisation and the Lookout web interface for monitoring jobs, queues and clusters.

## Sources

- [armadaproject/armada on GitHub](https://github.com/armadaproject/armada)
- [License: Apache-2.0](https://github.com/armadaproject/armada/blob/master/LICENSE)
- [Project website](https://armadaproject.io)
- [README](https://github.com/armadaproject/armada/blob/master/README.md)
- [Releases](https://github.com/armadaproject/armada/releases)

---

Hysen Labs editorial analysis, written from the project's own repository and release notes. Cite the canonical page: https://hysenlabs.com/projects/armadaproject-armada