Escalator: an autoscaler for Kubernetes batch workloads

The batch workload is the reason this autoscaler exists

Escalator is described as a batch or job optimized horizontal autoscaler for Kubernetes, and the design follows directly from the shape of the workload it serves.

The problem is stated in the readme: it is designed for large batch or job based workloads that cannot be force-drained and moved when the cluster needs to scale down. A service pod can be rescheduled onto another node, but a batch job that has been running for hours, or a pod holding a local disk, cannot simply be killed and moved. So Escalator ensures pods have completed on nodes before terminating them, and it is also optimised for scaling up the cluster as fast as possible to ensure pods are not left in a pending state.

That is two problems in one tool, and both come from the same source. The readme says the need for this autoscaler is derived from Atlassian's own experiences with very large batch workloads being scheduled and the default autoscaler not scaling up the cluster fast enough. These workloads cannot be force-drained by the default autoscaler and must complete before the node can be terminated.

So this is a tool built from a specific operating experience rather than a general purpose autoscaler with a different algorithm. The audience is anyone running a Kubernetes cluster where jobs take minutes or hours and where a slow scale up means people wait.

The licence is Apache 2.0, with the copyright line covering 2018 Atlassian and others, and the primary language is Go.

It shares the cluster with the default autoscaler rather than replacing it

The architectural decision that matters most for deployment is that Escalator does not take over the whole cluster. It is designed to work on selected auto-scaling groups so that the default Kubernetes Autoscaler continues to scale service based workloads.

That division of labour is the right answer for a mixed cluster. The default autoscaler is good at the service case, where pods come and go quickly and any pod can be replaced anywhere. Batch nodes are a different animal, and giving the two problems to two different controllers avoids the default autoscaler force-draining a long running job.

The node group configuration is what selects the split. The readme's quick start shows it being passed as a file:

go run cmd/main.go --kubeconfig=~/.kube/config --nodegroups=nodegroups_config.yaml

Other operational details in the feature list are worth noting because they are the parts that usually go wrong in an autoscaler. It automatically terminates the oldest nodes first, which is a policy decision with real consequences for long jobs, since the oldest node is likely holding the oldest and longest running work. It supports slack space to ensure extra space in the event of a spike of scheduled pods, which is the cheap insurance against a queue forming.

And it does not terminate or factor cordoned nodes into calculations, which allows cordoned nodes to persist for debugging. That is a genuinely useful property. Cordoning a node to inspect what went wrong is a routine incident response step, and an autoscaler that immediately counts that node as reclaimable capacity makes debugging harder at exactly the wrong moment.

Requests, capacity and the decision to scale up, down or stay

The scaling logic is described at a high level in the readme: Escalator calculates requests and capacity to determine whether to scale up, down or to stay at the current scale. Scaling and utilisation metrics are also listed as supported.

That is the whole algorithm description given, and the documentation lives in the docs directory rather than in the readme, with a separate design page, a configuration page and a deployment page. That split is normal for a project with this much configuration surface, and it means the readme is an overview rather than a specification.

What can be said from the readme is the shape of the inputs and the intent of the outputs. Requests and capacity are the two quantities being compared, so the decision is a comparison of what the workloads have asked for against what the node group can provide. The three way outcome, up, down or stay, is what you would expect from an autoscaler that has to reason about a job queue rather than a set of steady state services.

Two other capabilities show up in the feature list and are less common in homegrown autoscalers. There is leader election, so you can run a highly available deployment inside the cluster, which matters because an autoscaler that loses its leader election mid-decision can double scale. And there is basic support for multiple different types of instances in a node group, which is a practical need when a cluster mixes general purpose and memory optimised nodes.

Building, testing and running the binary

The build is a Makefile with the usual targets. Fetching dependencies and building is a single command:

# Fetch dependencies and build Escalator
make build

And the test target runs the whole suite with coverage and the race detector enabled:

make test

If you are working on one area, you can run a single package. The readme uses the controller package as its example:

go test ./pkg/controller

The Makefile also has a test-vet target running go vet across the tree, a docker target using buildx with a platform argument, a clean target and a lint target running golangci-lint. The build rule itself sets GOARCH from the target platform when one is supplied and defaults to amd64, and it builds with CGO_ENABLED=0 by default, which the ENVVAR variable can override.

The Dockerfile does the same thing in two stages. The builder stage uses golang:1.26 with the build platform argument, copies go.mod, go.sum, the Makefile, cmd and pkg, and runs make build. The runtime stage is alpine with ca-certificates added and the compiled binary copied to ./main, with the command set to run it directly.

The requirements are Kubernetes 1.24 or newer, with the readme noting that older versions may have bugs or issues that prevent it working properly, and Go 1.26 or newer. The pinned dependencies are in go.mod and go.sum, and the module requires the Kubernetes client libraries at v0.32.3, the AWS SDK, kingpin for the command line, Prometheus client for the metrics, logrus for logging and testify for the tests.

Running it in a cluster, and the AWS limit

The in-cluster path is a Docker build plus three kubectl applies, one for RBAC, one for the config map and one for the deployment:

# Build the docker image
docker build -t atlassian/escalator .

# Create RBAC configuration
kubectl create -f docs/deployment/escalator-rbac.yaml

# Create config map - modify to suit your needs
kubectl create -f docs/deployment/escalator-cm.yaml

# Create deployment
kubectl create -f docs/deployment/escalator-deployment.yaml

The comment on the config map step is worth reading literally: modify it to suit your needs. The RBAC file, the config map and the deployment are all in the repository under docs/deployment, and the config map is where the node group definitions and the credentials live, so it is the file you will be editing for every environment.

The AWS limitation is the one hard boundary. Support for different cloud providers is listed as AWS only at the moment, and the go.mod confirms the shape of that dependency with the AWS SDK pulled in at v1.55.7 alongside the Kubernetes client libraries. There is no abstraction over the cloud API in the dependency list, so this is not a case of a provider plugin you can add.

That rules Escalator out for teams on GKE, AKS or EKS with a non-AWS substrate, and it is worth checking before anything else, because the rest of the design is provider independent in principle. The node termination and drain logic is generic Kubernetes work, so a port to another provider would mostly mean replacing the instance and auto-scaling-group calls, but that port is not in this repository.

Contributor expectations and what a fork would cost

The contributing section is short but specific about what a good pull request looks like. New features and bug fixes come with tests, the existing style is followed since the project uses goreturns to format and lint, and unrelated changes are separated into multiple pull requests. For bigger changes the project asks you to start a discussion first by creating an issue and explaining the intended change.

That last point is a reasonable request for an autoscaler, because a change in the scaling decision logic is hard to review in isolation and hard to test without a cluster. The controller package tests are where you would look to understand the expected behaviour of the node group configuration.

The Contributor License Agreement is a requirement, with separate corporate and individual links, and the explanation given is that the CLA records the contributor as entitled to contribute the code, documentation or translation and as willing to have it used in distributions and derivative works.

The repository also carries a CODE_OF_CONDUCT.md and a CONTRIBUTING.md, and the top level is small and conventional: cmd for the entry point, pkg for the packages, docs for the documentation, a Makefile, a Dockerfile, go.mod and go.sum, plus a .dockerignore and .gitignore.

For a team considering running a modified version, the practical picture is a small Go codebase with a clear package structure, a documented config map and RBAC pair in the repository, and a permissive licence that allows forking. The pieces that would need work in a fork are the cloud provider integration and the documentation, which lives in the docs directory and would have to be maintained alongside the code.

Editorial conclusion

Adopt Escalator if you run large batch or job based workloads on AWS Kubernetes where pods cannot be force drained and the default autoscaler is not scaling up fast enough, and you can accept splitting responsibility with the default autoscaler across node groups. Do not adopt it if you are on a cloud provider other than AWS, since support for different providers is listed as AWS only at the moment, or if your workload is service shaped, because the default autoscaler already handles that case. Before you commit, read the docs directory for the design and configuration pages, check that your cluster is on Kubernetes 1.24 or newer, and run the controller package tests to see how the node group configuration is expected to behave.

Frequently asked questions

Official sources

1. atlassian/escalator on GitHub
2. Issues
3. License: Apache-2.0
4. README
5. Releases