kubernetes-on-aws: CloudFormation templates shaped by one company's use case

Templates for one platform, not a general purpose installer

The repository contains configuration templates to provision Kubernetes clusters on AWS using CloudFormation and Ubuntu Linux, and the README marks it WORK IN PROGRESS in bold at the top. That framing matters more than the feature list. The next paragraph says it plainly: many values are parameterised and values are not always visible, and the project is focused on solving the team's own, specific use case. Only after that does it say the team is open to ideas from the community about turning the idea into a project with universal value. The documentation site at kubernetes-on-aws.readthedocs.io is where the extracted internal documentation lives, and the README points to a Zalando case study on kubernetes.io and to a document on running Kubernetes in production on AWS for the setup details. The primary language of the code is Go, the licence is MIT, and the default branch is dev rather than main.

Four control components, and not all of them are open

The configuration was initially based on kube-aws, and it now depends on four components that are not all open sourced. The Cluster Registry keeps the desired state of a cluster, such as the configuration channel and the version to use. The Cluster Lifecycle Manager provisions the cluster's CloudFormation stack and applies the Kubernetes manifests for the system components. The Cluster Lifecycle Controller handles rolling updates from inside the cluster, for example node termination. And the Authnz Webhook validates OAuth tokens and authorises access. This is the single most important fact for anyone reading the repository. The templates are not self-contained, and the pieces that drive them, the registry that holds the desired state and the manager that applies it, are internal components. A team that copies the CloudFormation files without those pieces gets templates that describe a cluster but no pipeline to reconcile it.

What sits in the cluster/ directory

The directory layout explains how a cluster comes into existence. `cluster/cluster.yaml` holds the CloudFormation template files for the cluster, and they are applied by the Cluster Lifecycle Manager. `cluster/config-defaults.yaml` holds default values for different kinds of use, which can be overridden by values from the cluster registry, again applied by the manager. `cluster/etcd-cluster.yaml` is Senza CloudFormation to deploy etcd, so the etcd layer is provisioned by the same templating approach rather than by a separate mechanism. `cluster/manifests` holds the Kubernetes manifests for the system components, and `cluster/node-pools` holds CloudFormation template files and userdata, in the form of cloud-init, for ContainerLinux node pools. The docs directory holds extracts from the internal Zalando documentation. Note the two node operating systems in play: the templates target ContainerLinux nodes, while the project description names Ubuntu as the operating system of the images it builds, and the environment an AWS account and a VPC you already own.

Two networking profiles for two cluster types

The feature list names two cluster flavours and two different overlay approaches, which is the clearest sign that this repository encodes an existing platform rather than a product. One cluster type, zalando-aws, uses Flannel overlay networking, and one, zalando-eks, uses the aws-vpc-cni network. Beyond that, the infrastructure covers the operational ground rather than the cluster API. Master nodes are highly available, in an Auto Scaling Group behind an Elastic Load Balancer, and there is a worker Auto Scaling Group with node pool support, plus cluster autoscaling through the upstream cluster-autoscaler. DNS is handled by Kubernetes DNS with node-local dnsmasq as a daemonset and the CoreDNS resolver for the cluster.local domain running in the same pod, and Route53 integration arrives through External DNS. AWS IAM integration is handled by kube2iam and by AWS OIDC IAM, and standard components such as node exporter and kube-state-metrics are installed from the manifests directory.

Ingress, egress and the routing layer

Traffic in and out of the cluster is handled with a specific set of controllers, and the combination tells you what this platform was built for. Ingress support is full, with ALB and NLB, TLS integration through kube-ingress-aws-controller, and HTTP routing through skipper. East-west ingress is set up through a coredns template with a fixed IP in a service of type clusterIP. Outbound traffic has its own controller, kube-static-egress-controller, which routes through NAT Gateways with Elastic IPs to give static egress IPs. Managed stacks and blue-green deployments come from stackset-controller and skipper, and a Fabric API Gateway can be combined with stackset-controller. Autoscaling is covered on both axes: horizontal pod autoscaling scaled by requests per second, SQS queue size or other custom metrics via kube-metrics-adapter, and vertical pod autoscaling for something like Prometheus. EFS and GPU support are also listed, and etcd backup runs as a Kubernetes cronjob using etcdctl snapshot and an upload to S3.

Token authentication and the emergency access roles

Access control is described in more detail than most of the feature list. Node and user authentication is done via tokens, using the webhook feature, and SSL client-certificate authentication is disabled, which removes a mode of authentication that some environments rely on. The webhook grants roles named ReadOnly, PowerUser, Manual, Emergency and Administrator. There is also an emergency access path through an internal emergency-access-service, which grants the Manual and Emergency roles under a four-eyes principle with audit logging, and an Authnz Webhook that validates OAuth tokens and authorises access, with an OAuth Token Info service expected to be reachable. Configuration management is described as being based on ConfigItems, overridden through an internal configitems repository and a cluster-registry CRUD service. The pattern to notice is that authorisation decisions are pushed into a webhook rather than evaluated by Kubernetes RBAC alone.

The assumptions list is the actual specification

Before the feature list, the README states the assumptions, and they are worth more attention than the features. The AWS account must have one or more hosted zones in Route53 including a proper SSL certificate, which the free ACM service can provide. The VPC must have at least one public subnet per availability zone, either from the AWS default VPC setup or with a public subnet named dmz-<REGION>-<AZ>. The VPC must be in eu-central-1, eu-west-1 or eu-north-1. The etcd cluster must be available via DNS discovery, with SRV records at etcd.<YOUR-HOSTED-ZONE> for the zalando-aws cluster, and an OAuth Token Info endpoint must be available to validate user tokens. A restricted region list, a naming convention for subnets and a DNS layout assumption are the kind of constraint that decides adoption faster than any feature, because a team outside those regions has to change the templates rather than configure them.

Where kubernetes-on-aws is the wrong tool

Three things disqualify it for most readers. First, it is not a cluster installer you can run on a fresh account. It depends on a registry, a lifecycle manager, a lifecycle controller and an authorisation webhook, and the README states that not all of them are open sourced, so the repository is the visible half of a system whose other half lives elsewhere. Second, the notes section lists the rough edges without softening them: SSL client-certificate authentication is disabled, many values are hardcoded, and secrets, for example a shared token, are not KMS-encrypted in the cluster. Third, the project status is a work in progress on a dev branch with no GitHub releases, so there is no version to pin. The alternative approach for a team that simply needs Kubernetes on AWS is to use a managed control plane, or a mainstream installer with a documented upgrade path. The value of this repository is different: it is a readable record of how a platform team provisions and operates clusters at scale, and it is worth reading for that, with the understanding that you are reading configuration rather than installing software.

Editorial conclusion

kubernetes-on-aws fits an engineer who wants to read how a large platform team provisions Kubernetes on AWS, or who works in an environment that already has the cluster registry, the cluster lifecycle manager and the authorisation webhook in place, since the templates assume those components exist. It does not fit a team that wants a self-contained cluster installer, and it should not be treated as a general purpose tool: the README marks it work in progress, says many values are hardcoded, and notes that secrets are not KMS-encrypted in the cluster. Before adopting any part of it, read the assumptions section, check that your VPC is in one of the three named regions with the expected public subnet naming, and decide who owns the four components that are not open sourced.

Frequently asked questions

Official sources

1. Issues
2. License: MIT
3. Project website
4. README
5. zalando-incubator/kubernetes-on-aws on GitHub