AWS VPC CNI: pod networking on Elastic Network Interfaces

The problem: pods that need real VPC addresses

Most CNI plugins solve pod addressing by building an overlay. Pods get addresses from a private range, packets get encapsulated, and the underlying network only ever sees node traffic. That works anywhere, but it hides pods from the VPC. A security group cannot reference a pod. A VPC flow log shows node addresses, not workload addresses. A route table entry cannot point at a pod.

amazon-vpc-cni-k8s takes the opposite approach. The README describes it as a networking plugin for pod networking in Kubernetes using Elastic Network Interfaces on AWS. Pods receive addresses from the ENI's own subnet, so a pod IP is a VPC IP. Anything that already reasons about VPC addresses keeps working without translation. The cost is that pod density is bounded by the instance type's ENI limits, and the plugin only makes sense on AWS EC2 nodes.

This is aimed at platform teams running Kubernetes on EC2, and at EKS users who never chose a CNI because the default already is this one. If your cluster runs on Fargate, on-premises hardware, or another cloud, the model does not apply.

Two components: the CNI binary and the ipamd daemon

The repository splits the work in two. The CNI plugin is the binary the container runtime invokes when a pod sandbox is created; it wires up the host and pod network stack at that moment. It is short-lived and stateless between calls.

ipamd is the long-running, node-local IP Address Management daemon. The README assigns it two jobs: maintaining a warm pool of available IP addresses, and assigning an IP address to a pod. Because it runs continuously, it can call EC2 to attach ENIs and assign secondary addresses ahead of demand, so a pod start does not wait on an API call.

The warm pool is the design decision worth understanding. On join, a node has one ENI. Without configuration, ipamd always tries to keep one extra ENI. When running pods exceed the addresses on a single ENI, the README's allocation scheme kicks in: between 0 and 29 pods ipamd allocates one more ENI, giving a warm pool of 2 ENIs times 29 addresses, or 58; between 30 and 58 pods it allocates two more, for 3 ENIs times 29, or 87. Those numbers assume the example ENI shape of 30 addresses. Your instance type sets the real ceiling, which is why the README points at vpc_ip_resource_limit.go and at the WARM_ENI_TARGET, WARM_IP_TARGET and MINIMUM_IP_TARGET knobs.

Pre-allocating addresses is what makes pod startup fast. It also means a node holds IPs it is not using, and those addresses are unavailable to other nodes in the subnet. On a small subnet with several nodes, that waste is the first thing you will notice.

Installing the manifest and setting kubelet flags

The README's setup path is short. Download the latest YAML from the config directory and apply it:

kubectl apply -f aws-k8s-cni.yaml

After that, the kubelet must be launched with the CNI network plugin enabled and the CNI directories configured. The default manifest expects the config directory at /etc/cni/net.d and the binary directory at /opt/cni/bin, and the README notes the node IP should be the primary IPv4 address of the primary ENI:

--network-plugin=cni \
--cni-config-dir=/etc/cni/net.d \
--cni-bin-dir=/opt/cni/bin \
--node-ip=$(curl http://169.254.169.254/latest/meta-data/local-ipv4)

The README also recommends setting max-pods to (number of ENIs for the instance type times (IPs per ENI minus 1)) plus 2. That formula is not decoration. If you leave max-pods at a value larger than the node can actually address, the scheduler will place pods the kubelet cannot give an IP to, and they will sit pending. The count of ENIs and IPs per ENI comes from the EC2 documentation on Elastic Network Interfaces, and the repository encodes the per-instance-type values in pkg/vpc/vpc_ip_resource_limit.go.

There is also a Helm chart at eks/aws-vpc-cni if you prefer that route. The README points to docs/iam-policy.md for the IAM permissions the daemon needs; the daemon calls EC2, so an incomplete policy shows up as ENI attachment failures rather than a clean error at install time.

ENI allocation is a capacity plan, not a background detail

The allocation scheme in the README is easy to skim past, and it determines how your cluster behaves under load. A node does not fill one ENI before touching the next; it adds ENIs in steps, and each step multiplies the warm pool. The README's own example is a m4.4xlarge, which can have up to 8 ENIs with up to 30 IP addresses each. That is a hard ceiling on pod count per node unless you use a different addressing mode.

The practical consequence: pod density is a property of the instance type, not of CPU and memory. A node with plenty of spare CPU can still refuse new pods because it cannot attach another ENI or assign another secondary IP. Cluster autoscaler decisions that ignore this will overshoot and undershoot.

The warm pool also interacts with subnet sizing. Since pods take addresses from the ENI subnet, a subnet that comfortably holds your nodes may not hold nodes plus warm pools plus running pods. Teams that size subnets only for nodes tend to discover this at the worst moment.

On the tuning knobs, the README defers to docs/eni-and-ip-target.md for WARM_ENI_TARGET, WARM_IP_TARGET and MINIMUM_IP_TARGET. The README itself does not explain the interaction between these three, so read that document before changing defaults rather than guessing at values.

Network policy support and the privileged containers behind it

VPC CNI versions v1.14 and greater implement the Kubernetes NetworkPolicy API, with the usual pod selector plus ingress and egress rules. By default all pod-to-pod communication is allowed, and policy objects restrict it.

The implementation detail that matters for cluster hardening is in the README's privileged mode section. The manifest runs the aws-vpc-cni-init and aws-eks-nodeagent containers with privileged: true. The init container needs elevated privilege to set networking kernel parameters; the node agent needs it to attach BPF probes that enforce network policy. So enabling policy enforcement is not a pure control-plane change. It puts a privileged container on every node.

For EKS clusters the README directs readers to the EKS User Guide page on configuring a cluster for Kubernetes network policies. For self-managed clusters, the README states that the AWS VPC CNI implementation of network policies may be enabled, and that this requires both the VPC CNI agent and the network policy controller. The README's sentence on this is truncated, so treat the exact self-managed enablement steps as something to confirm in the EKS User Guide and the amazon-network-policy repository rather than in this file.

If you run a version below v1.14, NetworkPolicy objects are accepted by the API server and ignored by the dataplane. That is a silent failure mode worth checking before you rely on policy for isolation.

Upgrades, state, and what the README does not promise

The README makes a specific claim about version changes: upgrading or downgrading should result in no downtime, existing pods should not be affected and will not lose network connectivity, and new pods will be pending until the CNI is fully initialized and can assign pod IP addresses. The pending window is the observable part of an upgrade, and it is expected rather than a fault.

State handling changed at v1.12.0. From that version on, VPC CNI state is restored via an on-disk file at /var/run/aws-node/ipam.json. In lower versions, state is restored through calls to the container runtime. That difference matters if you build node images or run node-replacement workflows: the file path is part of the node's persistent state, and a node that comes back without it takes the slower restore path.

What the README does not document is rollback. There is no stated procedure for reverting a failed CNI upgrade beyond the general no-downtime claim, and no guidance on what happens to the ipam.json contents when you move between versions that use different restore mechanisms. Plan a node-level rollback strategy yourself, and test it, because the project does not hand you one.

The recommended-version table is the other thing to read before upgrading. It lists the oldest recommended VPC CNI version per Kubernetes release: v1.17.1 or greater for 1.33 and 1.32, v1.16.4 or greater for 1.31, v1.16.0 or greater for 1.30, v1.14.1 or greater for 1.29, and v1.13.4 or greater for 1.28. The README states that for all Kubernetes releases it recommends installing the latest VPC CNI release. That table is a floor, not a target.

Where it is the wrong choice, and what to compare it against

The clearest boundary is the platform. This plugin uses Elastic Network Interfaces, so it requires EC2 instances in an AWS VPC. On-premises clusters, other clouds, and Fargate nodes cannot use it. If your cluster is not on EC2, stop here.

The second boundary is addressing. Because pods consume VPC addresses, you inherit VPC address-space limits and subnet sizing as cluster constraints. If you want pod addressing fully decoupled from the VPC, an overlay CNI gives you that decoupling by design. The trade is that pod IPs stop being meaningful to VPC-native tooling.

The comparison people search for is AWS VPC CNI versus Cilium. The difference in approach is structural. VPC CNI is an ENI and IPAM manager first: its job is to attach ENIs, keep a warm pool, and hand out addresses that the VPC already routes. Cilium is built around an eBPF dataplane and its own identity model, and it can run as an overlay independent of the underlying network. Choosing VPC CNI means accepting the instance-type pod ceiling in exchange for pods that security groups and VPC flow logs see directly. Choosing Cilium means giving up that direct visibility in favor of a dataplane that is not bounded by ENI limits. Neither is a default; they encode different assumptions about what the network should know about pods.

Network policy is a second axis. VPC CNI only enforces policy from v1.14 onward, and doing so requires the privileged node agent described above. If policy enforcement is a hard requirement on day one, verify the version and the controller before you commit.

Editorial conclusion

Adopt it if your nodes are EC2 instances in one VPC and you want pods to hold real VPC addresses that security groups, VPC flow logs and route tables already understand. Do not adopt it for on-premises clusters, for clusters that need an overlay independent of VPC address space, or where a service mesh already owns pod addressing. Before you apply the manifest, confirm three things: your instance type's ENI and IP-per-ENI limits against the max-pods formula, the IAM policy in docs/iam-policy.md, and whether you need NetworkPolicy, which requires v1.14 or greater plus the network policy controller.

Frequently asked questions

Official sources

1. aws/amazon-vpc-cni-k8s on GitHub
2. Issues
3. License: Apache-2.0
4. README
5. Releases