Demystifying AWS EKS CNI end to end

Today we will cover one of the most important topic in EKS networking as CNI (Container Network Interface) . So let's start from scratch .

1. What is CNI ?
2. Key Components of the CNI Architecture in EKS
3. How CNI Architecture Works in EKS ?
4. Advantages of CNI in EKS cluster
5. What is Prefix Delegation ?

What is CNI ?

The copy book definition illustrates as the AWS VPC CNI plugin, which is the default networking plugin for EKS, integrates deeply with Amazon VPC to leverage AWS networking features and also ensure the robustness ,scalable & seamless networking for Kubernetes pods .

Key Components in CNI :-

1. Amazon VPC CNI Plugin:

• aws-vpc-cni-k8s: The primary component that manages the networking for pods within an EKS cluster. It assigns VPC IP addresses to Kubernetes pods and configures the necessary network interfaces.
• ipamd (IP Address Management Daemon): A component of the VPC CNI plugin that runs on each node. It manages the allocation and deallocation of IP addresses for pods from the VPC subnet.

Elastic Network Interfaces (ENIs):

• ENIs are virtual network interfaces that can be attached to an EC2 instance (node). Each ENI can have one primary IP address and multiple secondary IP addresses.
• The CNI plugin attaches one or more ENIs to each node and assigns secondary IP addresses from these ENIs to the pods running on the node.

Please note that there is a limitation of ENI associate with EC2 instance based on its type . Follow the below table illustrates the same .

• VPC Subnets :-

The IP address sourced from VPC subnets CIDR block on where the nodes associated with . This integration ensures that those pods associate with nodes can route traffic within the VPC .

Prefix Delegation (Optional):

Allows the allocation of larger prefixes to ENIs, which can then be subdivided into individual IP addresses for pods. This enhances IP address utilization and increases the maximum number of pods per node.

How CNI Architecture Works in EKS

1. Node Initialization:

• When a new node (EC2 instance) is added to the EKS cluster, the aws-vpc-cni-k8s plugin initializes and attaches one or more ENIs to the node.
• The primary IP address of the node is assigned from the subnet's CIDR block.

1. IP Address Allocation:

• The IP Address Management Daemon (ipamd) requests secondary IP addresses for the ENIs attached to the node. These IP addresses are also sourced from the VPC subnet's CIDR block.
• Optionally, if prefix delegation is enabled, larger prefixes are allocated to ENIs, which can then be subdivided into multiple IP addresses for pods.

1. Pod Network Configuration:

• When a new pod is scheduled on the node, the aws-vpc-cni-k8s plugin assigns one of the secondary IP addresses from the ENIs to the pod.
• The plugin configures the pod's network interface, ensuring it is connected to the VPC network and can communicate with other pods, services, and AWS resources.

1. Routing and Security:

• The pod's IP address is part of the VPC network, which allows it to communicate with other pods across the cluster, VPC endpoints, and AWS services using the same network.
• Security Groups and Network ACLs (NACLs) applied to the ENIs govern the traffic to and from the pods, providing an additional layer of security.

Benefits of CNI Architecture :-

Scalability:

• The CNI plugin leverages ENIs and secondary IP addresses to efficiently manage pod IP addresses, allowing for high-density pod deployments.
• Prefix delegation further enhances scalability by increasing the number of pods per node.

Integration with AWS Services:

Pods can interact with other AWS services (S3,DynamoDB , RDS) using the VPC network and connected with VPC with the assist from NACL & Security group .

Performance:

• Network traffic for pods stays within the VPC, benefiting from the high performance and low latency of AWS's networking infrastructure.

• Security:

Security groups and NACLs provide fine-grained control over the traffic to and from pods, ensuring a secure network environment.

• Simplicity:

The VPC CNI plugin abstracts the complexity of networking, making it easier to manage and deploy Kubernetes clusters on AWS.

What is Prefix Delegation ?

With prefix delegation enabled, you can allocate larger CIDR blocks to ENIs, effectively increasing the number of IP addresses (and thus pods) that can be assigned per node. For example, if you allocate a /28 prefix to each ENI, you can significantly increase the IP address capacity beyond the default limits.

Now simplify the entire concept by examples and diagrams :-

Let’s consider an m5.large instance:

• ENIs per instance: 3
• IP addresses per ENI: 10
• Total IP addresses per instance: 30

This means an m5.large instance can host up to 30 pods (each requiring one IP address).

With Prefix Delegation

With prefix delegation, instead of assigning individual IP addresses to each pod, a larger prefix (subnet) is allocated to each ENI, and the CNI plugin then assigns IP addresses from this prefix to the pods. This can increase the number of IP addresses (and thus pods) that can be assigned per ENI.

Example Setup

Let’s assume the following:

• Each ENI can be assigned a /28 prefix.
• A /28 prefix provides 16 IP addresses (14 usable for pods after excluding network and broadcast addresses).

Calculations

Without Prefix Delegation

1. ENIs per instance: 3
2. IP addresses per ENI: 10
3. Total IP addresses per instance: 30

An m5.large instance can host up to 30 pods.

With Prefix Delegation :-

1. ENIs per instance: 3
2. IP addresses per ENI: 14 (from a /28 prefix)
3. Total IP addresses per instance: 3 ENIs * 14 IP addresses = 42 IP addresses

Therefore additional 12 pods the nodes can be scheduled and it ensures the proper scalability of nodes and pods on where the scenario comes of Pod density in nodes .

Conclusion :-

AWS CNI plugin ensures to establish seamless networking policies which enhances robust and scalable networking solution increases efficiency & performance towards EKS cluster .