Kubernetes Docker Networking: Use Cases and Design Patterns

Today Docker is one of the most flexible and user-friendly container systems available in the cloud market because Docker Containerization is the best way of deploying applications quickly, cheaply and reliably. The networking aspects of containers primarily center around the use of the Linux kernel’s namespaces and complex applications that spread around multiple containers can be communication using Kubernetes supported plugins, Software Defined Networks (SDN) and other tools which supports to create an overlay network on top of an existing infrastructure.

Docker and Kubernetes Networking Fundamentals

Overlay and Underlay Network: Linux system networking can be defined in two ways: overlay or underlay and Kubernetes makes it possible to use both.

Overlay is a virtual network composed of vlan, veth (Virtual Interface) and VxLAN (Virtual Extensible LAN); Overlay network encapsulates the network trafic and its bit slower than underlay as it creates tunnels between hosts that reduces available MTUs (maximum transmission unit). Overlay helps in specific situations like when we don’t have enough IP space or network can’t handle the extra routes. A common use case is the limitation of how many routes the cloud provider route tables can handle. For example AWS route tables support up to 50 routes without impacting network performance but let us assume if we have more than 50 Kubernetes nodes then the AWS route table won’t be enough so using an overlay network will help. Underlay is defined at the physical level namely switchs, routers etc.

Network Namespace: It provides a way of having separate network stack for each instance of a container. Network namespaces are listed in /proc/<PID>/ns/net in Linux Systems.

NAT and IPtables: Network address translation is an IP-level remapping of one address space into another. The mapping happens by modifying network address information in the IP header of packets while they are in transit across a traffic routing device. IPtables is a policy engine in the kernel used for managing packet forwarding, firewall and NAT features. Docker manipulates iptables rules to provide network isolation on linux system.

Virtual Ethernet (Veth): It's a logical interface that acts as a connecting wire between two network namespaces. Docker network drivers utilize veths to provide explicit connections between namespaces when Docker networks are created. 

Docker0 Bridge: The docker0 bridge is the core part of the default networking and its created when the Docker service is started on the host machine. The interfaces on the containers talk to the bridge and the bridge proxies to the external world. Multiple containers on the same host can talk to each other through the Linux bridge.

Layer 2/4/7 Network: Layer 2 is the data link layer providing Node-to-Node data transfer; Layer 4 is the transport layer which controls the reliability of a given link through flow control; Layer 7 is the application layer is the layer closest to the end user, which means both the application layer and the user interact directly with the software application.

VXLAN (Virtual Extensible LAN): Virtual Extensible LAN (VXLAN) is a network virtualization technology that addresses the scalability problems across data center networks. VXLAN is an L2 overlay over an L3 network. Each overlay network is known as a VXLAN Segment and identified by a VXLAN Network Identifier. VXLAN mode is the default networking mode for host overlays in Docker Swarm mode.

Container Networking Standards:

There are two Network standards for configuring network interfaces for Linux containers namely container network model (CNM) designed for Docker and container network interface (CNI) for Kubernetes.

Container Network Model (CNM): The CNM is a standard proposed by Docker and adopted by projects such as Libnetwork which is a library used for creating and managing network stacks for containers. CNM has been used in integrations from companies such as Cisco Contiv, Kuryr, VMWare, Weave, Open Virtual Network “OVN” and Project Calico. On the other hand, CNI is a container networking standard proposed by CoreOS, which has been implemented by projects such as Apache Mesos, Cloud Foundry, rkt and Kurma. 

Container Network Interface (CNI): Its an interface that abstracts away the network stack from the container runtime ( eg: Docker ) and act as an interface between a network namespace and network plugin. It has 2 components namely NetPlugin ( responsible for setting up network plumbing ) and IPAM ( Responsible for IP address allocation ). The plugin exists as an executable and when invoked then it reads in a JSON config to get all the required parameters to configure the container with the network.

Docker Networking / Docker Native Network Drivers

Docker has two types of networks: single-host and multi-host. The Multi-host container networking pertains more to software like Kubernetes or Docker Swarm. Docker comes with three ways to network containers by default. There are other popular container networking methods for example Macvlan, IPvlan and VXLAN.

The basic network primitive in Docker is an Ethernet bridge called "docker0" and when Docker boots up on a Linux server then it creates this default docker0 bridge inside the Linux kernel. This docker bridge "docker0" creates a virtual subnet on the Docker host then It can pass packets back and forth between containers on that host. Docker also creates a pair of virtual Ethernet interfaces on each container, randomly assigning them an IP address and a subnet from a private address range not already used by the host machine. All the Docker Containers on docker0 can freely communicate with each other and controlled by start flag ‘icc=true’ (by default).

There are various ways in which container-to-container and container-to-host connectivity are provided and in this section I have discussed various Docker Network Drivers or Network Types currently available.

Bridged (default): This is the default network driver . Its An automatically generated network with a subnet and a gateway. Bridge networks are usually used when our applications run in standalone containers that need to communicate.

Host : This enables container to share the networking namespace of the host which means it is directly exposed to the outside world. There is no namespace separation and all interfaces on the host can be used directly by the container. So there’s no automatically port assignment neither it is routable but we need to use port mapping to reach services inside the container because configurations inside container are same as outside it.

None : The none driver gives a container its own networking stack i.e it lacks a network interface. The container-specific network stack which does not have an external network interface and it only has a local loopback interface.

Macvlan: Macvlan networks allow us to assign a MAC address to a container (each container receives a unique mac address ) making it appear as a physical device on our network. The Docker daemon routes traffic to containers by their MAC addresses. Using the macvlan driver is sometimes the best choice when dealing with legacy applications that expect to be directly connected to the physical network rather than routed through the Docker host’s network stack.

Overlay: Overlay networks connect multiple Docker daemons together and enable swarm services to communicate with each other.

The Kubernetes networking model 

Kubernetes manages containerized applications across a cluster of nodes and makes choices about how Containers, Pods & Hosts communicate. A pod consists of one or more containers that are collocated on the same host and shares a network stack.

Container-to-Container Communication: This happens within a pod and can be treated similarly to the localhost traffic.

Pod-to-Pod Communication: A pod is the smallest deployable computing unit that can be created and managed in Kubernetes. Each pod in a Kubernetes cluster is assigned an IP in a flat shared networking namespace. This forms a networking model where each pod can communicate with the network just like in a virtual machine.

Pod-to-Service Communication: In the Pod-to-Service Communication model generally services are assigned to client-accessible IPs. They are then transparently proxied to the pods grouped by that service. Requests to the service IPs are intercepted by a kube-proxy process running on all hosts, which then routes to the correct pod.

Internet-to-Service Communication: Permitting external traffic into the cluster is finished mostly by mapping outside load balancers to explicitly uncovered services in the cluster. This mapping permits the kube-intermediary procedure to course the external requests to the proper pods using the cluster’s pod-network. Once traffic arrives to a node then it is routed to the correct service back ends via the kube-proxy.

Kubernetes Networking Plugins / Software-defined networking (SDNs)

Kubernetes does not provide any default network implementation but it only defines the model and leaves to other tools to implement it. Some of the plugins listed below were developed exclusively for Kubernetes.

Kubenet: Kubenet is typically useful for single-node environments and It can be utilized for communication between nodes by using it together with a cloud provider that establishes the rules. Kubenet is a very basic network plugin, so if you are looking for features such as cross-node networking or network policy then Kubenet solution can be considered.

Flannel: It provides an overlay network between each hosted and allocates a separate subnet per host. It supports VXLAN that runs a layer 2 network on top of a layer 3 infrastructure. It also supports host-gw that maps direct routes between hosts.

Calico: Using Calico, Developers & system admins can configure inbound and outbound rules by port, direction, protocol, and other attributes. Calico also runs an agent on each host (uses Layer 3 network) and it connects to hosts using the Border Gateway Protocol (BGP).

Canal: Canal combines two of the projects i.e Flannel and Calico to create a unified networking solution.

Big Switch Network: BCF offers much broader control over our network architecture, from layer 2 to layer 7. For containers, BCF lets we define virtual networks and assign containers to a specific network on creation.

Weave Net: Its a complete networking solution offering a virtual network with service discovery, policy management, and fault tolerance. It encapsulates and routes VXLAN packets in the kernel instead of user space, saving CPU overhead and latency using fast datapath feature.