Secret to securing your Kubernetes networking in 9 steps
Vartul Goyal
Securing Company Infrastructure | Expert in ASPM | Automating Remediation with AI
Kubernetes cluster components include a Master node which has the following components:
Worker node:
We will see how the different parts communicating with each other are secure.
So what communications should I be concerned about?
Out of all of the above, the API service, kubelet and etcd are the most critical components. Let’s say the pod needed a certificate, it would talk to the kubelet and then the kubelet would communicate with the API server to get access to the certificate stored in etcd.
All communications with the master happen through the API server. Similarly all communications with the worker node happen through the kubelet. Etcd holds all the important information for your cluster. So anything communicating with etcd should be well protected.
So let’s pick up the cases one by one.
Case 1: Communication from API server to kubelet
The API server communicates with kubelet for things like fetching logs, takes in configuration from the user, and communicates with the kubelet to make it a reality.
By default, the communication from API server to kubelet is over unauthenticated TLS. This makes it unsafe to run the cluster over untrusted networks since this communication is susceptible to man (or woman) in-the-middle attacks. If you’re running things on your private network, this behavior should be acceptable. If you want, however, you can use the flag “kubelet-certificate-authority” to verify kubelet’s server certificate. Note SSH tunneling is deprecated so that is no longer a recommended solution.
Case 2: Communication from kubelet to API server
This would happen in cases such as when a new node comes online and that node will need a new certificate to communicate with the API server. The API server has a Kubernetes API that let's get TLS certs signed by a CA that you can control.
Requests are using mutual TLS and API server listens on port 443. So by default, the master can run on untrusted networks.
Another thing to note here is that you will need to setup certificate rotation for your kubelet certs. By default, they are issued with a 1 year expiration date.
kubeadm alpha certs check-expiration
The command for kubelet to rotate certificates is --rotate-certificates
Case 3: Communication from user/admin to API server
The admin interacts with the API server on various instances to manage the cluster.
The authentication method for this communication depends on your needs. There are several options such as:
Case 4a: Communication from API server to etcd
This might happen in cases where a specific node or pod asks for a certificate from the kubelet, the kubelet makes a request to the API server and then the API server fetches it from etcd.
The communication with etcd happens over localhost on port 80 and is not authenticated or encrypted. You should setup TLS for this part of the communication to verify identity of the etcd instance with the etcd server by specifying the flag “etcd-certfile”.
领英推荐
In the reverse direction, the API server listens on the secure HTTPS port 443.
Case 4b: Communication from etcd to API server
The API server listens on the secure HTTPS port 443.
Case 5: Communication between two instances of etcd
Etcd instances talk to each other to keep the state in sync.
Instances of etcd communicate over mutual TLS already so this part of the cluster should also be safe to run on untrusted networks.
Case 6: Between two pods
As you know, pods in different or same nodes communicating seamlessly with each other without NATing is one of the foundational principles of Kubernetes networking. So this is another important path of communication.
Currently, this communication is not encrypted and not authenticated. There are ways to make this more secure such as using some network policy provider like?Tigera’s Calico?or?Cilium?to protect service-to-service traffic. At the very least, it is highly recommended to use some network policy provider based on your application.
You can also use a service mesh like?Istio?to enforce mutual TLS for traffic between services. Using a whole service mesh just to enforce mutual TLS between service-to-service communication might seem like a huge ask, so maybe use a network policy provider that provides the same without needing a proxy in front of every pod.
One obvious thing to recommend here is Pod Security Policies. I will address this more in a separate blog post.
This communication path is so critical this probably requires a whole post of its own.
Case 7: From pods to API server
This type of communication is done over plain HTTP but really this type of communication shouldn’t even be happening directly.
Case 8: Other components like kubescheduler, kube-controller-manager talking to the API server.
This happens over localhost and you can set up secure communications here as well. Use
--secure-port
for securing communication from the kube-controller-manager to API server. This uses port 10259 by default and uses HTTPS authentication. You can also set the option and many others:
--tls-version-min
Starting Kubernetes 1.13, secure port 10259 can be used for secure connections from kube-scheduler to api server.
Case 9: Between two nodes
You can use the same certificate signing API if you want to ensure node-to-node communication also uses certificates.
Conclusion
Summarizing all of this information in a table for you:
Source: https://medium.com/google-cloud/secure-kubernetes-internal-networking-5f2556f7efde
Chief Executive Officer @ OMVAPT | Information Security | Secure CEO as a Service |
2 年Nice ??