Kubernetes - Things you should know
Bikash Kumar Sundaray
Solution Architect at Airbus - Data, Backend, DevOps, Cloud and Security
I have been working on Kubernetes and deploying cloud-native apps for over 6 years now. Being at both ends of Cloud-Native Apps where I design/develop and deploy. Seen the transition from the most difficult way of on-prem K8s deployment to managed services like EKS and AKS. Here I am sharing my experience and things you should know about Kubernetes.
This is the Part-1 of this series. I will be releasing around 4 more articles on this topic.
This content targets both cloud and on-prem Kubernetes deployment. Some content may not required for services like AKS, EKS, as internal things are managed by AWS and Azure.
Fact: Kubernetes is also referred to as K8s, not k8s :) - this was just a fun fact.
What we are going to cover in this article
#1 Fun with K8s
Can I visualize my K8s cluster in 3D :) Yes you can, watch out the demo here (Please don't use it for production)
#2 From where I can play with Kubernetes
Online playground
Offline single node K8s ( you need to install)
#3 Can I use Kubernetes for my Apps deployment without knowing it
There are platforms like OpenShift and cloud foundry which are built on top of Kubernetes. This platform gives you an abstraction layer for engineers to focus on engineering problems and App deployment rather than focusing on and knowing what is Kubernetes and how to configure it. But these platforms are built heavily on K8s.
If you don't have Kubernetes engineers in your team, you can take help from the following list of companies listed on CNCF. Who can help you doing K8s setup ( of course they will charge you for their service) https://www.cncf.io/certification/kcsp/
#4 Kubernetes Database
Yes, Kubernetes as an orchestration tool internally used a Database to store information. When you create an Object like Deployment, Pod, Service, Secrets - all such information are stored in etcd database. When you execute kubectl command, kubectl command talks to KubeAPI and KubeAPI request to binary ( internal K8s software ), then those binary extract information from etcd.
etcd is a strongly consistent, distributed key-value store that provides a reliable way to store data that needs to be accessed by a distributed system or cluster of machines. It gracefully handles leader elections during network partitions and can tolerate machine failure, even in the leader node
etcd Benchmarked at 1000s of writes/s per instance
Do you want to understand how practically etcd works ? let's use this website playground to do some write, Read and Delete operation https://play.etcd.io/play
In-future we will see, there will be options to choose between etcd and other distributed database.
Kubernetes API life cycle
The core of Kubernetes' control plane is the API server. The API server exposes an HTTP API that lets end users, different parts of your cluster, and external components communicate with one another.
The Kubernetes API lets you query and manipulate the state of API objects in Kubernetes (for example: Pods, Namespaces, ConfigMaps, and Events).
Have you seen this ? for few objects you write alpha1 and for few write v1 and for others you write beta1
领英推荐
apiVersion: kubelet.config.k8s.io/v1alpha1
apiVersion: v1
When Kubernetes add new features, mostly it never release it as GA (General available), rather it goes through API life-cycle called alpha, beta, betav1, betav2 and GA (v1, v2 etc).
Each time you are upgrading kubernetes, you need to look at your YAML files, in-case any alpha API promoted to beta. Or any beta api promoted to GA. Based on that you need to make changes to your YAML files ( RAW K8s Object YAML files or Helm Chart ) to point to correct API version. As there might be cases that, K8s might deprecate the API it-self.
CRI: Container Runtime Interface
Container is not all about Docker. There are a lot of container solution available, which can be used as Container engine/runtime for Kubernetes.
Out-of-box kubernetes is having a API layer called CRI, which promotes any container engine/runtime which can talk to K8s via CRI ( by means protobuf and grpc ) - kubernetes is going to support it. Kubelet communicates with the container runtime (or a CRI shim for the runtime) over Unix sockets using the gRPC framework, where kubelet acts as a client and the CRI shim as the server.
There are container runtime/engines like Docker, RKT , CRI-O, containerd. But still Docker is the default runtime for K8s till v1.23 and CRI is still under Alpha.
Starting from K8s 1.24, docker-shim is deprecated, means you can't use Docker engine starting from v1.24 of K8s. So what's next ?
For your local docker build and docker run you can use Docker engine. Also in your DevOps pipeline you can use Docker engine. But in actual kubernetes cluster, you can't use Docker are runtime starting K8s v1.24.
There is a company called mirantis, who said it will maintain dockershim for kubernetes along with support from Docker. This OpenSource project will keep dockershim alive. https://github.com/Mirantis/cri-dockerd
If you are confused with, what is docker-shim, then docker-shim is a module in kubernetes which allows kubernetes to support docker runtime as container engine. But now kubernetes is removing that module from v1.24. So you need to either install cri-dockerd by your-self or use other engine like RKT or CRI-O.
Node affinity and anti-affinity
Remember, life is easy until we have just 2~3 worker nodes. Once we will scale to 10 or 20 worker nodes and you are dealing with distributed app deployment then node affinity is very vital.
Do we really need to control pod schedule to particular nodes? well, the answer is YES.
How many ways we can do this
Node Selector way:
kubectl label nodes node-abcd-name workerType=ssdgpu
apiVersion: v1
kind: Pod
metadata:
name: nginx
labels:
env: test
spec:
containers:
- name: nginx
image: nginx
imagePullPolicy: IfNotPresent
nodeSelector:
workerType: ssdgpu
So first we assign a custom key/value pair to worker node. Here we assigned as workerType as ssdgpu. Then in our Pod definition, we said, it should select worker having label as workerType = ssdgpu.
Node affinity (new way)
While defining pod Spec in YAML, you need to include nodeAffinity and shortly there are 2 types of affinity rule, 1st one is hard, if that doesn't match then pod will not be scheduled to any worker nodes. soft affinity is like, this can be ignored by K8s scheduler if it doesn't matches.
spec
affinity:
nodeAffinity:
requiredDuringSchedulingIgnoredDuringExecution:
nodeSelectorTerms:
- matchExpressions:
- key: az-name
operator: In
values:
- az1
- az2
preferredDuringSchedulingIgnoredDuringExecution:
- weight: 1
preference:
matchExpressions:
- key: worker-hardware-type
operator: In
values:
- ssd-gpu
In the above example you can see, i am asking pod to be scheduled in availability az1 or az-2. It should be not scheduled to any other availability zone like az3 or az4. This was hard rule.
In the soft part i am asking to prefer worknode of type ssd-gpu with hard rule as either az1/az2.
Taints and Toleration:
This is bit advanced way of node selection, usually this is not widely followed.
$ kubectl taint nodes node1 key1=value1:NoSchedule
tolerations
- key: "key1"
operator: "Equal"
value: "value1"
effect: "NoSchedule":
First you need to configure worker node with the taint with key-value pair. This can simply done using kubectl command. Then in the pod spec you have to define the toleration. Any pod matching toleration of the node's taint, then pod will be scheduled. One worker node can have multiple taints and any pod wants to be scheduled in that worker node, they have to match all the taints as part of tolerations spec.
Where to ask K8s questions:
https://discuss.kubernetes.io/ and of-course stack overflow.
Thanks for reading :)