Interesting Kubernetes Problem
Mutha Nagavamsi
Kubernetes, Devops, Cloud & Tech. I run a supercool k8s community, do join. 75K+ strong all socials ??
When you choose to have more than 110 pods/node, kubelet can face many issues.
And many more.
This is a follow up to this post here.
But today let's talk about - "PLEG issues."
PLEG stands for "Pod Lifecycle Event Generator".
It is responsible for tracking pod events and keeping the cluster stable. This will "report back to Kubelet" about the state of Pods.
With more Pods in cluster, PLEG can experiences more lifecycle events.
This can include (creation, deletion, restarts) that it needs to track.
With heavy load, time out issues can creep in. Mainly because it can't track everything due to overload. This can mark the node as "NotReady" in the cluster.
You will see errors like - "PLEG is not healthy."
Sometimes, these issues will come and go.
But they will reappear as you increase total number of Pods running on a node.
How do you fix them?
1. Increase nodes in cluster.
2. Distribute the Pods across all nodes.
Perfectionism kills progress. Focus on getting thing done ??
Also.
Sometimes PLEG issues can come in a fresh cluster.
And there can be variety of reasons for it.
Some of them include:
1. Hostname resolution issues in nodes.
2. CNI Plugin issues on cluster.
3. Firewall rules to allow CNI.
We can discuss more about this in future Newsletters. Knowing possibilities will keep you ready when things go wrong.
So, bookmark this page for future reference.
It can happily save you from some trouble ??
Hope it's useful.
Let me know what you think.
A repost helps.??
Thank you so much for reading this. If you found it interesting, do spread the word about it. You may also find my other content interesting, find them below.
1. Mutha Nagavamsi on Youtube. (Subscribe, it really helps)
3. Me on X.
Senior Infrastructure Engineer | 2x AWS | CKA | Terraform Certified | Docker | Kubernetes | Helm | CI/CD | Python
9 个月Insightful
That was very insightful, thank you Mutha !
Cloud Specialist DevOps at Niveus Solutions Pvt. Ltd.
9 个月Thank you for giving such an informative knowledge.
DevOps with AWS | Jenkins | Docker | kubernetes
9 个月Informative Mutha Nagavamsi
Engineering Leader | Technology Transformation | SaaS | Cloud | DevOps | Innovation
9 个月Nice & Crisp Mutha Nagavamsi