Tanzu worker node created with wrong operating system
In the VMware 8 Tanzu environment, after upgrading to WCP 1.27, worker nodes and control plane nodes are created with the default operating system (Photon) even though Ubuntu was selected in the TKR selection.
This issue was observed only in existing Tanzu Kubernetes clusters. When the worker nodes are created with the wrong operating system, some system pods may remain in a pending state due to node selector labels not matching on the newly created worker nodes.
Workaround: Apply the node label with the Ubuntu version tag:
kubectl label node --overwrite run.tanzu.vmware.com/kubernetesDistributionVersion=v1.25.13---vmware.1-fips.1-tkg.1.ubuntu
To apply the label to all nodes:
kubectl get nodes | awk '{ print $1 }' | xargs -I {} kubectl label nodes {} run.tanzu.vmware.com/kubernetesDistributionVersion=v1.25.13---vmware.1-fips.1-tkg.1.ubuntu --overwrite
In vSphere version 8u3, Tanzu introduced a TKC annotation for the operating system:
IWe need to annotate the cluster with run.tanzu.vmware.com/resolve-os-image: os-name=ubuntu which will trigger cluster worker and control plane nodes to be created
kubectl annotate tkc -n kubernetes-vns run.tanzu.vmware.com/resolve-os-image="os-name=ubuntu"