Proxmox, Ubuntu 22.04 & Kubernetes How-To

A Kubernetes cluster consists of a control plane and one or more worker nodes. These instructions are specifically for Ubuntu 22.04 on the control plane and worker node servers. If you're using any other operating system, these instructions are unlikely to work.

First, install Ubuntu 22.04 Server for the control plane and as many worker nodes as you plan to use. During installation, make sure to specify a manual IP address, not DHCP. In addition, choose the installation option to install ssh. That way, on a separate machine, you can open an ssh session for each server and finish setup in those terminals.?

Do the following on all servers.?

?

sudo ufw allow "OpenSSH"
sudo ufw enable        

?

Once everything is installed, open up an ssh terminal for each server and login your user on each terminal. From here on out, you perform actions in your ssh terminal sessions?for all servers.

Update the system. I like to install net-tools (so you can run ifconfig) and micro (a simple text editor that's easier to use than nano). You can use nano or vi, but micro is so much easier to use than either of those, I strongly recommend you try it. Do the following on all servers.

?

sudo apt update
sudo apt dist-upgrade
sudo apt install net-tools micro        

?

Kubernetes requires that you turn off swap. Do this on all servers.

?

sudo swapoff -a        

?

Now edit /etc/fstab and remove or comment out the line that specifies the swap file. Like I said above, I like micro for editing. Do this on all servers.

?

sudo micro /etc/fstab        

?

Here's an example of a commented out line in /etc/fstab for the swap file if you install Ubuntu 22.04 on Proxmox. Do this on all servers.

?

#/swap.img      none    swap    sw      0       0        

?

You can verify that the swap is off with this command (and sample output). Do this on all servers.

?

free -m

               total        used        free      shared  buff/cache   available
Mem:            1963         619          66           2        1278        1184
Swap:              0           0           0        

?

On the server you plan to use as the control plane, set the host name.

?

sudo hostnamectl set-hostname cplane1        

?

On the servers you plan to use as workers, set those host names. For the sake of brevity, I'm only including instructions for one worker node.

?

sudo hostnamectl set-hostname worker1        

?

You want to edit the /etc/hosts files on every server to point to your control plan and workers. Although I'm using only one worker, I'll give you a sample of what it should look like if you have 3 workers. Edit the hosts file on every server:

?

sudo micro /etc/hosts        

?

Add lines like these, depending on the number of nodes and your IP addresses. The IP addresses here are just my samples; use your own. Do this on all servers.?

?

192.168.10.40 cplane1
192.168.10.41 worker1
192.168.10.42 worker2
192.168.10.43 worker3        

?

Now do the following kernel module operations on all servers. This installs the overlay and br_netfilter modules and makes the change permanent by adding it into /etc/modules-load.d/k8s.conf.

?

sudo modprobe overlay
sudo modprobe br_netfilter
sudo cat <<EOF | sudo tee /etc/modules-load.d/k8s.conf
overlay
br_netfilter
EOF        

?

Set up iptables on all servers.

?

cat <<EOF | sudo tee /etc/sysctl.d/k8s.conf
net.bridge.bridge-nf-call-iptables  = 1
net.bridge.bridge-nf-call-ip6tables = 1
net.ipv4.ip_forward                 = 1
EOF        

?

Apply these settings without rebooting by using this command on all servers.?

?

sudo sysctl --system        

?

Now install the container runtime on all servers and back up the default Docker container configuration file because we're going to edit it in a moment.?

?

sudo curl -fsSL https://download.docker.com/linux/ubuntu/gpg | sudo gpg --dearmor -o /usr/share/keyrings/docker.gpg
echo \
  "deb [arch=$(dpkg --print-architecture) signed-by=/usr/share/keyrings/docker.gpg] https://download.docker.com/linux/ubuntu \
  $(lsb_release -cs) stable" | sudo tee /etc/apt/sources.list.d/docker.list > /dev/null

sudo apt update
sudo apt install containerd.io
sudo systemctl stop containerd
sudo mv /etc/containerd/config.toml /etc/containerd/config.toml.orig
sudo containerd config default > /etc/containerd/config.toml
        

?

Now edit /etc/containerd/config.toml and change one setting from false to true.

First, find SystemdCgroup, which is normally set to false. Set it to true.

?

SystemdCgroup = true        

?

Now start the containerd.?

?

sudo systemctl start containerd        

?

Check if everything is kosher.

?

sudo systemctl is-enabled containerd
sudo systemctl status containerd        

?

You should see that the container is active and running. For example:

?

● containerd.service - containerd container runtime
     Loaded: loaded (/lib/systemd/system/containerd.service; enabled; vendor preset: enabled)
     Active: active (running) since Thu 2023-09-28 13:56:49 UTC; 1h 5min ago        

?

Install Kubernetes

Do this on all servers. The apt-mark tells Ubuntu not to change any Kubernetes versions from now on.?

?

sudo apt install apt-transport-https ca-certificates curl -y

sudo curl -fsSLo /usr/share/keyrings/kubernetes-archive-keyring.gpg https://dl.k8s.io/apt/doc/apt-key.gpg
echo "deb [signed-by=/usr/share/keyrings/kubernetes-archive-keyring.gpg] https://apt.kubernetes.io/ kubernetes-xenial main" | sudo tee /etc/apt/sources.list.d/kubernetes.list

sudo apt update
sudo apt install kubelet kubeadm kubectl
sudo apt-mark hold kubelet kubeadm kubectl        

?

Now add flanneld to your installation. Do this on all servers.

?

sudo mkdir -p /opt/bin/

sudo curl -fsSLo /opt/bin/flanneld https://github.com/flannel-io/flannel/releases/download/v0.19.0/flanneld-amd64

sudo chmod +x /opt/bin/flanneld        

?

The Control Plane

Now go to the ssh terminal for the control plane. Open up these ports and check the status:

?

sudo ufw allow 6443/tcp
sudo ufw allow 2379:2380/tcp
sudo ufw allow 10250/tcp
sudo ufw allow 10259/tcp
sudo ufw allow 10257/tcp

sudo ufw status        

?

?You should see something like this:

?

To                         Action      From
--                         ------      ----
OpenSSH                    ALLOW       Anywhere
6443/tcp                   ALLOW       Anywhere
2379:2380/tcp              ALLOW       Anywhere
10250/tcp                  ALLOW       Anywhere
10259/tcp                  ALLOW       Anywhere
10257/tcp                  ALLOW       Anywhere
OpenSSH (v6)               ALLOW       Anywhere (v6)
6443/tcp (v6)              ALLOW       Anywhere (v6)
2379:2380/tcp (v6)         ALLOW       Anywhere (v6)
10250/tcp (v6)             ALLOW       Anywhere (v6)
10259/tcp (v6)             ALLOW       Anywhere (v6)
10257/tcp (v6)             ALLOW       Anywhere (v6)        

?

Now pull the kubernetes images on the control plane server only.

?

sudo kubeadm config images pull        

?

?Now initialize kubernetes. Substitute YOUR IP address for 192.168.10.40 in this command:

?

sudo kubeadm init --pod-network-cidr=10.244.0.0/16 \
--apiserver-advertise-address=192.168.10.40 \
--cri-socket=unix:///run/containerd/containerd.sock        

?

Now set up the credentials you need to run the control plane.

?

mkdir -p $HOME/.kube
sudo cp -i /etc/kubernetes/admin.conf $HOME/.kube/config
sudo chown $(id -u):$(id -g) $HOME/.kube/config        

?

Check to see if it's running.

?

kubectl cluster-info        

?

Now apply the settings to get kubernetes to use flanneld.

?

kubectl apply -f https://raw.githubusercontent.com/flannel-io/flannel/master/Documentation/kube-flannel.yml        

?

You can check everything with this command:

?

kubectl get pods --all-namespaces        

?

Get the Worker Join Command

On your control plane, issue this command:

?

sudo kubeadm token create --print-join-command        

?

Copy the output of this command and save it somewhere. You will need it for the worker nodes. For example, the output for my cluster looks like this:

?

kubeadm join 192.168.10.40:6443 --token d79u9c.22evtkkn0arwpltx --discovery-token-ca-cert-hash sha256:591027ac99b4e04319e67e79e5251d5d106b38c036f4b5f744ecdf5c3c9c0549        

?

The Worker Nodes

Switch to the ssh terminal for your worker nodes, one at a time. For the sake of brevity, these instructions are only for one worker node. If you have multiple worker nodes, perform these actions on all of them.?

Open up the firewall for your worker node and check the status:

?

sudo ufw allow 10250/tcp
sudo ufw allow 30000:32767/tcp

sudo ufw status        

?

?Remember that join command you saved?? Run it now.? For example (don't use this exact command, use the one you saved):

?

kubeadm join 192.168.10.40:6443 --token d79u9c.22evtkkn0arwpltx --discovery-token-ca-cert-hash sha256:591027ac99b4e04319e67e79e5251d5d106b38c036f4b5f744ecdf5c3c9c0549        

?

Back To The Control Plane

Go back to the ssh terminal for the control plane and check to see if the worker was added. First, check all the pods:

?

kubectl get pods --all-namespaces        

?

You should see something like this:

?

NAMESPACE      NAME                              READY   STATUS    RESTARTS      AGE
kube-flannel   kube-flannel-ds-nwvgl             1/1     Running   5 (89m ago)   15d
kube-flannel   kube-flannel-ds-z64gx             1/1     Running   2 (89m ago)   13d
kube-system    coredns-5dd5756b68-9zvqw          1/1     Running   3 (90m ago)   15d
kube-system    coredns-5dd5756b68-t7s5m          1/1     Running   3 (90m ago)   15d
kube-system    etcd-cplane1                      1/1     Running   3 (90m ago)   15d
kube-system    kube-apiserver-cplane1            1/1     Running   3 (90m ago)   15d
kube-system    kube-controller-manager-cplane1   1/1     Running   3 (90m ago)   15d
kube-system    kube-proxy-2gj8t                  1/1     Running   3 (90m ago)   15d
kube-system    kube-proxy-65z9m                  1/1     Running   1 (89m ago)   13d
kube-system    kube-scheduler-cplane1            1/1     Running   3 (90m ago)   15d        

?

If everything looks kosher, now check to see if the worker is added:

?

kubectl get nodes -o wide

or

kubectl get nodes        

?

If you're not using "-o wide" you should see something like this:

?

NAME      STATUS   ROLES           AGE   VERSION
cplane1   Ready    control-plane   15d   v1.28.1
worker1   Ready    <none>          13d   v1.28.1        

?

Congratulations! You now have a Kubernetes cluster with one worker node online. Rinse and repeat these last two sections for any additional worker nodes.?