An edgecase kubernetes cluster 4 fun

"Let's create a working kubernetes cluster on weird machines, like Raspberry Pis!", Gerrit, our Fullstaq CEO, exclaimed in a one-on-one call. "Just for a quick demo at Fullstaq's Kubernetes on the Edge event!"

Oh, yeah. I had wanted to do that years ago, but my two little Raspberry Pi 1Bs never got the power (cpu or memory-wise) to do that. But with the newer versions, it should be easy-peasy.

So I jumped in and said I'd do that. I'd also investigate if we could get other machines working, like a Wii, a Playstation 2, 3 or 4, or even my PSVita. That'd be real fun, to have a cluster with all kinds of these nodes! Gerrit grinned but was not sure about all those kinds of machines.

However, I was going to try to get all of these on to one cluster. In for a penny, ...! How hard could it be? Little did I know.

Getting Hardware

I had a Playstation 2, 3 and 4 myself. And the PS Vita. And two Wiis. That should do, right? If they could pull their k8s weight. I just needed some Pis to get this freaky cluster going.

First problem: getting your hands on a Raspberry Pi 4. That turned out to be virtually impossible without breaking criminal laws. Chip shortages cut down production of this little machine and they were sold out everywhere. Even the Raspberry Pi 400, the "full-keyboard-version" of the Raspberry Pi 4, was nowhere to be found. With a few tricks I did found a handful at Bol.com - but with a French keyboard layout. In the late nineties I had been forced to work with such a layout for a week in France and nightmares and key-frustration reappeared in my head. No, please, no...

So I looked for a few weeks until I finally found a company that still held a few hundred of them. This shop hadn't been looted by nerds like us yet. I ordered five immediately. Two for myself and three for the demo. A day or two later, we had the hardware. So far so good.

Powerbanks

All and well to have Raspberry Pis, but you don't want them hooked up into a wall socket on an edge-conference. You want these Pis to be thrown around, running on a powerbank. But how long can a Pi run on a powerbank?

After some soul- and google-searching, I found the specs for the Pi 400's power consumption. It wasn't much. About 1000+mA at full throttle, which could easily be remedied with a nice simple powerbank.

Using a 10K powerbank I found out that it could run for about 7.5 hours, while showing video on YouTube via HDMI. Just for safety, we bought a handful of 20K powerbanks. That should work out fine. And it did. The nodes eventually ran for about a day on such a powerbank.

Some double-sided tape helped stick these battery-packs to the bottom side of the Pis. Now we had truly portable nodes!

Which Kubernetes?

The first question was: what kind of Kubernetes should we run on these machines? The hardware specs weren't too amazing, so lean-and-mean was the way to go. Micro-k8s, k3s, k0s, or minikube? And there are even more options nowadays, but let's stick to these. All seemed viable options.

Minikube, great to use on your laptop for testing, was not meant to create clusters with other nodes. It also spins up a VM on the host machine to run Kubernetes - way too much impact. That one was gone from the list.

k0s turned out to be significant larger than k3s. The binary was almost three times the size of k3s (150MB vs 50MB). We were on the edge, so each bytes counts. k0s was scrapped at this time.

Microk8s looked intriguing. But every other hands-on project on odd clusters that I found used k3s. A Microk8s node ate up over 500MB of memory, while k3s just required 256MB.

So k3s it was. Lean, mean, the smallest footprint and easy to install.

Looking for weird nodes

I talked about all these odd machines for using with kubernetes. Well, a few didn't match the required specs for k3s. Like my old trusty Commodore 64. Others did, but we did not have these.

Here's the list I checked:

Wii: Can run Linux, has the right ARMv7 for cross-compiling Kubernetes, but has only 24MB main memory; status: not viable

PS3: Can run a Linux cluster, so why not? Well: 256MB memory... That is too little for this demo. Status: not viable

PS Vita: Can run Linux, has 512MB main memory, though Linux also eats memory from that 512MB, the status is: not impossible!

PS4: Definitely possible. See?https://zhekunhu.xyz/ps4-kubernetes.html

iPod Nano: can run Linux, but not enough memory by far (see?https://www.ipodlinux.org/?)

iPad: Linux on iPad is not yet finished (see?https://ipadlinux.org/?)

XBOX: 64MB RAM - oh never mind

XBOX360: Linux is possible of course (an XBOX is just a PC) (?https://teqnation.com/linux-op-een-xbox-360/?), memory: 512MB ; see PS VITA; status: not impossible!

XBOX ONE: 8GB memory; no problemo; status: possible

Nintendo xDS: ARM processor (nice), but just 64MB memory in the largest of them: the 3DS. That’s not enough.

Nintendo Switch: 4GB memory (whoot!), Tegra processor (ARM compatible), L4T Ubuntu is Linux on Switch; see?https://github.com/t04glovern/nintendo-switch-k8s ; so it might be possible, but not within our time frame for this demo.

Conclusion: We were going to stick to the Raspberry Pis for now. And perhaps a PS4, if we had some time left. Or that PS Vita - a small and somewhat old handheld that was capable of quite a digital whallop. Especially for the time it was released: Jan 2012!

Running a first cluster

k3s was set up easily on my home network. One Pi was server, the others manually got the token via SSH and could attach and start. So with a command line it was easy-peasy indeed. Me like k3s! Now for scripting the whole thing, because we want the cluster to start from scratch, if possible.

Force HDMI

One of the things that needed to be changed immediately is the HDMI setup of the Pi 400. When you do not have an HDMI cable connected to a monitor, the Pi 400 switches off HDMI completely. The machine doesn't show anything when you hook up an HDMI monitor after booting. That is annoying.

Changing the HDMI settings in /boot/config.txt fixed that. A small change, but very handy one.

Displaying of a Pi 400

After forcing HDMI to be used, when working with multiple Raspberry Pis it was cumbersome to switch between all these Pis by changing HDMI cables.

So I looked for a simple solution and found that by installing x11vnc, I could just sent the display via my hotspot to my android phone! How easy was that?

All I needed was a crontab that tried to start x11vnc for the :0 display. If it failed it either had no display yet (no one was logged on) or it was already running (fine by me too).

The crontab entry I created was:

* * * * * * sleep 10 && x11vnc -forever -bg -usepw -httpdir /var/web/html/ -httpport 49152 -display :0

The httpd and httpport are necessary, otherwise each minute a new x11vnc server is started on a new display port and that's something you don't want. With this little trick, only one can claim port 49152 ($c000 from my C64 days #sigh) and it will run until it gets shot down.

AWS master, Pi clients

The first thing I noticed was that when you start scripting, one of the nodes needs to be set to be the master. But I wanted more that just a three-node cluster. I wanted a cluster that could be started from anywhere in the world, in any order. Broadcasting your IP, claiming a DNS entry, selecting a master in the cluster, it all turned out to be cumbersome. So I went one step further: I added a control plane node in AWS, using a small EC2 instance. With that, nodes could be added anywhere in the world, not just the ones on my internal network. In a way, that's even cooler. We could add PSX or Pi nodes from everywhere in the world. #nice2have

Unfortunately it couldn't be a free-tier instance. Those only have one vCPU. I had the experience that these machines lock up when they get busy: the CPU is claimed by processes and it can't reply to AWS health checks. That means AWS considers them broken, the IP bound to that machine is revoked and you end up with a connection-less machine. So we needed at least two vCPUs. A t3.small was good enough.

Setting up k3s on the EC2 was simple. However, I also wanted a small httpd running on that machine. That would be the basis for any client that wanted to connect. I'd install a small install-script that would retrieve the client-token and hand out a custom script to install k3s on the client node with the correct token.

But traefik got in the way. Traefik, the default ingress controller on k3s, claimed port 80 on the master node. As I had no intention to use an ingress for this demo, I just switched off traefik as it clashed with httpd. Now httpd ran fine and my little website on the EC2 showed up. This was going great.

GitHub

As I was doing things in code, I needed a github repo. That was easily created, however the client nodes also needed access. A user-token was created on the repo so that any client could clone this repo and start the appropriate script.

So that github user-token needed to be on the website too, so the clients could retrieve my private repo. As it was read-only and the website was password-protected, it posed no true threats.

From there on, all the scripts were added to GitHub and just one install script needed to run to hook a server up to the master and become a node.

Hotspot!

My wifi network at home wouldn't be accessible at the K8s-on-the-edge venue.

So I was tempted to use my phone as a hotspot to get these Pis hooked up to the internet. I started my hotspot, connected the Raspberry Pis to it and lo-and-behold: no problems at all. That made the cluster even more "edgey", in my eyes. Cool.

Multi-arch clusters

Another challenge reared its head: we now had a control plane node that ran on an X86 machine, and Pis that ran on ARMv7 machines. The master was also used as a client node (default with k3s) and I didn't want to change that at this moment.

So deploying pods with images would be a problem if these images would not support both these architectures. With some testing I found that busybox could run on both platforms, so for this demo I'd stick to busybox as the main image to run everything.

For example: Rancher was not ready to run on ARMv7, only on X86. ArgoCD does support all platforms, so that could be nice to have too. (However: we lacked an ingress without Traefik - do we need Argo? Nah - let's skip it for now.)

But (note-to-self) if ever we were to create new images for specific purposes, we would need to (cross-)compile images for all these architectures!

NB: to cope even better with this multi-arch cluster setup, we thought of creating different namespaces or labels for nodes that could match deployments. That way, specific deployments with specific images would run on the nodes that matched them. Using Open Policy Agent, you could even intervene and change the image settings of a deployment to match the node that it would attach to. But that was something we could do in a later session. For now, we just needed some pods to run on the nodes.

Master Node

To create the master node, I concocted a script that would:

1. install some basic requirements for a machine that we needed to work on, like: emacs-nox git curl httpd ; yep, I am still not fond of vi.
2. install a crontab line that copies the master's node-token and the github's client-install script to the root of the webserver
3. retrieve and install the latest kubectl
4. create some basic aliases for bash in the root's .bashrc
5. clone my fullstaq-k8s-on-the-edge repo
6. UNinstall (!) k3s if it is present on the system (purge all earlier versions)
7. UNinstall the k3s-agent if it is present on the system (purge all earlier versions)
8. set a few variables like K3S_NODE_NAME and INSTALL_K3S_EXEC
9. start the actual k3s install script (which is a oneliner with curl piping its output to sh)

EC2: Public or Private IPs?

Strangely enough, the clients could connect to the master but failed to register as a node. With verbose logging I found that it was an IP issue.

It turned out that EC2s at AWS have both a public and private IP. And of course, by default the private IP is used by k3s as the network interface only has that IP available to the system. The public IP can only be retrieved via accessing the special website from within the EC2: https://169.254.169.254/

So retrieving the public-ip from there, and adding it to the environment variables in K3S_EXTERNAL_IP and voila: we had a cluster with client-nodes again!

Client nodes install script

Meanwhile I had also created an install-script for the clients. It ran somewhat similar to the master-script with a few additions. We had to retrieve more tokens etc, as we needed to connect to the kubernetes cluster too.

The client-install script fell into two parts: the install script and a setup script that would install a trigger to retrieve and call the install-script at boot time. Crontab has this nice @reboot option, which triggered the retrieve of the client-install script. So each time the system booted, it would clean up the mess of the previous installation and start fresh.

The install script, like the master-install-script, also ran through a few cleanup chores before it installed and ran k3s:

1. run k3s-uninstall.sh if it exists
2. run k3s-agent-uninstall.sh if it exists
3. Set a number of tokens and variables in the K3s environment, like: K3S_TOKEN (retrieve it from the master's website), K3S_URL (that's the master's domainname), INSTALL_K3S_EXEC, K3S_NODE_NAME (retrieve that from the hostname)
4. Run the k3s install script: curl -sfL https://get.k3s.io | sh -

...and we're done.

Result

Each node could be started by hooking up the powerbank. It would automatically connect to the hotspot, retrieve k3s and the kubernetes node token and hook itself into the cluster. When the power was cut, the node remained "known" but NotReady in the cluster. Pods would be reshuffled in all remaining nodes. As soon as it came back, new pods would be deployed to that node again. It was wonderful.

Just for fun I created a number of fake pods in namespaces like "control", "hacking", "thug-detection" and the like. I named a deployment after a bitcoin miner, one set seemed to be hacking phones via bluetooth, we had vacuum-controllers for our Roomba's and camera-control pods for the north, south and west area's.

To be continued..?

I'd still love to get k3s working on my PS Vita. And one of the Playstation systems. So that's something I'll continue to work on, in the next few months in the spare free time I have next to work and a large busy family. Wouldn't it be great to see kubernetes running on a handheld machine from 2012? :D