Experiment in Operational Resilience and views on ads about mental health

Friends and followers: This post is about my personal experiment in Operational Resilience (OpRes), mental health and Elon.

The Background

The ability to rebuild servers can be considered as one of the key components of our preparedness against a 'severe-but-plausible' scenario of a 'destructive ransomware attack' under OpRes testing. This is what I 'preach' in my speeches, and I wanted to practise it in my own laboratory.

The Problem

I noticed that the VirtFusion VPS management panel for rebuilding ArchLinux servers is broken. The panel will tell you that the rebuilt server is ready, but it would be impossible to connect to it.

My first server is in Coventry, UK, and the second is in Ohio, USA, and the respective IaaS public cloud service providers are different (Layer and Linveo) and both had exactly the same problem.

(My third server in London from another cloud service provider, Vultr, and as they do not use VirtFusion, the rebuilding was successful in one shot.)

The Linveo engineer told me they could not say how much time VirtFusion would take to fix this. I reached out directly to VirtFusion, and they did not respond. At this rate, the rebuilding of my server could take weeks and months. The 'impact tolerance' became Mission Impossible 8 of Operational Resilience in my personal laboratory!

The Root Cause

I will not bore you with technical details. The root cause is that when IPv6 is enabled, the script used by the VirtFusion panel used by cloud service providers completely messes up the IPv4 gateway. As a result, you cannot use SSH to connect to the server. You could use VNC to enter the server, but you cannot even ping.

(Digression: I give high importance to IPv6 connectivity, which is one of the many modern practices recommended by internet.nl, who are supported by the Dutch government. This is why I believe we Brits could learn a few things from the Dutch.)

As you and I do not have the source code, it is impossible to prove it or fix it. You cannot even create your own fork of the software and make it work.

(Digression: This shows the importance of open source! If we had the code, we could write a permanent solution.)

The Workaround Solution

Every problem has one, and I found it. I created an elegant ten-step workaround to rebuild an ArchLinux server and make it work properly on IPv4 and IPv6. (DM if you need my solution). Thanks to this solution, I could prove that the step of rebuilding an ArchLinux server was successfully tested.

Once we know the solution, the rebuilding took 15 minutes and was no longer a bottleneck in testing my own impact tolerance (which is one weekend).

The Reward

In recognition of the creative solution, which I shared with Linveo, they have doubled the bandwidth for my cloud server to 6 TB per month for free!

I do not consume all of that, so if you need a VPN with an exit node in Ohio, USA, please DM. Free for friends.

OpRes Lesson Learnt

Real-life testing (instead of theoretical assumptions) helped me to prove that server rebuilding was problematic and required a solution.

Mental Health and Twitter/Elon

It was 1:45 AM in London. I had just solved a problem that two cloud service providers could not. And then a naughty Twitter/X ad thought I may have ADHD. I did not even know if I should feel offended? I take mental health seriously and am always there to support my friends and colleagues. But this looked like a targeted advertisement based on my time pattern.

Weekends and nights are the only time I can pursue my passion for cybersecurity. But nobody, not even Elon Musk, should speculate about anyone's mental health. They may be using the stupid AI, profiling and machine learning.

Maybe I am upset over nothing, and this is just a spam ad and in that case I would forgive you, Elon. When your OpRes is messed up, you can call me for a solution. I promise I will not show you an ADHD ad or give the phone number for the Samaritans.

Santosh

8 April 2024