The O2 outage and my front teeth

Christmas is upon us and so are the novelty Christmas songs. I do have a special connection with “All I want for Christmas is my two front teeth” (1,2) which I remember from my childhood. Written in 30 minutes by Donald Yetter Gardner while teaching music in a public school when he noticed most of the pupils had at least one front tooth missing. It was released in 1948 and it is  truly a perennial ear worm!

The outage on December 6th when 32 million UK subscribers lost their data service wasn’t what the O2 and Ericsson team want(ed) for Christmas. The network had been prey to what in hindsight was a small SW certificate issue (3). This has been covered in the Azenby article https://azenby.com/archives/2698 which I penned a few days ago.

I feel a special connection with O2 and Ericsson. Enough to say unequivocally that I’d be proud to work again with either company and that both have a great engineering heritage, sincere and capable engineers, managers and good technology. To be frank, unlike some other slippery irritating fly by nights these institutions want to do the right thing the right way by their customers. And the underlying cause of failure is the amazing sophistication and complexity of networks, that bring so much value to their users, under incredible commercial pressure. After all even when you’re a government and have in effect infinite money, complex systems like nuclear power stations sometimes go wrong. So my brain worm in this instance is how we figure out how to take our industry to a better place.

There was a time when if you bought a posh stitched football it was undoubtedly assembled by kids in developing countries in poor conditions. That time has passed. I suspect the executives thought it was impossible to take such a vast and complex logistic system and establish end to end provenance. There was a time when planes crashed with frequency (4). It’s not by accident that flying is now the safest form of transport. Just like it’s not accidental that our drugs work and our chemical plants don’t blow up.

What I now want for Christmas, rather than two front teeth, is that we take the risk management of our telecom networks more seriously. I want our engineers and CTO  teams to have the strength to say ‘our networks are super valuable assets to the UK economy. With systematic processes, hard work and vision we’re going to make our system more robust, truly robust. And you’ll be happy to pay for it as these are attributes people will buy, just like they buy a Volvo for safety, or prepared to associate with a brand that doesn’t indulge in child labour. And we’ll have a true asset to differentiate our offering and ready for massive IOT.’

I lost my two front teeth in 1986 when I elected to cycle into the back of a parked car while asleep. Not my best moment of risk management on a bicycle. I still ride a bike, but not so much into cars, but I'm sure the ICT industry is smarter than me. Merry Christmas.

1 https://www.youtube.com/watch?v=WI02_UJ1C6I

2 https://en.wikipedia.org/wiki/All_I_Want_for_Christmas_Is_My_Two_Front_Teeth

3. https://www.theregister.co.uk/2018/12/06/ericsson_o2_telefonica_uk_outage/

4. Matthew Syed covered this very well and the story of fixing the B17’s in Black Box Thinking. Great book. https://www.matthewsyed.co.uk/

Photo by Sharon McCutcheon on Unsplash

More notes

I wrote this piece as:

1. I want to keep a connection with my friends on LinkedIn which has become my de-facto address book and I want to practise writing. Maybe some of the companies I work with like Flint https://www.flint-international.com/  and Azenby https://azenby.com/ can help with many telco challenges. If not, no worries. Hello either ways!
2. Humour works in the office – as long as it’s self-effacing. Goodness knows LinkedIn needs a smile or two.
3. Wherever the world finds you and you find yourself in the world – be safe