How did your Disaster Recovery plan perform?

Investopedia summarises BCP as follows:

“Business Continuity Planning (BCP) is the process a company undergoes to create a prevention and recovery system from potential threats such as natural disasters or cyber-attacks.”

It’s always been an area of interest for me personally from when it first came across my desk as a facet of Operational Risk to a stand-alone, highly sophisticated process in which full-mock disasters were carried out to see how a company copes under great stress. More recently, the white paper on Operational Resilience was published and the black & white line between a ‘disaster recovery situation’ and ‘BAU’ became even more blurred.

When I have discussed disaster recovery (DR) plans with professionals who specialise in this field in the past, the conversations typically focussed on terrorism, earthquakes (non-UK locations), fire, power blackouts, cyber-attacks... I really don’t recall ‘pandemic’ being discussed much at all; it is certainly fair to say I don’t remember it being cited in the typical ‘top 3’ most likely disasters... and this includes the time since Bill Gates correctly prophesied recent events in his March 2015 TED Talk.

Last week we experienced what I believe to be the first real test of global BCP / DR plans when most of my major clients (namely banks, fund managers and fintechs) initiated their plan and moved to a classic Red Team / Blue Team model. Those that were in Red Team bemoaned the fact they had to remain in the office all week and run the gauntlet of the daily commute, whilst those in the Blue Team got to work from home for the foreseeable 1 week / 2 weeks / 3 weeks depending on the blueprint – plans that were individual and idiosyncratic in design, but all variations on a theme in reality.

One notable exception to the Red & Blue Team set up was the far less colourful but equally arbitrary (of course!) A-Team / B-Team model; not something that helped matters for that business as the political fallout experienced from those categorised under ‘B-Team’ required some serious management attention and TLC to get them back onside.

Following 48 hours for some, 72 hours for most, the whole system then fell apart as Boris (in London, substitute in the key decision-maker for other locations) and his scientific experts quickly moved to ‘flatten the curve’ and we cycled with extraordinary rapidity through the phases of its (our) planned response to an epidemic – Containment, Delay, Research, Mitigate... and we almost found the 5th gear of ‘lock everything down and ground the public, but it was eloquently worded and presented far better.

Then came the memes. Endless videos, photos, jokes and one-liners of varying taste and seriousness. I wonder how many DR plans cater to the mass distraction of the torrent of memes that take all day to read and pass on. It’s a strange phenomenon, I don’t know where the source of this mass creativity and ingenuity is and I’m also not sure whether they serve the public trust through providing a source of humour (the old adage is that laughter is the best medicine after all) or have an adverse effect through constant interruptions. Isn’t it amusing how very clever and witty a meme is when you first receive it and how utterly bland and trivial it is when received from a secondary source? They used to say today’s news is tomorrow's fish n chip wrapper; with memes the shelf life is on a different time scale. Pass it on straight away or not at all is my rule.

Back to disaster recovery... the other notable point I wanted to raise was the expense, planning and shear office space some businesses have gone to in order to set up a home-from-home. City to Croydon, Docklands to Dunstable – always somewhere out of town where the postcode might not be as famous, but the local pint is half the price. We have all heard of the rumours of whole offices sitting dormant, banks of empty desks, working phone lines that never ring and a recovery plan that requires a simple flip of the switch to turn off the part in one building in one location; and get the party started in another. 

I don’t know what captured my imagination about these sites in the same fashion photographs of old Olympic venues or abandoned fairgrounds (Google them) inspire thoughts of what had been or, in this instance, what could be achieved in the location – but I have always been fascinated by the secrecy and emptiness of these DR sites, and particularly the juxtaposition of a vacuous expanse and empty expense to a throbbing hub of BAU activity and searing profitability when THAT switch is flicked.

Stage left, enter Covid-19 and the secondary site went the same way as the whole Red Team / Blue Team. The idea was sound and well thought out for most disasters – but not a pandemic. We had a new team – Purple Team, red & blues together, logging on from home trying to find a free plug socket for the extra kit required to WFH, struggling to find the room for a desk given the sheer amount of toilet rolls and pasta filling up the spare room.

The second site was also designed to be a temporary home, a house with limited mod-cons simply because the disaster will pass - the terror strike, the earthquake, the blackout would just be a temporary blip; a limited and finite time proposition. We have now seen the stark reality that we have no end time on the Covid-19 situation and timeframes are estimated from as little as 4-8 weeks through to 12 months depending on the definition of what ‘better’ looks like.

I think there have been many learning points that have come out of all of this from a business continuity viewpoint, and I hope that the vast majority of DR plans were robust, worked well and served & protected those they were designed for.

How did your week pan out last week? Was your company ‘operationally resilient’ and is your DR plan pandemic-proof??

Did you know what your disaster recovery program looked like and, most importantly... did it work?

And finally... were you part of the A-team... or were you ‘randomly’ assigned to the B-Team....?? <you don’t have to answer that one if it is a sensitive question>

Good luck everyone and stay safe.

Disclaimer – This article is not meant to paint a negative picture of hard-working BCP professionals or make light of a very serious situation. I hope it is read in the spirit intended and I sincerely want to thank all those key workers on the front line and all those DR and IT professionals who spend their time working behind the scenes so we can flip that switch to work from home when it is required.