???? A Day of Fire Alarms and a High-Stakes Rescue! ????
Picture this: a typical day as an infrastructure architect in the NHS, bustling with the tasks of deploying a new system, when suddenly, chaos ensues. I was knee-deep in server setups with the supplier, engrossed in stakeholder calls, when an unexpected symphony of alarms sounded.
As the fire alarm ?? screamed and a thunderous mechanical rumble echoed through the data centre, panic surged through the corridors. Rushing with the IT team alongside, we sprinted towards the data centre; Usain Bolt would have had a hard time keeping up.
Amidst the chaos, an HVAC engineer – signed in by the estates team earlier that day – was tinkering with the air handling units. Little did they know that a simple act, soldering a pipe, would set off a chain reaction that would put the entire data centre in jeopardy and bring the hospital’s computer systems to a standstill.
During the frantic evacuation, thoughts raced through my mind. Could the gas leaking into the office and corridor pose a threat? The panic intensified until the realisation dawned on me that the gas couldn’t be toxic; it was designed solely for fire suppression. Phew! ??
The onsite emergency response team ?? arrived, assessing the situation while I rallied to assist the customer. With precision and focus, we navigated the power cut triggered by the fire system – a reminder of the singular infrastructure stack of yesteryears in the NHS, where failover mechanisms were a luxury. We turned on the extraction system and started to assess the damage.
The real challenge lay ahead of us which was restoring the critical systems that went offline – A&E, Radiology, Pathology, PAS – to name a few, and each integral to patient care.
Through tireless efforts that stretched into the early hours, we meticulously restored the infrastructure, ticking off each step in the recovery process. As dawn approached, the systems were back online, safe and sound.
领英推è
What I learned from this dramatic experience was:
1: ?? Emergency Preparedness is Key: Swift and coordinated responses can mitigate even the most unexpected crises. To this end I helped the customer introduce some additional checks to implement before allowing anyone to enter the data centre.?I think being in the scouts also helped me here and I can't wait to take my two boys one day soon.
2: ? Redundancy and Failover: The importance of backup systems and failover mechanisms cannot be overstated, especially in critical infrastructure setups. I don’t deploy a system today without making sure that we explore all the options we can to introduce some kind of failover; whether via existing kit or new, there are always options for either these days.
?? In moments of crisis, resilience and quick thinking are key. This unforgettable day, while challenging, served as a reminder of the invaluable lessons that shaped my approach to infrastructure management; how your day can start off the way you intended it and then quickly descend into chaos.
This has also helped me in some way to prepare for the fun and chaos I have helping my two young boys grow up ??
#NHS #EmergencyResponse #InfrastructureManagement #LessonsLearned #TechChallenges #dadlife
Senior Security Policy Specialist
1 å¹´Great story, thanks for sharing. I'm a great believer that the best outcomes & learning can stem from chaos! Calm waters don't make skilful sailors!
(PriCSP MCIIS) | CSAE | CSAP | Blue Teamer | Cloud Security | Cyber Strategy
1 å¹´I feel a book coming ??
Keep ‘em coming Mark. You’ve cheered me up.
Senior Technical Project Manager at Mid and South Essex NHS Foundation Trust
1 年I remember a lovely morning in a large organisation where a load of systems went down all of a sudden. It transpired that it was down to a visiting engineer who, on visiting a datacentre, decided that the fibre cabling “looked a bit untidy†and decided to take it on himself to do the tidying. I remember sitting next to one poor chap trying to make sense of what to bring up when alongside another engineer in the datacentre. He was surrounded by quite a few important people tapping their fingers somewhat!