7 Real-Life Business Continuity Examples You’ll Want to Read

This post was originally published at https://invenioit.com/continuity/4-real-life-business-continuity-examples/

It’s no secret that we believe in the importance of disaster preparedness and?business continuity? at every organization. But what does that planning actually look like when it’s put to the test in a real-world scenario?

Today, we look at 7 business continuity examples to show how organizations have worked to minimize downtime (or not) after critical events.

Business Continuity Examples: The Good, The Bad & The Ugly

1) Ransomware disrupts Ireland's healthcare system

For years, healthcare organizations have been a top target for ransomware attacks. The critical nature of their operations, combined with notoriously lax IT security throughout the industry, are a magnet for ransomware groups looking for big payouts.?

But despite the warnings, healthcare orgs still remain vulnerable. A prime example was the 2021 ransomware attack on Ireland's healthcare system (HSE) – the fallout from which was still being understood nearly a year later.?

According to reports, the attack had a widespread impact on operations:?

·???????Dozens of outpatient services were shut down

·???????IT outages affected at least 5 hospitals, including Children's Health Ireland (CHI) at Crumlin Hospital

·???????Employee payment systems were knocked offline, delaying pay for 146,000 staff?

·???????Covid-19 test results were delayed and a Covid-19 vaccine portal went offline?

·???????Appointments were canceled across numerous facilities and medical departments

·???????Near-full recovery and restoration of all servers and applications took more than 3 months

All told, the attack was projected to cost more than $100 million in recovery efforts alone. That figure does not include the projected costs to implement a wide range of new security protocols that were recommended in the wake of the attack.?

Like several of the business continuity examples highlighted below, the Ireland attack did have some good disaster recovery methods in place. Despite the impact of the event, there were several mitigating factors that prevented the attack from being even worse, such as:

·???????Once the attack was known, cybersecurity teams shut down more than 85,000 computers to stop the spread.

·???????Disaster recovery teams inspected more than 2,000 IT systems, one by one, to contain the damage and ensure they were clean.

·???????Cloud-based systems were not exposed to the ransomware.

However, there was some luck involved.?

As HSE raced to contain the damage from the attack and secured a High Court Injunction to restrain the sharing of its hacked data, the attackers suddenly released the decryption key online. Without that decryption, HSE would?not?have had adequate data backup systems to recover from the attack. As the group concluded in its?post-incident review :

"It is unclear how much data would have been unrecoverable if a decryption key had not become available as the HSE’s backup infrastructure was only periodically backed up to offline tape. Therefore it is highly likely that segments of data for backup would have remained encrypted, resulting in significant data loss. It is also likely to have taken considerably longer to recover systems without the decryption key."

2) The city of Atlanta is hobbled by ransomware

There has been no shortage of other headline-making ransomware attacks over the last few years. But one that stands out (and whose impact reverberated for at least a year after the incident) was the March 2018 SamSam?ransomware attack on the City of Atlanta .

The attack devastated the city government’s computer systems:

·???????Numerous city services were disrupted, including police records, courts, utilities, parking services and other programs.

·???????Computer systems were shut down for 5 days, forcing many departments to complete essential paperwork by hand.?

·???????Even as services were slowly brought back online over the following weeks, the full recovery took months.

Attackers demanded a $52,000 ransom payment. But when all was said and done, the full impact of the attack was projected to cost more than $17 million. Nearly $3 million alone was spent on contracts for emergency IT consultants and crisis management firms.

In many ways, the Atlanta ransomware attack is a lesson in inadequate business continuity planning. The event revealed that the city’s IT was woefully unprepared for the attack. Just two months prior, an audit found 1,500 to 2,000 vulnerabilities in the city’s IT systems, which were compounded by “obsolete software and an IT culture driven by ‘ad hoc or undocumented’ processes,” according to?StateScoop .

Which vulnerabilities allowed the attack to happen? Weak passwords, most likely. That is a common entry point for SamSam attackers, who use brute-force software to guess thousands of password combinations in a matter of seconds. Frankly, it’s an unsophisticated method that could have been prevented with stronger password management protocols.

Despite the business continuity missteps, credit should still be given to the many IT professionals (internal and external) who worked to restore critical city services as quickly as possible. What’s clear is that the city did have?some?disaster recovery procedures in place that allowed it to restore critical services. If it hadn’t, the event likely would have been much worse.

3) Fire torches office of managed services provider (MSP)

Here’s an example of business continuity done right:

In 2013, lightning struck an office building in Mount Pleasant, South Carolina, causing a fire to break out. The offices were home to Cantey Technology, an IT company that hosts servers for more than 200 clients.

The fire torched Cantey’s network infrastructure, melting cables and burning its computer hardware. The equipment was destroyed beyond repair and the office was unusable. For a company whose core service is hosting servers for other companies, the situation looked bleak. Cantey’s entire infrastructure was destroyed.

But ultimately, Cantey’s clients never knew the difference:

·???????As part of its business continuity plan, Cantey had already moved its client servers to a remote data center, where continual backups were stored.

·???????Even though Cantey’s staff were forced to move to a temporary office, its clients never experienced any interruption in service.

It was an outcome that could have turned out very differently. Only five years prior, the company had kept all of its client servers on site. But founder Willis Cantey made the right determination that this setup created too many risks. All it would take is one major on-site disruption to wipe out his entire business, as well as his clients’ businesses, potentially leaving him exposed to legal liabilities as well.

Cantey thus implemented a more comprehensive business continuity plan and moved his clients’ servers off-site. And in doing so, he averted disaster.

4) Computer virus infects UK hospital network

In another?post , we highlighted one of the worst business continuity examples we saw in 2016 – before ransomware had become a well-known threat in the business community.

On October 30, 2016, a nasty "computer virus" infected a network of hospitals in the UK, known as the Northern Lincolnshire and Goole NHS Foundation Trust. At the time, little was known about the virus, but its impact on operations was devastating:

·???????The virus crippled its systems and halted operations at three separate hospitals for five days.

·???????Patients were literally turned away at the door and sent to other hospitals, even in cases of “major trauma” or childbirth.?

·???????In total, more than 2,800 patient procedures and appointments were canceled because of the attack. Only critical emergency patients, such as those suffering from severe accidents, were admitted.

?Remarkably, a report in Computing.co.uk speculated that there had been?no?business continuity plan document in place. Even if there had been, clearly there were failings. Disaster scenarios can be truly life-or-death at healthcare facilities. Every healthcare organization must have a clear business continuity plan outlined with comprehensive measures for responding to a critical IT systems failure. If there had been in this case, the hospitals likely could have remained open with little to no disruption.

The hospital system was initially tight-lipped about the attack. But in the year following the incident, it became clear that ransomware was to blame – specifically, the Globe2 variant.?

Interestingly, however, hospital officials did not say the ransomware infection was due to an infected email being opened (which is what allows most infections to occur). Instead, they said a misconfigured firewall was to blame. (It’s unclear then exactly how the ransomware passed through the firewall—it may have come through inboxes after all.) Unfortunately, officials knew about the firewall misconfiguration before the attack occurred, which is what makes this incident a prime example of a business continuity failure. The organization had plans to fix the problem, but they were too late. The attack occurred “before the necessary work on weakest parts of the system had been completed.”

5) Electric company responds to unstable WAN connection

Here is another example of well-executed business continuity.

After a major electric company in Georgia?experienced failure ?with one of its data lines, it took several proactive steps to ensuring its critical systems would not experience interruption in the future. The company implemented a FatPipe WARP at its main site, bonding two connections to achieve redundancy, and it also readied plans for a third data line. Additionally, the company replicated its mission-critical servers off-site, incorporating its own site-failover WARP.

According to Disasterrecovery.org:

“Each office has a WARP, which bonds lines from separate ISPs connected by a fiber loop. They effectively established data-line failover at both offices by setting up a single WARP at each location. They also accomplished a total site failover solution by implementing the site failover between the disaster recovery and main office locations.”

While the initial WAN problem was minimal, this is a good example of a company that is planning ahead to prevent a worst-case scenario. Given the critical nature of the utility company's services (which deliver energy to 170,000 homes across five counties surrounding Atlanta), it's imperative that there are numerous failsafes in place.

6) German telecom giant rapidly restores service after fire

Among the better business continuity examples we’ve seen, incident management solutions are increasingly playing an important role.?

Take the case of a German telecom company that discovered a dangerous fire was encroaching on one of its crucial facilities. The building was a central switching center, which housed important telecom wiring and equipment that were vital to providing service to millions of customers.

The company uses an incident management system from Simba, which alerted staff to the fire, evaluated the impact of the incident, automatically activated incident management response teams and sent emergency alerts to Simba’s 1,600 Germany-based employees. The fire did indeed reach the building, ultimately knocking out the entire switching center. But with an effective incident management system in place, combined with a redundant network design, the company was able to fully restore service within six hours.

7) Internet marketing firm goes mobile in face of Hurricane Harvey

Research shows that 40-60% of small businesses never reopen their doors after a major disaster. Here’s an example of one small firm that didn’t want to become another statistic.

In August 2017, Hurricane Harvey slammed into Southeast Texas, ravaging homes and businesses across the region. Over 4 days, some areas received more than 40 inches of rain. And by the time the storm cleared, it had caused more than $125 billion in damage.

Countless small businesses were devastated by the hurricane. Gaille Media, a small Internet marketing agency, was?almost? one of them. Despite being located on the second floor of an office building, Gaille’s offices were flooded when Lake Houston overflowed. The flooding was so severe, nobody could enter the building for three months. And when Gaille’s staff were finally able to enter the space after water levels receded, any hopes for recovering the space were quickly crushed. The office was destroyed, and mold was rampant.????????????????????????????????????

The company never returned to the building. However, its operations were hardly affected.

That’s because Gaille kept most of its data stored in the cloud, allowing staff to work remotely through the storm and after. Even with the office shuttered, they never lost access to their critical documents and records. In fact, when it came time to decide where to relocate, the owner ultimately decided to keep the company decentralized, allowing workers to continue working remotely (and providing a glimpse of how other businesses around the world would similarly adapt to disaster during the Covid-19 pandemic three years later).

Had the company kept all its data stored at the office, the business may never have recovered.

Examples of poor business continuity planning

Some of the real-life business continuity examples above paint a picture of what can go wrong when there are lapses in continuity planning. But what exactly do those lapses look like? What are the specific failures that can increase a company’s risk of disaster?

Here are the big ones:

·???????No business continuity plan:?Every business needs a BCP that outlines its unique threats, along with protocols for prevention and recovery.

·???????No risk assessment:?A major component of your BCP is a risk assessment that should define how your business is at risk of various disaster scenarios. We list several examples of these risks below.

·???????No business impact analysis:?The risk assessment is useless without an analysis of how those threats actually affect the business. Organizations must conduct an impact analysis to understand how various events will disrupt operations and at what cost.

·???????No prevention:?Business continuity isn’t just about keeping the business running in a disaster. It’s about risk mitigation as well. Companies must be proactive about implementing technologies and protocols that will?prevent?disruptive events from occurring in the first place.

·???????No recovery plan:?Every disaster scenario needs a clear path to recovery. Without such protocols and systems, recovery will take far longer, if it happens at all.

Examples of threats to your business continuity

It’s important to remember that business-threatening disasters can take many forms. It’s not always a destructive natural disaster. In fact, it’s far more common to experience disaster from “the inside” – events that hurt your productivity or affect your IT infrastructure and are just as disruptive to your operations.

Example threats include:

·???????Data loss

·???????Cyberattacks

·???????Malware and viruses

·???????Network & internet disruptions

·???????Hardware/software failure

·???????Fire

·???????Natural disasters

·???????Severe weather

·???????Flooding (including pipe bursts)

·???????Terrorist attacks

·???????Office vandalism/destruction

·???????Workforce stoppages (transportation blockages, strikes, etc.)

The list goes on and on. Any single one of these threats can disrupt your business, which is why it’s so important to take continuity planning seriously.

Business continuity technology

Within IT, data loss is often the primary focus of business continuity and disaster recovery (BC/DR). And for good reason …

Data is the lifeblood of most business operations today, encompassing all the emails, files, software and operating systems that companies depend on every day. A major loss of data, whether caused by ransomware, human error or some other event, can be disastrous for businesses of any size.

Backing up that data is thus a vital component of business continuity planning.

Today’s?best data backup systems ?are smarter and more resilient than they were even just a decade ago. Solutions from Datto, for example, are built with numerous features to ensure continuity, including hybrid cloud technology (backups stored both on-site and in the cloud), instant virtualization, ransomware detection and automatic backup verification, just to name a few.

Like other BC initiatives, a data backup solution itself won’t prevent data-loss events from occurring. But it does ensure that businesses can rapidly recover data if/when disaster strikes, so that operations are minimally impacted – and that’s the whole point of business continuity.