Rethinking Recovery: BackupSec

Let me set the scene….

Remember all of those movies that show the world in the very distant future like 2030, 2040, or 2050 where various forms of global peril, destruction, disasters, and even artificial intelligence consume us or even rule us. Just to name a few:

• "Contagion" (2011) - A deadly virus spreads rapidly around the world, causing widespread panic and death.
• "Transcendence" (2014) - A scientist's drive for artificial intelligence takes on dangerous implications when his consciousness is uploaded into one such program.
• "Ready Player One" (2018) - When the creator of a virtual reality called the OASIS dies, he makes a challenge to all OASIS users to find his Easter Egg, which will give the finder his fortune and control of his virtual world.

If life was a movie then I think a few of the characters from those movies I just mentioned are walking amongst us now.

We are living in what economists call the Digital Age, the Computer Age, the Information Age, or the Fourth Industrial Revolution. The age that is characterized by a shift from traditional industry to an economy based on information technology. This era is associated with the widespread use of supercomputers, cloud computing, the explosion of social media platforms, smart technology, artificial intelligence, Internet of Things, and 5G & 6G wireless communication technologies.

In my opinion, the world has gone Quantum already!?

We have made a significant, revolutionary leap forward (a quantum leap) in terms of innovation, technology, and how data is at the forefront of this shift. It’s the way business is operated today around the world.

What really catapulted us or spring boarded us beyond Industry 4.0 into the Fifth Industrial Revolution or Industry 5.0 happened on November 30, 2022.?

Does anyone know what happened on this date, November 30, 2022?

And “no”, since I’ve been referring to movies…..it wasn’t the release of Cocaine Bear!

It was ChatGPT!

ChatGPT signed up 1 million subscribers in the five days after its release, according to a Dec. 5 tweet from Sam Altman, the CEO of OpenAI.

By January 2023, it had become the fastest-growing consumer software application in history, gaining over 100 million users and contributing to OpenAI's valuation growing to US$29 billion.

So going retrograde or going backwards from here -- I’ve not properly set the same to share my Summit learnings.

Remember the book by Thomas Friedman, that the world is flat? It was about the impacts of globalization in the 21st century. And, how technology and other factors have "flattened" the world and minimized economic and political barriers, thus changing the way societies and businesses operate.

The new reality that we live in now is one that is so exciting and brimming with opportunity and there’s also an underbelly of opportunity for darkness that looms as well -- and it’s that old adage that “with great power comes great responsibility.”?

No longer can a Backup Administrator only think about backups.

No longer can a Network Administrator just think about Networking. And, no longer can the CEO just think about investor relations, financial strategy, or budget management.?

In our current digital age, it's imperative for us all to adopt a security-centric mindset and become globally aware citizens and activists. As we've been thrust into this New Quantum World, the foundation lies in one key aspect of the digital universe – and yes, it's data. Protecting this vital resource is more important than ever in this rapidly evolving landscape.

Now it’s time for you to raise your head out of the sand, and be aware of your surroundings -- not just within your organization -- but globally.

I noticed a theme running through most of the sessions at the Gartner Security Summit that mentioned Recovery in them. That theme was that there a few Emerging Technologies at the forefront of keeping your crown jewels safe and secure along with a few factors to consider when trying to increase your security posture:

1. Macroeconomic / Global Uncertainty
2. Artificial Intelligence
3. Cyber Recovery Strategy

Macroeconomic / Global Uncertainty -?

You have to take the global economic headwinds into consideration now which include recession, inflation, layoffs, interest rates, regulatory changes, etc…according to Gartner an “economic downturn is the top concern of Global CEOs in 2023.

You also have to connect cybersecurity to your business outcomes according to William Candrick (Director Analyst) in his session on “5 Cost Optimization Techniques for Security Leaders Facing Economic Headwinds”

As the Backup & Recovery SME you should be able to help your leaders answer questions like

1. What is our cyber-risk posture?
2. What industry standards do we align to?
3. Are we resilient to cyber risk?
4. What is our ransomware recovery plan?
5. Are we secure and compliant?

Now we all know that ransomware attacks and data breaches are the #1 threat to today’s digital organizations. You should continue to align your efforts to the NIST Cybersecurity framework that provides a flexible approach to address and manage cybersecurity risk in a cost-effective way. It's centered around five core functions where your objectives and controls should lean toward:

Identify - refers to the necessary steps taken to handle the cybersecurity risks that the organization may face.

Protect - the organization's data, systems, networks, and procedures.

Detect - Identify unusual activities and cyber threats that could potentially harm your organization.

Respond - involves halting ongoing cyber threats, mitigating their effects, and deriving lessons from previous incidents.

Recover - minimize the impact of cyber incidents on the organization by protecting the backups and storing them in an air gap, isolated, offsite location where the data is immutable and indelible.

As decisions are made taking into account the macroeconomic / global uncertainty, you also need to prioritize your cybersecurity strategy without skimping or piecemealing together your entire cyber risk solution.

If you are not already tracking cybersecurity metrics like Incident Remediation Time, OS Patching Cadence, Ransomware Recovery Exercise, Ransomware Downtime Workarounds, or Privileged Access Management to name a few; then the security team and backup/storage teams should get together and discuss and define these super critical metrics to establish a baseline and increase your overall IT visibility.

According to Katell Thielemann (Distinguished VP Analyst at Gartner), during her session “Security View of the 2023 CIO and CEO Agenda” you should set Protection-Level Agreements with the business where you make an investment decision based on the measured level of protection; for example, if you have a 30-day patching plan that sets the budget at $1M per year you use that and stick to it as a plan for IT leadership and delivery.

I would also recommend that the VP of Infrastructure Operations (or whoever has the responsibility of the Backup/Storage team) is part of the Cybersecurity Steering Committee where the CISO, CIO, Chief Risk Officer, and other IT leaders all discuss your security controls, your risk tolerance, and your ransomware recovery plan.

Artificial Intelligence

As you may already know, Artificial Intelligence is the superhot buzzword and everyone is racing to implement some type of AI/ML product into their platforms or solutions. Besides ransomware there are more risks that can take down your business like zero day vulnerabilities, supply chain attacks, unmanaged devices, phishing, cloud misconfigurations, and the list goes on … this is why you need an automated way to help fight against these types of attacks with the ability to recover.

There was a lot of talk about adding AI and also ChatGPT to existing and future product portfolios and there seems to be a race to who will be able to do it first and make that press release or partnership public.

With AI/ML you have to be able to detect those anomalies in your data and watch the model learn based on the behavior of the different types of data and frequency of change. You can also automate manual tasks like scanning for encrypted files or even recovering your data into a completely separate environment for disaster recovery testing purposes. Cyber criminals are executing attacks in both on-premises as well as cloud environments, so no digital location is safe.

Also, SIEM - Security Information & Event Management, SOAR - Security Orchestration Automation and Response, XDR - Extended Detection and Response was a rolling component which appeared in almost all of the sessions that I attended and how those capabilities and enhancements are increasing and offering more APIs and integrations with things like backup solutions.

During Dennis (shoe,uh - Xu) (Sr Director Analyst at Gartner) session where he talked about “Technical Insights: How ChatGPT Can Improve Security Operations” and a few key insights that I gained were:

• There are certain risks and misuses to utilizing ChatGPT right now where it provides wrong answers or hallucinates altogether. It’s only a matter of time when it will be able to connect to the internet.
• How ChatGPT can fit into SecOps like with detection engineering, incident response, training, operationalizing and measuring your SOC - Security Operations Center, and even vulnerability management and attack surface management in the future.
• Make sure you define standards around data sanitization to protect things like PII (Personally Identifiable Information), PHI (Protected Health Information), IP Addresses, Usernames, and geolocations
• The future of ChatGPT and SecOps will grow and its skills will increase, the rise of generative AI from existing providers, more vendor integration into things like Microsoft Security Co-pilot, the democratization of SecOps, and ways to help reduce MTTR (Mean Time To Resolve) or MTTD (Mean Time To Detect)

In your SecOps department, put in place a management and supervision system for the utilization of Generative AI tools like ChatGPT. It's essential to specifically focus on data cleansing and privacy to ensure optimal operation.

Recognize that ChatGPT is not meant to supersede your experienced SecOps engineers. Instead, it serves as a powerful tool to enhance their performance, helping them accomplish their tasks more proficiently.

Cyber Recovery Strategy

Make sure your data is stored in an immutable, indelible, WORM like fashion within a Zero Trust architecture with intrusion protection, malware scanning, an independent compliance clock, and with FIPS 140-2 compliant data encryption.

A super informative session that Rhonda Childress, (VP, Chief Innovation Officer & Security) & John Greenough, (Head of Market Relations, both at Kyndryl gave on “What They Learned From 100+ Cyber Attack Recoveries” and a statistic that was hard hitting was that 92% Have experienced an adverse event that compromised or disrupted their IT in the past 2 years, including malware, hardware failures, DDoS attacks and data center outages.

They gave a formula that business continuity + disaster recovery + cybersecurity = cyber resilience which means your organization has the ability to anticipate, protect against, withstand and recover from adverse conditions, stresses, attacks and compromises of cyber-enabled business.

Similar to the NIST cybersecurity framework, they proposed:

Anticipate - know your toolset, continually test, strengthen your logs, and align recovery to business objectives

Protect - monitor your data, guard identities, patch vulnerabilities, and secure your devices

Withstand - complete forensics first and know who to call

Recover - track your data recovery and be internally transparent

Finally, Wayne Hankins (Sr Director Analyst at Gartner) led a session on the “Top 7 Critical Checklist Items When Faced With Ransomware Attack”

1. Managing stress - it’s a very stressful time when your systems are down and the clock to recover is ticking. You should have regular drills and recovery practices for your teams.
2. Have a Copy of Your Incident Response Plan and Ransomware Playbook Handy - you need to practice how you would respond and recover during a cyberattack and be ready
3. Execute your containment strategy - a containment strategy is a plan that aligns infected systems with a containment method and utilizes the information learned from the analysis phase so you can make an informed decision. This is where you know your last known clean copy, your backup point, when the infection happened, link all of that to your defined RPO or Recovery Point Objective and execute on your RTOs or Recovery Time Objectives with your expected recovery and required recovery times which hopefully will not take you past your MTR or Maximum Tolerable Recovery
4. How are you doing on time? - remember your executives will have to make big decisions with little or no information so don’t lose track of the time and what’s at stake
5. Are you ready to recover from backups? - going back to the old 3-2-1 rule where you have 3 copies, 1 offsite on at least 2 different types of media + make sure it’s immutable and indelible. An added bonus is having a secure Isolated Recovery Environment that is air gapped and securely stores your backup images, OS installation, application, and deployment images and patches. Use multiple methods with a combination of local snapshots and backups, all immutable replicating or syncing to a DR site, the cloud, isolated archive and your primary data center to increase your bandwidth recovery options.
6. Do we pay the ransom? - Wayne provided a stat that 80% of organizations who pay the ransom incur another attack and 68% who paid the ransom once were hit again in less than a month for a higher ransom - the overall lesson is that multiple extortions is increasing and paying is not a protection strategy
7. Lessons learned reminder list or incident response checklist - a key takeaway is that Ransomware planning has very significant legal implications. And, make sure you can answer the question - Are your lawyers involved in the planning process? Ensure a line of communication with your cybersecurity insurance provider is open, and continue to rehearse, plan, test, and practice.

A related separate story that I would like to share for you to understand the implications of everything I just mentioned where a Financial Group was fined over $4M for Cybersecurity Failures in May of this year.

The claims were that a review of their cybersecurity policies uncovered deficiencies in compliance and internal controls that led to an enforcement investigation regarding violations of the Cybersecurity Regulation. The findings included the following:?

1. Inadequate Cybersecurity Policy Documentation - The organization lacked sufficient paperwork to properly enact and uphold a written policy for addressing business continuity and disaster recovery planning resources (BCDR). The existing BCDR was deficient, as it failed to record the necessary requirements, functions, and interconnections of their system.
2. Excessive Access Privileges - The need to restrict access privileges presented several problems, including the storage of crucial passwords in a shared folder labeled “PASSWORDS.” This vulnerability could potentially enable anyone with access to the shared internal drive to modify, erase or relocate the folder.
3. Application Security Issues - Despite conducting extensive in-house software development, the firm's written policies for safeguarding information systems and nonpublic information (NPI) lacked a formal methodology that covered the complete software development lifecycle.
4. Shortcomings in Cybersecurity Staff Training - The company failed to monitor effectively or thoroughly implement training for over 500 IT employees. Secure coding training was also not provided to developers, which could have aided them in spotting security flaws in their applications.
5. Third-Party Service Provider Security Policy Problems - Although the company had vendor risk assessments, which matched the risk level with the necessary diligence level to be conducted, the company did not promptly carry out due diligence on some high and medium risk vendors. Certain vendors were permitted to start work without completing onboarding security questionnaires. Moreover, when a vendor mishandled NPI, the company did not revise risk scores but merely ended its contract with the vendor without reinforcing its own third-party policies.

So, rethinking recovery is not about just having good backups that you can restore from.

It’s being intentional about how you are storing your backups. Getting the security, risk and compliance and governors, backup storage networking teams, all in the room, and on the same page understanding planning and executing, the proper security controls in your environment in order to prevent what just happened to this previous financial services company.

No longer are we in a world where back ups happen in asylum. We are in a global world where security is of the utmost importance, and at the forefront, and also integrated into back up platforms from the majority of all vendors, making sure that you’re fast with software as a service solutions also protected, and you have a strategy around backing up and also recovering that data in a granular fashion and also making sure that that’s implemented into your recovery plan as well.