The Rock and the Hard Thing

Between the Rock and the Hard Thing

This article investigates the crisis that can develop when a Disaster Recovery Plan (DRP) is offered to a Client as a service but that service proves to be untested, incomplete, or even not in place at all.

There are three internal players here as listed below. There is also a fourth?player, the external Client, but that is another story.

So the players, for now, are:

1.???A marketing team charged with selling a DR service. Let’s call them the Rock.

2.???A computer application team charged with developing a technical DR capability. Let’s call them the Hard Thing (HT).

3.???A Business Continuity Team (BC) charged with co-ordinating this DRP sub-project. Let’s not call them at all, they will turn up anyway.?

So, three Teams with different views of the same thing. A dive into what they are all about is called for but before that we should be clear about the joint mission. What are they all trying to do ??

Just this:?

On behalf of the Client a tested DRP will be developed and delivered as an integral part of the overall Client service. This DRP will provide effective resilience for the Client in the event of unexpected service failure. The DRP will be shown to deliver recovery within the time and data objectives.

It too often isn’t, but this mission can be achieved, it is simply that, like cooking chips, it is not as easy as it looks. The consequences of that achievement include Client satisfaction, enhancement of reputation within the business sector and beyond, a minimum amount of re-work/maintenance and a maximum amount of revenue accrued. The consequences of failure are the opposite. “Why” we will look at later, but for now back to our Teams.

?

?

The Rock works to sell to the Client a DR capability as part of the overall package. The Rock strength lies in their understanding of their three key drivers: profit, repeating business and auditable compliance. Their weakness is not fully understanding of the capabilities (and limitations) of the DR product or the resources (time, skills, IT configurations) it takes to develop, test and then maintain that DR capability.

?

The HT works to develop the DR capability, based on the recovery objectives and priorities agreed with the Rock. The HT strength lies in their technical ability to convert the objectives and priorities into an effective technical recovery capability. Their weakness is not fully understanding the DR development schedule and costs agreed between the Rock and the Client.??

?

The Business Continuity Team works to co-ordinate the efforts of the Rock and the HT in such a way as to deliver the DR capability on time and to specification.?The BC strength lies in their methodical and documented approach from Risk Assessment to Business Impact Analysis (BIA) to Disaster Recovery planning and testing. Their weakness lies in that they do not fully understand either the commerce of the Rock or the technology of the HT. They do, however, control the BIA process which effectively translates between the Rock and the HT.?The BIA guides the behaviour of the Rock towards the Client and the thinking of the HT towards the DRP. The BIA acts as broker in the eternal balancing act : The more the Client tends to pay for the product the better it will tend to be. [how sure are we of that ?]?

?

?

So, what can go wrong ?

There is a series of links, or milestones, in the chain between the dream of full resilience and the reality of an active DR capability. Here are some key ones with their prime deliverable (which should be documented, stored and distributed) as an insert:

·?????[1] Rock and the Client agree the need for a DR capability.

o??Clarity of what is required

·?????[2] Rock and the HT agree the recovery to be offered to the Client.

o??A definition of what will be delivered

·?????[3] BC, agree with the Rock and HT the proposed recovery service.

o??The technical architecture of the recovery solution

·?????[4] All three agree the cost estimate to be offered to the Client.

o??The business case for the recovery solution

·?????[5] All three agree to monitor progress and costs.

o??The Project Management plan for the development process

·?????[6] The BC and the HT agree to jointly develop the DRP.

o??DRP to be verified and tested

·?????[7] The Rock and the HT agree to an internal DRP Test run by BC.

o??Run Book for the DRP Test

·?????[8] The HT agree with the BC to update the DRP based on the test.

o??Internal post Test report

·?????[9] The Rock invites the Client to sign-off on the service.

o??Approval of the DR capability by the Client

·?????[10] The BC agrees with all the annual DR re-test schedule.

o??Update to the DR Exercise plan

As you can see all the key links are based on open agreements. This is a good thing because the most common failing of a DR facility is where people who actually need to know about DR, don’t know. This [1] to [10] list is not cast in stone, but it is better than being cast adrift.

Consider (in your own time) the implications of the failure to agree of each of the [1] to [10]. One thing is sure, we are unlikely to even get to the happy place of [10] if the chain of development is broken, and if we do it will have cost us dear.

?

?

There is one other thing to be understood before we can proceed to an action plan and that is Risk Assessment.

How likely is it that any given destructive event will result in a disaster, and what actually constitutes a “Disaster”. What are we protecting against ?

Some threats are constants such as fire, flood and epidemic. Others are specific to the business being protected. Does, for instance, the business handle sensitive or personal data ????????

These considerations inform agreement [1] above and also set the scope of the overall DRP.?Here are a few questions on this topic to spur thought:

·?????If the business handles personal data how will the DR be tested without violating the rules of Data Protection ?

·?????If an epidemic strikes the work force (again) how will we know that we still have the base skills to carry on ? Will Human Resources be involved to guide on this point ?

·?????Will the DRP clearly address any requirements for ISO compliance or certification ?

·?????Is the business dependent on 3rd Party services and if so do they in turn have a tested DR capability ?

·?????Is the business sited in a region of instability or even hostility and if so does the DRP identify safer sites and the way of migrating to them ??

·?????If failover to a DR site is the technique used in the DRP, is that DR site as secure as the home site and does it meet the same Service Levels ?

Two final points for now, one general, one specific, both vital

1.???Generally invest in training in Business Continuity for all parties

2.???Specifically make sure that the contact names and numbers in the DRP are kept up to date.?

So there we have it. I hope you found this interesting, even useful.

Do respond. I’m trying to make sure that I have my answers questioned.

?

Comments welcome on [email protected]

Roger Jarvis MBCI, Fulham, London. November 2022