Do a Risk Analysis on your Risk Analysis

Risk- and Criticality Analysis is broken in many organizations without them even realising it. If your organization uses a matrix or scoresheet to calculate risk or criticality then you are using a broken tool, and you are in peril of making terrible business decisions that will destroy value.

If you read this article I hope to show decisively that scoresheets and matrices are broken methods, that there are better ways to evaluate risk, and what you can do about it.

Scoresheets and Matrices are Broken

Scoresheets with ordinal scales and Risk Matrices are generally referred to as examples of Qualitative Analyses. These were introduced to businesses back in the 80s, because they seemed more intuitive and fun than the alternative. They have little to no scientific backing and they are usually 1. highly subjective, 2. mathematically wrong, 3. based on arbitrary values, and 4. worthless for business cases and decisions.

Let's have a look at a typical Risk Matrix and discuss the issues that arise. The figures below show the components from a weighted 5 x 5 x 6 matrix.

The first problem I'd like to point out is the language, the hallmark of qualitative analyses. The words used are words like 'serious' 'minor' 'remote' 'occasional' etc, i.e. these terms are not quantities, they are 1. relative or subjective terms. This supposes that everyone has the same idea for each term, but we know that a maintenance manager may think that regular is something that happens every week whereas the Operation Manager thinks it is something that happens every other month.

To compensate the designers of the qualitative frameworks try to guide the users in its interpretation. On Probability in the figure above for instance they'll have a guide that looks like this scale below:

With this guide the user will more consistently get to the qualitative term, but there is a serious mathematical error with this guide, namely 2. mathematical compression. You simply have to look at the transition points to understand what mathematical compression is, according to this guide for instance:

• something that happens every 4.9 years (Remote = 2) is half as probable as something that happens every 5 years (Improbable = 1) - this guide's logic is therefore: (4.9/5) == (1/2).
• something that happens 100 times per quarter (Frequent = 5) is equally probable to something that happens once per quarter (also Frequent = 5) - this guide's logic is therefore: (100/1) == (5/5)
• something that happens once per 6-months (Probable = 4) is 20% less probable than something that happens once per 3-months (Frequent = 5) - this guide's logic is therefore: (3/6) == (4/5)

All of these show the absurdity of ordinal scales, but this seems to be ignored by most managers, since they pop up everywhere in today's business. The figure below shows the same ordinal scales and quantitative language popping up with the consequence analysis of this risk matrix.

The figure above shows another problem with many qualitative analyses: 3. arbitrary (subjective) weightings. Intuitively we know that some areas shown in the figure above are more important than others, so designers of such a matrix give 'weights' to each area. I have, however, never encountered a matrix or scorecard where the weightings were objectively calculated, they were always subjective and based on 'gut feelings' that may or may not have been an accurate value.

When the risk analysis is performed, the risk score is put through another ordinal scale (such as the one shown below) to turn scores into categories, also based on arbitrarily selected cut-off points. To be fair though: usually this last distinction is not made unless an ABC-indicator needs to be maintained. From here the risk managers rank risks according to risk score, not risk category, so that they can manage the top 10 or top 20%

Once the score or category is determined the manager sits with the another problem to answer "how can I compare score or category with the costs required to drive it down?". Can he easily answer a question like "is it worth $1,2m per year to reduce the risk-score of a machine from 403 to 288?".

When such a question is posed, the managers will say that this is not sufficient information, they need to know more about it to answer. But If you asked the same managers "is it worth it to spend $1,2m per year to increase annual throughput of product X by 120,000?" you'll tend to get an answer without having to provide further context.

When you do provide context to the risk-score problem, then differences tend to erupt. The maintenance manager insists that it is life and death to replace the rope-shovel, whereas the financial manager politely disagrees and feels it will impact the bottom-line too negatively. Either way, you'll find a disagreement on what the risk-score-reduction is actually worth to the organization, and only after much more deliberation, further studies, and assessments from subject matter experts, is a final decision obtained. And even then it is not evident whether it was the right decision, because apples (risk-score) was compared to oranges (costs). If it is not possible to do ROI or NPV calculations to make a proper business decisions around risk reduction, then the risk-score or risk category is essentially 4. worthless.

I recommend that you read "The Failure of Risk Management" from Dr Douglas Hubbard, who does a much better job than I to explain why most qualitative analyses such as scorecards and matrices are no better than snake-oil.

Quantitative Analysis is much better and not as scary as it seems

The alternative to Qualitative Analysis is Quantitative Analysis, the approach based in statistics and used in Actuarial Science.

Quantitative Analysis is generally acknowledged as superior, but 1. too complicated to use, 2. not intuitive for non-mathematical people such as maintenance supervisors, 3. too much effort or too expensive to get usable results, and 4. it is not possible to quantify everything (such as safety). 

Quantitative Analysis does seem more scary than Qualitative Analysis because the mathematics in it has exotic terms such as stochastic, parametrization, Monte Carlo, Markov Chains, t-distributions, binomial, kurtosis, inverse functions, and so forth. But I do not really see engineering personnel shying away from CAD software just because they do not understand the underlaying theory of the algorithms that drive it. The same goes for Quantitative Analyses, 1. you do not have to understand the underlaying mathematics and algorithms of the model, you simply have to know how to feed it with the right information.

The right information may seem as though it requires a great deal of statistical savvy, but when you have a quantitative model that handles uncertainty, then it becomes more intuitive than a qualitative model. For instance, in my dissertation, an adequate answer to the question "what happens when this part fails?" is normally something that sounds like "the machine is down for 4 to 6 weeks, we have to get a contractor to pick up the slack, the part is $1,2m to replace… but can be as high as $1,5 if it's not in stock and we have to expedite its purchase… and we start paying penalties because of missed shipments…" Ask them to put their head on block to put these values into a qualitative model and they'll be too conservative (always opting for worst case scenario), but in a quantitative model it simply accepts the inputs as provided with the confidence intervals. The maintenance personnel from artisan level upward were astounded, it felt as though the solution 'understood their language'. So 2. using a good quantitative solution is more intuitive than qualitative methods that will have them arguing about what is the difference between localised leak and major leak.

Managers know that their operations are complicated and have a lot of variation in it, and they are often reluctant to try and model it because it feels that the model will never be able to get to usable results. My experience told me that this wariness came from previous attempts at modelling that failed, usually because these attempts:

1. Tried to model the operation in too much detail from the bottom-up and got overwhelmed by the amount of information required to feed it
2. Tried to model the operation with oversimplified approaches from the top-down and got answers that was far removed from reality (or really suspect).

I can write another article for another day on how to approach modelling, but for now it should suffice to say that every prediction model MUST account for the variations and uncertainty in its inputs and parameters, otherwise it is doomed to become a white elephant. 3. If your model simply starts with the ability to handle uncertainty then the effort of modelling becomes a fraction of what you thought it would be. Also remember that we're in 2017. Back in the 80s when scoresheets and matrices were introduced there was not an abundance of computing power so it was a valid justification that the modelling was resource intensive, but in 2017 we have computers that handle spreadsheets with a million rows and thousands of columns easily.

The most contention I was confronted with so far was the notion that not everything can be quantified. Things like safety, reputation, and environment. My contention is that they can be quantified, but it is just not always considered couth to do so. It is not pleasant to put a $ value to a loss of limb, or fatality, but that it exactly what Actuarial Scientists do on a daily basis to determine your insurance instalments. Reputation is linked to your forecast/outlook and share price, environment is linked to fines and rehabilitation costs, legal is linked to the legal proceeding costs and section 54s, and so on. I recommend that you read "How to measure anything" again from Dr Douglas Hubbard that shows quite nicely that 4. you are indeed able to quantify much more than you though you could.

Quantitative models are not without faults and challenges, but they are vastly superior to qualitative analyses in every aspect. Especially when it comes to making good business decisions. Let me demonstrate by example: which question is easier to answer and will lead to the least contention?

• is it worth $1,2m per year to reduce the risk-score of a machine from 403 to 288?
• Is it worth $1,2m per year to reduce the risk from an average $1.8m per year to $1,2m per year?

Even though the second question clearly has the answer of NO, it is a much better decision.

So What Now?

The challenge with addressing a qualitative risk framework at your organization is 1. it normally belongs to a Governance, Risk and Compliance department that have set the risk framework policy or standard and require that everyone abide by it, 2. it was implemented and used by a number of engineers or maintenance personnel and they now feel too heavily invested in it to abandon, 3. it still feels too comfortable to use (fail together instead of succeed alone), or 4. some try to marry the qualitative and quantitative approaches to get 'the best of both worlds'.

If your organization has a 1. risk framework policy or standard that you have to abide by, then you must use it. But start with awareness campaign of its issues, and at the same time show the owners of the qualitative risk framework a better way of doing it.

If you 2. feel too invested in the qualitative matrices to abandon it, then please consider this: ITS RESULTS ARE WRONG AND USELESS! Would you continue to base your business on a belief that you'll get money from the tooth fairy when you collect buckets of teeth just because you have already invested in the buckets? You can use the qualitative rankings to prioritise the sequence of equipment due for a quantitative analysis. Based on my experience, you'll see the criticality rankings shift dramatically, but at least you have some use for the qualitative rankings.

If you feel 3. everyone else is doing it qualitatively, so I'll rather stick to that, then please consider this: ITS RESULTS ARE WRONG AND USELESS! In business it is not acceptable to destroy value because everyone else is doing it. For your business to be a leader and rise from the competition then it is a small concession to refuse approaches that destroy value. If you insist on sticking to such a dysfunctional method, then why do the effort. You can just abandon it and go with your gut in anyway. Why spend the money and effort to come up with a worthless result in anyway?

I have encountered organizations that tried a 4. hybrid qualitative-quantitative models, but they suffer from all the deficiencies I listed in the section discussing qualitative methods. Qualitative Analyses really destroy any kind of accuracy or worth of a model, it is best to just avoid them to begin with.

Take Aways

1. Qualitative Analyses (with ordinate scales) are so bad, that you might as well drop them entirely.
2. Quantitative Analyses are always better than Qualitative Analyses, even when it 'feels' as though the Quantitative Analysis has too much uncertainty in it.
3. Do a Risk Analysis on your Risk Framework - i.e. determine what is the potential impact of getting decisions wrong versus the probability of getting them wrong

If you want to discuss this subject more, feel free to leave comments or contact me. This is a subject I am passionate about.