The Risk of Using Risk Matrices

[This is a repost and update of an older post but now in a LinkedIn article format]

This summarises research on Risk Matrix (RM) limitations and adds some new limitations.

First, purported benefits of RMs is that they are intuitive & simple, and easy to explain and score.

However, the authors note that the development of RMs has taken place isolated from scientific research in decision making and risk management. Research and practice commonly positions RMs as a best practice technique, but do not discuss the possible detriments of using them.

Some identified issues:

• Risk acceptance & inconsistency: Inconsistent ways to prioritise outcomes, where colour codes (red, yellow) aren’t aligned to Expected Loss; making it hard to prioritise issues. 
• Range compression: flaw that compresses the distance between risks resulting in identical ratings for different quantitative risks. E.g. a well blow-out may be far worse than a loss of well control but the RM may not show the difference effectively.
• Centering bias: tendency for people to avoid extreme values or statements. As highlighted in the paper, one study found that 75% of chosen values on risk matrices were scores of 3 or 4. This exacerbates range compression.

Regarding inconsistency in the colour coding, in some RMs it’s evident that some “cells designated as red were “less risky” than some of the cells that were designated as yellow” (p56). For range compression, it’s said to be “unavoidable when consequences and probabilities are converted into scores” (p59).

Furthermore, how you rank the scores – ascending or descending – is also a consideration in RMs. With ascending scores, severe losses on an offshore rig will be prioritised for risk mitigation, but with descending scores, a blowout will be prioritised instead. Therefore, quoting the authors “RM rankings are arbitrary; whether something is ranked first or last, for example, depends on whether or not one creates an increasing or a decreasing scale” (p61).

As argued in the paper, RMs can “produce arbitrary decisions and risk-management actions”, where some of “[t]hese flaws cannot be corrected and are inherent to the design and use of RMs” p. 64. That is, in some uses and configurations RMs will likely always produce some arbitrary recommendations (p60). Expanding on their arguments, they argue that many proponents of RMs extol the benefits of RMs but the commonly used scoring system is said to distort the scales and “removes the proportionality in the input data” (p63).

RMs are said to possibly “foster miscommunication and misunderstanding ... [and] result in misallocation of resources and the acceptance of suboptimal levels of risk” (p60). This miscommunication is said to possibly create an “illusion of communication” (p60).

The authors note that highlighting the clear issues with RMs, such that, in their view, “RMs should not be used for decisions of any consequence” (p63).

Interestingly in stating this, they argue that the fact that RMs produce arbitrary rankings isn’t conditional on whether alternative methods exist or not to replace them – RMs still produce arbitrary rankings irrespective and their outputs are said to be unduly influenced by the RM design.

Authors: Philip Thomas; Reidar B. Bratvold; Bickel J. Eric, 2013, SPE Annual Technical Conference and Exhibition

Link below.