Severity and probability both affect risk; but do they have the same effect?

The answer seems to be an overwhelming no based on responses to a question I asked in a recent LinkedIn poll:

Note that the question did not specify how a reduction would be achieved, or the level of difficulty one might encounter in reducing severity compared to reducing probability. It simply asked whether reducing severity by 1 level, assuming that probability remained the same, would have the same effect on risk as reducing probability by 1 level at the same level of severity.

A vast majority of respondents (85%) did not agree with the statement that the two approaches would have the same effect on risk. There was a lot of discussion in comments, but the most direct response came from Alex Saegert P.Eng, CRE :

Simply put, if severity is 4, and occurrence is 3, so that SxO is 12, reducing severity by 1 would make the SxO 9 (3x3) but reducing occurrence by 1 would make the SxO 8 (4x2).

Here, Alex is referring to the common industry practice of assigning a rank/level each to severity and occurrence (i.e. probability), and multiplying the two as one way of estimating the risk level. Note that there is no mathematical basis for this approach, but it is similar to the RPN approach used in FMEA (Failure Modes and Effects Analysis). A Risk Priority Number (RPN) is calculated by the multiplying ranks/levels of severity, occurrence and detectability on an ordinal scale. As an example, if a 1-5 scale is used to assign a rank/level each to severity, occurrence and detectability, RPN can range from 1-125 for each risk item. However, RPN does not represent a quantitative value of risk; rather it is used as a way to prioritize which risk(s) should be addressed first.

Other comments focused on prioritizing severity but highlighting that it may be more difficult to accomplish in practice. One commenter challenged this perception and provided multiple examples from the automotive industry where risk control measures such as safety belts have been highly effective in reducing fatalities and severe injuries. A few other comments pointed out differences in the scales used for severity and probability. While numerical probability ranges can be used for each of the 1-5 levels, a quantitative value for severity (of harm) is not feasible.

A new question emerges - if severity and probability do not affect risk equally, then does one affect more than the other?

Risk, according to ISO 14971:2019, is defined as a combination of the probability of occurrence of harm and the severity of that harm. The standard, nor its companion guidance ISO/TR 24971:2020, is silent on details of how these two attributes of risk should be combined.

A common practice in the medical device industry is to map each individual risk as an ordered pair of its severity and probability levels on a two dimensional risk matrix. As an example, three individual risks (of harm) are mapped on a 5x5 risk matrix illustrated in Figure 2 below:

Here, risks A(5,4), B(4,3) and C(2,5) are mapped on a risk matrix based on their respective severity and probability levels, with severity as the first value in each ordered pair. As an example, A(5,4) represents a risk with severity level 5 and probability level 4.

Arrows represent the direction of movement of each risk when either severity or probability is reduced by 1 level.

As an example, if we reduce severity of risk A(5,4) by 1 level, from life threatening to critical, it moves to a new position in the risk matrix, represented by A'(4,4). Alternatively, if we reduce probability by 1 level, from probable to occasional, it moves down to a new position represented by A"(5,3). We can imagine a similar movement for risks B(4,3) and C(2,5) to their new positions in the risk matrix.

So, let us ask the original question in more precise terms:

Can we say that reducing risk A(5,4) to A'(4,4) has the same effect as reducing it to A"(5,3)?

Can we say that reducing risk B(4,3) to B'(3,3) has the same effect as reducing it to B"(4,2)?

Can we say that reducing risk C(2,5) to C'(1,5) has the same effect as reducing it to C"(2,4)?

Now, assuming that a majority of respondents still do not agree, then a new question emerges - which of the two movements has a greater effect on risk?

This question is not easy to answer objectively.

One reason is that we are not able to quantify the harm associated with each risk in measurable terms. How do we measure the value associated with loss of life? Or a permanent impairment that results in reduced quality of life? Or a serious incident that sends us to the hospital but fortunately, we are able to recover fully?

If were looking at financial risk, we could easily calculate the expected value of the loss (or gain) by multiplying the value of the potential outcome by probability. As an example, if A(5,4) corresponds to a situation where S=loss of $1M and P=0.5, then we could say that the expected value at this risk level is a loss of $500,000 (multiply $1M by 0.5). Now, if we could reduce the impact by 1 level to a loss of only $100,000, the new expected value at this level would be a loss of $50,000 (multiply $100,000 by same probability of 0.5). Alternatively, if we could reduce the probability by 1 level to say 0.05, the new expected value at this level would again be $50,000 (multiply $1M by 0.05).

Notice that we used a logarithmic scale for both severity and probability in this case, where each level was separated by an order of magnitude (i.e. by a factor of 10). Further, we had a precise value for the financial loss at each level of severity.

Unfortunately, there is no such quantitative measure for the severity of harm that can be used to estimate an expected value of risk. As a result, practitioners in the medical device industry tend to evaluate risk reduction subjectively. Differences in risk perception play a significant role in risk control decisions and judgments about risk acceptability.

Context is important to understand the interplay between severity and probability, and judgments about risk.

The US FDA classifies medical device according to their risk level:

The FDA categorizes medical devices into one of three classes – Class I, II, or III – based on their risks and the regulatory controls necessary to provide a reasonable assurance of safety and effectiveness. Class I devices generally pose the lowest risk to the patient and/or user and Class III devices pose the highest risk.

However, there are no clear criteria to determine if the risk is low or high. Should risk be considered in discrete levels as low, medium or high? Or is there a continuum ranging from low risk to high risk?

A close look at FDA communications related to device malfunctions, recalls and warning letters gives an impression that risk is judged primarily by severity. Even if the probability of occurrence of harm due to an alleged device malfunction is practically zero, FDA would consider it to be serious if death, permanent injury or a life-threatening condition could occur. Risk is judged in terms of possibility, not probability for regulatory purposes.

No surprise then, that there is a clear severity bias in the practice of risk management in the medical device industry. Generally speaking, we consider risk to be high when severity is high, and low when severity is low. As an example, there is a common belief in the industry that a risk of severity level 5, which generally reflects a life-threatening condition including death, should not be considered as acceptable regardless of the probability of occurrence.

Whether it is right or wrong, is not particularly relevant; but how this perception affects our judgment about risk acceptability and decisions about risk control measures merits a thoughtful consideration.

Let us revisit the three individual risks A(5,4), B(4,3) and C(2,5) illustrated in Figure 2.

Given the current perception in our industry, most practitioners would consider risks A and B to be high, and in the "red zone" of their risk acceptability matrix. Risk C on the other hand, may be considered as either medium or low.

When asked if a 1 level reduction in severity would have the same effect on risk as an equivalent reduction in probability, the answer may depend on whether someone is thinking about risks A and B, or risk C.

If they are thinking about risks A and B, it is likely that they would consider a 1 level reduction in severity to have a much higher influence on risk. Lowering the severity from level 5 to 4 for risk A will be perceived to have an outsized effect on the residual risk, compared to lowering probability from level 4 to 3 if the severity still remained at level 5.

An equivalent reduction in severity and probability for risk C, on the other hand, may be perceived to have a similar effect on residual risk. This may be one reason why 15% of the respondents agreed with the statement in the poll.

Conclusion - risk perception influences how we think about the effect of severity and probability.

Risk is a combination of severity of harm and probability of its occurrence. There is no clear cut answer on how the two are combined and which of these two attributes has a larger effect on risk.

In the absence of a quantifiable measure for the severity, risk perception is often disproportionately influenced by severity, Even if the probability of occurrence is low, risk may be perceived as high if the severity of the potential harm is high. The mere possibility of occurrence of a high severity outcome, that may result in death or a life-threatening situation, is enough to dominate our risk perception. Therefore, we may consider a reduction in severity to have a disproportionate effect on risk compared to an equivalent reduction in probability.

Risks that are perceived to be low, because the severity of potential harm is low, may elicit a different response. We may treat both severity and probability to have a similar effect.

References

1. FDA: Overview of Medical Device Classification and Reclassification, accessed 7th Nov 2024.

You may also like

===========================================================

??Like this article? Click here to unlock exclusive subscriber-only content.