Do you have too many OOS results?

It might not be due to bad products.

It is probably due to Measurement System not fit for purpose.

Background

Working as a data analytical consultant in the pharmaceutical industry for many years, I have too often seen that measurement systems is the cause of non-conformities. When investigating Out of Specification (OOS) issues, it is in most cases concluded that the cause is the measurement, not the product.?

How can this happen again and again in an industry that use validated measurement systems? Apparently even though measurement method validation has been performed and passed, it does not ensure that the measurement system is fit for purpose.

As I see it, there are 2 main reasons for this:

1.???The acceptance criterion for method validation does not ensure that intermediate precision only takes a small part of the release tolerance

2.???The acceptance criterion for intermediate precision is not enforced with confidence, but typically passed based on an estimated value

Even though the solutions to the points above are straightforward and described below, they are typically not implemented. I think the reason is a combination of not knowing how to do it right (which this article try to solve) and not wanting to know the true result (which is will not try to solve here).

Issue 1: Large Intermediate Precision to Tolerance ratio.

Classically a Standard Deviation or Coefficient of Variation (relative standard deviation) is estimated when qualifying a measurement system in an Intermediate Precision Study. Several parts are measured repeatedly by different levels of the reproducibility factors (e.g., analytical run, day, operator). The acceptance criterion is typically in absolute units.

We often see that the acceptance criterion for the standard deviation is large compared to the +/- release tolerance. It is not rare that it is up to 50%, i.e., specification limits are only 2 standard deviations away from target. This means that a measurement system that will measure 5% of on-target products to be OOS, will pass the measurement method validation.

Solution: Enforce the ratio between the Intermediate Precision and the +/- Tolerance is lower than 10%.

For some measurement systems it might be hard to meet this goal. If Intermediate Precision cannot be reduced there are 2 options:

1.???Loosen the tolerances. I see many tolerances set tighter than necessary.

2.???Take averages of multiple measurements to average out measurement noise. Be aware that if the primary variance is between analytical runs (typically the case), it must be averaged over measurements in different analytical runs.

Issue 2: Intermediate Precision without confidence.

Most companies qualify their Intermediate Precision without confidence, although guidance’s for how to do it with confidence (like ISO GUM) have been there for years.

Typically, only few levels (classically 3) of reproducibility factors (e.g., analytical run, day, operator) have been used, making the estimation of reproducibility very noisy. There can be a large difference between estimated and true value. Even though the estimated precision is fine, the true value might not be, leading to false approval of measurement systems.

The typical textbook example is:

·????????10 parts

·????????3 days or operators

·????????Each combination measured 3 times

This study will have in total 10*3*3 =90 measurements and might seem statistically sufficient. However, let us look at how many replicates (Degrees of Freedom) we have. For repeatability we have a lot. The 30 combinations of Part and Day are each measured 3 times (2 replicates) leading to 10*3*2 = 60 Degrees of Freedom. This is more than enough. Measuring each combination only 2 times instead of 3 times, would lead to 10*3*1 = 30 Degrees of Freedom, still enough, so room for saving here.

But what about the between Day differences? Here we only have 3 Days and thereby 3-1=2 Degrees of Freedom.

Intermediate Precision is the sum of Between and Within Day Measurement Variation, that in the textbook case have 2 and 60 Degrees of Freedom respectively. The Degrees of Freedom on the sum can be calculated from Satterthwaite’s equation. The result will be somewhere in between 2 (if mainly between Day Variance) and 60 (if mainly within Day Variance).

In the graph on top Degrees of Freedom (blue curve) are plotted versus the ratio of Between and With Day Variance, so-called Variance Ratio (VR).

On the graph is also shown the ratio of the upper one sided 95% confidence limit and the estimated value (red curve) for the intermediate prediction standard deviation. For a typical Variance Ratio of 10, the upper confidence limit is more than 3.5 times higher than the estimate. This can dramatically change the conclusion on whether acceptance criterion is fulfilled or not.

Solution: Enforce Intermediate Precision with confidence.

Although statistical software packages in their Measurement Systems Analysis Platforms typically only give estimated Intermediate Precision, it can easily be calculated with confidence by going to their modelling platforms with random factors. With a little bit of scripting, this can even be fully automated.

It needs to be included in the design of the study that more days are needed if Variance Ratio is expected to be high.

Conclusion

By enforcing Precision Tolerance Ratio < 0.1 with confidence, it can be ensured that measurements systems are fit for purpose.

If requirement is hard to fulfill consider loosen tolerances and/or take averages of multiple measurements. The latter might increase QC costs, but lower non-conformity costs even more.

If a large Variance Ratio is expected more than 3 levels of reproducibility factors might be needed.

?