How to calculate and use control limits right.

The 99 years old control charts invented by Walter Andrew Shewhart in 1924 at Western Electric Company have been revitalized in the pharmaceutical industry after the request from Health Authorities to do continued/ongoing process verification.

Control limits are typically calculated from an estimated mean XBar and an estimated standard deviation S with the purpose of calculating an interval where only 0.135% of the population is outside (in each side) by random using the Normal Quantile Z:

This gives an interval of expected future observations if the process does not change. If observations are inside limits, it can be assumed that the validated state is maintained and if they are outside, mean and/or standard deviation has probably changed, which might need attention.

Often S is calculated as a short-term S, based on standard deviation among neighboring observations only. This makes limits more sensitive to drift.

This is a great concept, but we still see issues defining and using the limits right. If the limits are too narrow, you will get false alerts and over steering leading to waste of time and increased variation. If the limits are too wide, you loose time in responding to a problem leading to loss of productivity.

Issues

What are the typical causes we see, when control limits are either too wide or too narrow:

1.???The equation above assumes data are normal distributed. If this is not the case limits calculated this way can both be too narrow or too wide.

2.???For new products, the first set of limits are for obvious reasons based on few data. In the pharmaceutical industry many companies set it up in a way where limits are hard to change afterwards, especially if they need widening. For small sample sizes there is a large estimation uncertainty on XBar and especially on S. Thereby the limits can easily be wrong, typically too narrow. The reason they are more often too narrow than too wide is that you cannot seriously overestimate standard deviation (you cannot see something that is not there), but you can easily underestimate (oversee something that is there), especially in small sample sizes.

3.???Often subgroups are used when building control charts. The control limits are then typically calculated based on within-subgroup S to get a short-term S. By the same time building a control chart on a mean XBar of the subgroups (XBar-R or XBar-S charts), you solve the issue in bullet 1.?Mean values can always be assumed normal distributed if sample size is big enough (Central Limit Theorem). However, if you have systematic variation in the subgroups, they will be too wide and if there are unavoidable variation between subgroups, they will be too narrow. In addition, by only showing averages on a control charts, you do not see all individual observations.

4.???Hardly any process is so much in statistical control, that you can keep future observations inside limits calculated based on a short term S. Limits are in most cases too narrow to be used in practice.

Solution to issues

Fortunately, the solution to the issues mentioned above are straightforward and strangely enough not being used that much.

1.???Transform data to make them closer to being normal distributed. Then calculate control limits in transformed space and transform these limits back.

2.???Use prediction limits that take estimation uncertainty into consideration as control limits

3.???Use Individual on Means chart, instead of XBar-R and XBar-S.

4.???Build a regression model that can describe your data set and include sampling point (time) as random factor in the model. Then use control and prediction limits from this model as fixed limits going forward. This will account for the timely variation you had in historical data and give you a warning if it increases. This requires that the timely variation in historical data is acceptable. If not, we have another problem.

Examples in JMP from SAS on issues

?

1 Data not normal distributed

If data are not normal distributed control limits calculated with the equation above are useless. A classic example of this is bioburden data, that by nature are right skewed as shown in Figure 1.

No observations are even close to lower control limit and there are a lot of alerts above upper control limits. Lower control limit is negative which does not make sense for bioburden data. Upper control limit is equally unreliable.

Performing a Box-Cox transformation of the data, it becomes obvious that we need to log transform data to fulfil the normal distribution assumption. Figure 2 shows a control chart on log transformed data and it is clearly seen that the alerts on Figure 1 were false alerts.

To visualize data and limit in untransformed space, you can just build a model in JMP with log Bioburden as response and then save prediction formula and prediction limits from the model. They will automatically be transformed back as shown in Figure 3 where lower limit cannot become negative.

2 Estimation uncertainty on XBar and S

Control limits are for a normal distributed with mean μ and standard deviation σ:

For small sample sizes this can differ a lot from replacing μ and σ with the estimated values in the last part of the equation.

The solution is to use prediction intervals instead

This accounts for estimation uncertainty on σ, replacing the normal quantile with the t quantile with corresponding Degrees of Freedom DF and takes the estimation uncertainty on μ into consideration by including standard error on the mean. In JMP from SAS you can easily get prediction limits for any model you have built.

You might not always have as many DF as you think. If you calculate the first set of limits after 3 batches with 20 observations in each batch, your effective DF is between 3-1=2 and 3*(20-1) = 57, depending on whether your variance is mainly between batches or within a batch.

Figure 4 shows an example with a large variance ratio (Between Batch Variance divided by Within Batch Variance). Pay attention to the large difference between full red prediction limits (where you have proven future measurements will be within) and the dotted red control limits, which is best guess, based on only 3 batches. We strongly recommend always to use prediction limits as control limits to avoid them becoming too narrow, due to estimation uncertainty. However, this comes with the obligation to recalculate them when you get more data. After recalculation limits will typically be narrower due to more DF and smaller standard error on mean. It is easier to justify a narrowing than a widening. However, we see it as a general misunderstanding in the pharmaceutical industry, that control limits cannot be changed over time, as more and more data is collected enabling even more representative estimates of variation. Specification limits are fixed and cannot be changed. Do not mix these two sets of limits up.

3 XBar-R and XBar-S does often not work in practice

By far the most widely used control charts are XBar-R and XBar-S. It has several advantages:

1.???What you make control chart on can be assumed normal distributed. Mean values (XBar) can always be assumed normal distributed given that the group size n is big enough. However, this could have been solved by transforming data, still allowing control charts on individual observations.

2.???The variance on a mean is n times smaller than the variance on individual observations.?Thereby control limits gets narrower and we get a more sensitive control chart. However, moving average (MA) control charts like uniformly weighted (UW) of exponential weighted (EW) like UWMA or EWMA control charts also have this advantage still showing individual observations.

3.???It calculates short time standard deviation based on the average variation within a subgroup, thereby making the control chart sensitive too drift. But maybe too sensitive.

Unfortunately, this type of control chart also has some SEVERE disadvantages. It is not robust to the real life. It can only be applied if:

1.???There are no systematic variations within subgroups

2.???Variation between subgroups can be avoided

This is easy to write in a textbook, but very few processes can live up to both assumptions in practice.

Systematic variation in subgroups

Figure 5 shows a XBar-R chart on a 4-cavity injection molding process subgrouped by Box. The control limits are clearly too wide, leading to too late alarms.

The reason behind this is that range in a subgroup is not only dependent on the random Box-to-Box variation, but also on systematic cavity-to-cavity variation. In the middle graph we see the individual observations colored by cavity. It is clearly seen that the highest value is always coming from the blue cavity 3. As a result we end up using the sum of random Box-to-Box variation and systematic cavity differences as an estimation of our random variation thereby heavily overestimating it, leading to far too wide control limits.

One solution could be to build control charts per cavity, but if you have 192 cavities, this might not be what you want.

You could also jut choose another mean control chart. Since we are control charting across subgroups, why not calculate random variation across subgroups, instead of within? Then we are not sensitive to cavity differences as shown in Figure 6.

The middle control chart in Figure 6 shows the range between neighboring means (not sensitive to cavity differences, since it is the same cavities in all the means) being used to calculate limits on the mean chart. This chart trends the Box-to-Box differences.

The lower chart shows the range within a subgroup, i.e., it trends the cavity differences. Here control limits are still not right, because this is NOT the range of 4 numbers coming from the same distribution.

However, by Box-Cox transforming the ranges and then saving prediction limits, we get more proper limits as shown with the dotted red lines.

Variation between subgroups that cannot be avoided

Often several measurements are made on each batch and then it is natural to use batch as subgroup. Figure 7 shows an example of this, with 5 measurements per batch.

The process might seem out of control, but the problem here is that we calculate random variation as the average of within subgroup variance, which does not include between batch variances. So, the alerts are because we have between batch variance, which is expected, with the definition of a batch. A batch is a special homogenous production unit, that typically differ from other batches.

Let us again calculate variance across batches, since we are control charting across batches. This is shown in Figure 8. Now we can see the process is in control. The upper chart trends the level, the middle chart the batch-to-batch variation, and the lower within batch variation. What more do we need to know?

4 Hardly any process is in control

If you have more than one variance component and/or you want to accept historical timely variation even the Three-way chart does not work. Figure 9 shows data from a process in which each batch have 7 sampling points and at each sampling points 3 samples are taken. We thereby have 3 variances:

1.???Between Batch

2.???Between Sampling Point

3.???Within Sampling Point

No classical control charts can handle this. We need a unified approach.

The unified approach is to build a model that can describe your data set and then calculate control and prediction limits from this model.

The equation for control and prediction limits with one mean and one variance can be generalized:

Where XBar is replaced by Prediction Formula PF and S is replaced with Stot, the square root of the total variance.

DF is now the effective degrees of freedom for the total variance Vtot, and can be calculated from Vtot and its standard error:

If you use JMP from SAS, you do not need to bother, it will calculate the prediction limits for you even in the case of transformation and weighted regression. From the next version 18, it can even be shown on profilers.

However, it will not give you the control limits from the model and if you are doing weighted regression with random factors the variance will not be scaled back. For these reasons we have made a JMP data table with all the necessary formulas and a JMP workflow (requires JMP17), that will take the needed information from your model and enter it into the data table and make the needed graphs. If you join our coursers or a getting consultancy support from us, you will get these files. We do not just hand it out without ensuring that you are properly trained in using them right.

If you want separate limits from each level of a predictor in a model, just enter the predictor as a systematic factor and PF will have separate levels. If you want joined limits, just enter the predictor as a random factor and PF will have common levels. The width of the limits will be the same. If you need different widths due to variance heterogeneity, you can do weighted regression.

Figure 10 shows the control and prediction limits for a model with with Batch and Sampling Point[Batch] as random factors. These limits can be used for future batches.

If you want separate limits for each batch, which is relevant for batch release, just change Batch to be a systematic factor. Figure 11 shows the result of this.

If you want to use the limits for future batches and seeing them during batch execution, we recommend fixing them based on the historical data. Figure 12 shows an example of this for batch 10, where limits are calculated based on batch 1-9. Even though we use historical data to set limits, prediction limits get narrower over time, as the standard error of the mean drops.

The green centerline is the accumulated mean of the sampling points.

The Standard Error on the Accumulated Mean AM after n sampling points can be calculated from the variance components on sample points and residual (within sampling point) on Batch 1-9 and thereby control and prediction limits for batch 10 are calculated as: