Metrology Monday! #54 99% vs 95% level of confidence on specifications

Last week we talked about the difference between artifacts and instruments and provided some information about why Fluke specifies instruments at the 99% level of confidence.? Fellow Metrologist and friend Paul Reese noted in the comments the importance of understanding the difference between a single test point and the entire instrument.? As David Deaver stated in his paper “Having Confidence in Specifications” (a great paper to read if you have not yet), in order for an instrument overall to be at a 99% level of confidence, each test point must be much, much higher than 99%.? My favorite line from that paper is “Specifying a complex instrument at the true 95% confidence level for each point would be a manufacturing disaster.” His paper goes on to explain that the 5520A was tested at 552 test points, and if ever 100 of the test points were truly independent, the probability of all test points being in tolerance to a 95% level of confidence would be 0.95^100 = a 0.6% chance of being able to ship a product.

An example of how Fluke addresses this is when we manufacture and calibrate the 5730A.? We maintain each test point at about 4.5 sigma, or 99.9966% level of confidence in order to meet the 99% level of confidence for the instrument overall.? This is not to say that customer can use 4.5 sigma when considering uncertainty for each test point because the drift over time and environmental variation also need to be considered when using the product.

Here is a picture that illustrates the difference between a product that was specified at a 99% level of confidence vs a 99% level of confidence.

You can see that the product specified at 95% level of confidence does not perform nearly as close to nominal as the product specified to 99%.? Quite simply, 99% level of confidence show that there is only 1% probability of being outside of the specification limits as opposed to 5% with the 95% confidence limits.? This can be taken to an extreme where I have seen test equipment manufacturers specify their product at the 1 sigma level of confidence, which means that there is only a 68.27% probability that product will perform to specifications.

We have had customers try to require Fluke to ship them products at the 95% level of confidence, because the specification for 95% is understandably smaller than the 99% level of confidence.? For example, If the 99% level of confidence was 10 ppm, the 95% level of confidence would be 7.75 ppm.? Then they wanted us to guarantee that all products shipped would meet this specification.? This would simply mean that they are asking us to change the specification to 7.75 ppm at a 99% level of confidence, and the product was never designed to do this.

A lot of confusion comes in when people use these instruments to evaluate the uncertainty of a calibration.? For example, when we use a calibrator to calibrate a digital multimeter.? A lot of people think that their calibrator needs to be maintained at 95% level of confidence specifications.? This is not true, because the GUM Method requires us to convert our type B estimates of uncertainty to a 1 sigma level of confidence (see post #47).? To use a product that is kept at the 99% level of confidence, simply divide by the coverage factor of 2.58 to get to the one-sigma level of confidence for the standard uncertainty.? It is perfectly acceptable to use an expanded uncertainty of 2 for the final estimate of uncertainty because this is standard practice (we cover expanded uncertainty soon).?

I have also had people ask if you have to convert a calibrator from the 99% level of confidence to a 95% level of confidence to evaluate Test Uncertainty Ratios.? This is also a future topic, but is it is important to note that a Test Uncertainty Ratio of a device under test at its end of period reliability level is divided by the expanded uncertainty of the measurement, expressed at a 2-sigma level.? If the calibrator is the standard used, it is effectively already being converted to a 95% level of confidence in a properly evaluated estimate of uncertainty.

I hope that this has not only helped improve our collective knowledge, but it will make it easier to compare and contrast products when evaluating performance specifications.