Ten times resolution - measurement capability to tolerance or variance

A rule of thumb for measurement system resolution is the variance in the thing you are measuring should be ten times (or more) of the resolution of the means of measurement. The reason for this is to understand not just the measurement to the feature but the variance from the feature to gage the capability of the process performance. While it is fine for a tradesman to measure a cut piece of lumber in framing a house to within 1/8” using his tape measure (or even another piece of wood) it is not good practice to measure automated output of a process the same way.

Individual measurements can be accomplished in individual ways. A carpenter will tell you that they can cut pieces of wood to exactly the right size by cutting them and feeling the length compared to a standard piece – the human fingers are sensitive, almost to the micron level. If a piece of wood is not exactly right, the carpenter fixes it or tosses it out. This is fine, it is a one-piece flow and the inspection is 100%. This carpenter watches the process and if too many pieces of wood are being chucked he starts cutting things just a hair longer so he can avoid scrap by intentionally reworking. Fine for the carpenter or any one-piece process where rework is acceptable. Not fine for mass production or processes where rework is not acceptable or one-piece flow is not possible.

Production measurements need to be more thoroughly understood. The variance from the target needs to be monitored in ways tighter than the tolerance. Here are three measurements using a ruler that only shows inch increments. From this you can tell if the measurement is essentially 3, and you can guess it might be 3.1, 2.8, 3.2. If what the process expected was 3” +/- 1” we can see these measurements meet that but we cannot tell the precision of the process.

Here are the same measurements with a higher resolution ruler. Now you can tell with certainty that the measurements are about 3.1, 2.8, and 3.2 – BUT you can also tell that they’re not exactly those values. With the increase in resolution you can tell the first one is about 3.08, the second is about 2.81, and the last is about 3.18. The increase in resolution is needed to see the actual measurement. This gives better insight into the precision of the process – but without consideration of the tolerances allowed we still can’t gage the effectiveness the process.

Production volumes require processes that have performance that can be measured in terms of a standard deviation, also called Sigma. A process that is considered acceptable by most manufacturing in the United States shows a performance of 4 sigma or better, which means if you took the tolerance band allowed for the part and divided it into plus or minus 4 evenly cut parts, each cut part would be more than one standard deviation.  Using the same example, the target we are trying to meet is 3”, and the acceptable tolerance is plus or minus 0.4”. In order to know if our process can meet or beat 4 sigma we need to know what the standard deviation is, so we need to be able to measure accurately to 0.1” since 0.1 is 0.4 divided by 4 (4 sigma in the tolerance). We just saw that if the increment of resolution is at the same level as the measuring device we know if we’re close, but we won’t know how close – we can only guess. To get a better estimate or guess we need a little better resolution – at least twice as much. So if we add a line between the existing lines increasing the resolution from 0.1 to 0.05, now we can guess to a higher degree of accuracy – we can tell if we are at 3.10, 3.05, 3.15, or we can guess in the range between to estimate 3.12, 3.07, 3.16. Just adding that one increment, dividing the scale in two, opens the ability to measure to a much higher degree.

This is how this translates to resolution of measurement. If the goal is to exceed 4 sigma that means the resolution should be at least 1/8th of the tolerance, or more commonly 1/10th, since that’s slightly better than 1/8th. So if you have a tolerance of +/-0.1”, the resolution of the measurement device should be 0.1” divided by 10, or 0.01”.

All of this comes into play for the engineer. The design engineer needs to understand both the manufacturing capability of the process and the effectiveness of the inspection method. If the engineer is considering applying a tolerance of +/-.0005” to a feature that engineer needs to understand how this will be measured. It is easy (and common) for engineers and machinists to point to the precision of the equipment as justification for a tolerance – Haas mills have a step motion precision of .0005” via their glass scale encoders. Does this mean they can consistently cut parts to an accuracy of +/-.0005”? No, it does not. The scale is only part of the equation – there is thermal offsets as the equipment and the part heat up, there is tool wear, clamping force of the material, bending of the material based on feeds and speeds, etc. The only true means of measurement is the actual means of measurement – the engineer and the machinist cannot rely on the machine to deliver at the resolution of the motion system in the equipment. Also keep in mind that a step resolution of .0005” means the real resolution is somewhere between .00035” and .00065” since that is the best estimation that can be met – refer to the previous example of measurement.

The Process engineer needs to understand how this affects the sample rate of a high volume process. If the resolution of the measurement is, say, 100x the tolerance then it is very easy to track performance capability because variation in the process is clear. The lower the resolution gets the more sampling is needed to monitor process capability which increases cost, decreases lead time and throughput, and increases risk of scrap or rework.

The Quality engineer needs to know how confident they can be in the measurement of the output. If quality intends to scrap product because it doesn’t meet tolerance then quality needs to have high confidence in their ability to measure. Going back to the beginning of this section, a one-piece flow process that gets inspected 100% using any number of sensitive methods – gages, fingers, vision systems, etc. – is fine of your process can accept the cost of 100% verification.

Regardless of your role in the manufacturing process understanding the tight relationship between tolerance on the drawing and capability of measurement is critical. Keep these things in mind when moving a design into manufacturing as well as when you are creating the parts. If it can’t be effectively measured – meaning a measurement system of 10X the tolerance – it can’t be effectively produced.

===

David West is a mechanical engineer with over twenty years in engineering management and building teams. He has consulted with, or worked for, companies in Production Manufacturing, Pharma, High Tech and Med Device.