Our Favorite Deviation: Standard

When I was a kid, “standard deviation” was an often-quoted concept at our table. I had only a vague idea what it meant, but I did get that it was a great ploy to denigrate teenagers and journalists who thought they had discovered “truth” or “news” in statistical data they had let their laser-like attention focus on for about 30 seconds. So I tried to stay away from it.

Fast forward to the early 2000s when we had to combine a bunch of second-hand equipment, a motley crew of veteran glass specialists, and hundreds of local newbie techies to produce “acceptable” bottles.

Our customers were as fickle as our equipment and our crew, so it was hit-and-miss most of the time, all the more so since our product had to be up to food safety standards, independently from what our customers thought of it. It was chaos, with us wading through broken bottles up to our knees.

Then we got lucky. We figured out how to connect our 1980s online inspection equipment to a French SPC program, Vertech' s SiL, that turned our mountains of shards into readable diagrams.?It did not fix our production process directly, but it helped us find a way out of the mess we’d made.

Instead of having to listen to daily reports by those managing the situation (always a bad idea), we could see, within 30 seconds after coming into the office, what had worked overnight, and what hadn’t.

Fast forward to the 2020s. We’re now creating online quoting software that has to emulate our users’ quoting processes. Our basic dataset to set up algorithms consists of 10 CAD models, combined with 10 prices if one piece is ordered, and 10 prices if 10 pieces are ordered, all provided by our customer. Our machine learning tool then figures out a workable fit and we tweak it further.

One of the problems we’re facing now is a runaway marketing statement that claims we’re “99% accurate”. I’m not sure what the author meant by that, but customers have interpreted it as meaning that our quotes are never more than 1% off from what they would have quoted. Spoiler: That is not the case.?

In our quest to find an elevator pitch-fitting phrase that does catch people’s attention but is not misleading (let alone untrue), we are digging into our data. As the following case will illustrate, just loosening the tolerance in the original statement (say +/-20%) is not enough.

We landed an important North American CNC account at the beginning of this year. They required that our quotes be within +/-20% of what they would quote themselves. It so happened that they declined our standard data format and provided us with hundreds of real quotes and models instead, to set up their 3-axis mill.

After each setup, they would audit our automated outcomes, again using real quotes for the same models as a reference. Here is where we got lucky: They shared both their audit analyses and their raw audit data. What we found was that in different audits, they had used different quotes to check against. The models, specs, and quantities were identical, but the sums quoted were different.?

We had asked to hit a moving target. Their quoting consistency over time looked like this:

The diagram shows that over time, the customer is only just hitting the “accuracy” target of +/- 20% they had set for us. This means that, for any given data point provided by them, it is not unthinkable that it is “too high”, according to their own standards, by 20%. Now if we were to overquote by 20% and during the next audit, they would compare our outcome with another of his quotes that was 20% “too low”, our quote would be 80% higher than that underquote.

That’s what happened. Ky Olsen managed to get this at the last audit:

Now a diagram like this invites action, starting with the outliers on the right. As it turns out, one of the two parts represented there has this cross-section:

It’s clear why our system quoted much higher than the human quoting specialist: The specialist recognized the cross-section as a 6 x 2.5” channel, and knew that the only work to be done was to cut it to size and drill a few holes in it. The system thought the whole middle part had to be milled out.

A problem like this is easy to fix: Set up a separate tech, provide it with standard profiles, call it "Beam Cutting" and apply separate rules. The end user can then select that tech and get much lower prices. ?Once we can ignore channels and other profiles for 3-axis milling, all our remaining quotes become much more accurate.?

Looking a bit further down the line, what we are certainly going to do is:?

Keep track of manual edits by our users in automatically generated quotes, show the differences in diagrams like the above, and, every 10? 100? quotes,? ask questions like:?

“Did you have a look at the outliers??Maybe we can group them and split them off?

“Looking back, who was more correct, you or the system?”

“Would you like to change the current settings so the automated system becomes more accurate and you have to check less often?

If the user wants to change a parameter, we can immediately show how the bell shape changes for the earlier automated quotes.? All he will have to watch is whether the +/- 20% axes move wider apart (improvement) or closer together (deterioration) and the zero stays in the middle.?

And our marketing slogan? Our marketing slogan will be: “Working with you, and reviewing both your processes and your technologies, we will attain a standard deviation of 6.66%!” Catchy, no?