How Do You Know If It Is Working? Measuring the Accuracy of AI (4/5)

As I mentioned in the first of this article series (link), accuracy is a measurement of both effectiveness and efficiency. Effectiveness measures performance of the task – for example, finding drug traffickers. Efficiency measures the amount of work needed for a level of performance – how hard one works to find the next trafficker.???

Regulators usually demand effectiveness, and workers and corporate leadership usually seek ever-increasing efficiency. People on the front lines usually talk in terms of the number of “false positives” they must deal with each day.?

Think of “efficiency” as the number of accurate predictions above the threshold as a percentage of all entities above the threshold. Of everything predicted chihuahua, what percent were actually chihuahuas (true positives)?

Efficiency =? True Positives

___________

All Positives (True Positives + False Positives)?

In an average bank compliance department, one might expect to see efficiency of about 5%. This means that in the financial crimes space, about 5% of the cases predicted positive are true positives, and 95% are false positives:? five chihuahuas found, or every 100 cases reviewed. This is a difficult job that ML will improve, for sure.?

Think of “effectiveness” as the number of accurate predictions as a percentage of all possible cases of interest. Of all of the chihuahuas, how many did the algorithm move to the right of the threshold? 20 chihuahuas found out of 25 is 80% effective.?

Effectiveness = True Positives

___________

True Positives + False Negatives

Two factors impact efficiency and effectiveness, one controlled by the operators and the other by the model trainers:

First, where to draw the threshold for a given algorithm, set by the operators, can increase effectiveness by decreasing efficiency, or increase efficiency but decrease effectiveness. This decision must be human and based upon risk appetites and profiles of the institution’s leadership.?

Second, improving the algorithm will move more chihuahuas to the right and more not-chihuahuas to the left. More better training data will help the modelers do this. With better separation like this, both efficiency and effectiveness improve, making work life better for the operator and buying down risk at a lower cost for the institution.?

Efficiency and effectiveness can be computed with some counting, addition, and division; it’s not a difficult topic. Anyone can speak with confidence on AI/ML performance, even without knowing the details of deep learning and neural networks.?

But one challenge remains to be understood:? how to think about bias, the subject of my next article in the series.

This is the fourth article of a 5-part series. See my video short on this subject or read Article 3 here or Article 5 here.