Getting Misled By False Claims
Someone recently met me with a proposal saying they have an assessment that can predict socially deviant behavior with an accuracy of 90%. I have come across a lot of similar claims and it is easy to be carried away by such claims because most large organizations have the need for identifying such behaviors in potential employees. So, let’s look at this claim in some detail:
So, if the accuracy is 90% then the error rate is 10%. Let us assume that the meaning of this is that there is a 10% probability that someone who is socially normal is identified as socially deviant by the assessment. Seems an acceptable error rate.
Now let us dig a little deeper.
Let us assume that the prevalence of deviants in society is 1 in 10,000. This means that in a population of 10,000, the number of normal individuals is 9,999 and there is 1 deviant. So, if this assessment is applied to this population, the test will say that 1,000 people are deviants because of the error rate of 10%.
The reality is that there is only 1 deviant in this population. So, the error rate is not 10% but much higher. The error rate is (999/1,000)*100 which is 99.9!!!
Would you be okay with an error rate of 99.9%? Obviously not. So, what started off as an error rate of just 10% turned out to have a real error rate of 99.9%
The error rate of any test should be far lower than the prevalence of the defect that the test sets out to identify. If the defect rate is lower than the accuracy of the test, then the test is meaningless.
Next time you come across such seemingly impressive claims be skeptical and dig deeper!
Product Manager at RIL || IIM Mumbai
6 年This is explained in a great detail in the book "how to lie with statistics by darrell huff".?
Country HR Leader at Airtel Africa #Telecom #HealthTech #FinTech #EdTech #HR Leader
6 年Very true Hari!! If not interpreted & evaluated, it leads to disasters.. I personally keep taking assessments to know my pros & cons and offlate they show so many diff characters and at times i feel i have some sort of identity crisis with my own self.
Founder @ dSights | Applied Machine Learning| | Analytics
6 年Sir: Your example is of severe class imbalance in such a situation - Accuracy is not a good metric, it would good look at F1 Score , which almost similar to what you have calculated.