Understanding Intelligence: Humans vs Machines - Part 3

That is the 3rd in a short series of articles about different types of human decision making processes and their computer equivalents, inspired by my reading of 'Thinking, Fast and Slow' by Daniel Kahneman. Part 1 discussed different types of human decision making processes - System 1 is always-on, experience and memory-based and instinctive, while System 2 is requires concentration and calculation. Part 2 compared System 2 to traditional procedural programming, and System 1 to so-called machine learning. In this article I will discuss some of the problems faced by System 1 and the similar problems faced by Machine Learning.

A Machine for Jumping to Conclusions

In his 'Thinking, Fast and Slow' Daniel Kahneman describes System 1 as a 'machine for jumping to conclusions'. System 1 is seemingly purpose-designed by evolution to make decisions very rapidly, comparing only a partial set of evidence to prior experience. It is understandable in evolutionary terms that this would be so. It is better to be safe than sorry and to over-react in circumstances where a false negative will have much worse outcomes than a false positive - when identifying potential threats or the presence of predators, for example.

The problem is that System 1 doesn't stop to consider the available evidence, and whether or not it is reliable, or whether there might exist other evidence that would perhaps alter the resultant decisions. System 1 jumps to a conclusion immediately, right or wrong.

The models created by machine learning techniques have a similar problem. They have (generally) been created by feeding historical 'training' data to an algorithm designed to find patterns in the data. The patterns found are mapped by the algorithm to create 'models' of the world, in which certain patterns observed in data will drive it to familiar conclusions.

While a data scientist can check the outputs, source more data or tweak algorithm parameters to influence the model, the final model itself is 'dumb' and cannot learn or seek alternative data. This means that they might well ignore significant information and 'jump' to the conclusion that their training has pre-determined is appropriate. Notions like self-doubt are generally absent.

Characteristics of System 1 and AI

Kahneman lists a number of prominent characteristics of System 1, some of which are listed below:

• operates automatically and quickly, with little or no effort and no sense of voluntary control
• executes skilled responses and generates skilled intuitions, after adequate training
• creates a coherent pattern of associated ideas in associative memory
• links a sense of cognitive ease to illusions of truth, pleasant feelings and reduced vigilance
• is biased to believe and confirm
• focuses on existing evidence and ignores absent evidence ('what you see is all there is')
• sometimes substitutes an easier question for difficult one (heuristics)
• overweights low probabilities
• frames decision problems narrowly, in isolation from each other

These characteristics of the human mind are things that many data scientists will immediately recognise as equally characteristics of AI. A model that produces good outputs after adequate training based on the notion of finding patterns and associations is, after all, what Machine Learning is largely about.

Worrying Aspects

Some aspects are a concern. The idea that AI might be prone to providing 'illusions of truth' should worry many who are planning to rely on AI to solve serious problems. Similarly the likelihood of in-built bias to 'believe and confirm' should warn us to be doubly sceptical of results and to seek additional confirmatory evidence.

There are famous examples in AI of the model answering a different question to the one being asked. Marvin Minsky tells the story of a model intended to identify composers when shown sheet music. However the pages used in training the model had additional hand-written notes on the pages, and the model learned to identify the composers by these rather than the main musical score. Other (possibly apocryphal) examples include AI that learned to identify tanks based on the time of day that the aerial photo was taken, or that identified dogs and wolves by the surrounding landscape (snow/sun, woodland/urban).

A more worrying aspect is that in the real world it is very difficult to know when a model has given an incorrect answer. A model that is 99.9% accurate is pretty good, but when applied to a population of millions will still produce thousands of wrong answers. How significant this is depends on the types of decisions being made on the back of the model. If you are identifying likely purchasers of a new brand of aftershave, it is one thing. If you are trying to diagnose a terminal illness then it is another.

In illness diagnosis, assuming that early intervention would save lives, a false negative is potentially fatal. Equally, assuming that treatment is invasive and has severe implications, a false positive is no picnic either. In circumstances like this is might be preferable to make the assessment manually and to use an AI model in parallel to highlight results that need additional checking. That is to say, when the medic and the model agree, the diagnosis is probably good, but where the two disagree, perhaps a second (third?) opinion is advisable.

Conclusion

Many data scientists work hard to compensate for and minimise inherent weaknesses in their models, with varying degrees of success. However their masters in business, having drunk the Kool-Aid of analytics, may not be sceptical enough of the accuracy of AI's conclusions, and can pressure modellers to go live with an immature product. This is not to say that we should not trust the outputs of Machine Learning at all, just that we should not trust those outputs with the same confidence as we might treat the more determinative outputs of traditional procedural programming. Where there is no procedural equivalent, then Machine Learning, while not perfect, may well be the best alternative that we have.

Questions on Data Warehousing, Data Integration, Data Quality, Business Intelligence, Data Management or Data Governance? Click Here to begin a conversation.

John Thompson is a Managing Partner with Client Solutions Data Insights Division. His primary focus for the past 15 years has been the effective design, management and optimal utilisation of large analytic data systems.