Intelligible AI - The Final Frontier?
Flashback to my master's thesis in 2004. I was sitting in front of a massive auto-generated topic map allegedly representing all the most important topic clusters and their relationships for the topic it was trained on. Automatically building this type of massive knowledge system that was summarizing the contents of hundreds of books and scientific articles was an incredible experience, but it mercilessly revealed what would prove to be the key challenge in AI: intelligibility of results. How could we be sure that our model actually selected the most important topics and topic correlations, without mixing in random ones?
Why Intelligibility Is Critical
The intelligibility problem became clear during my next project, this time within a professional setting and for a paying client. In 2006 we created a system that was able to categorize scientific articles and news content based on each individual users' preferences. Each time the user ignored an article, the system would lower the importance of the key concepts within this article, but only for this specific user. Getting to the 80% mark in terms of result accuracy was rewarding, to say the least. But then, closing these final 20ish percent was not doable for us at the time. Why? Because we just could not explain why certain garbage articles would continuously appear and why, under certain conditions and for certain users, other articles would be discarded. In 2006, I was not aware of it, but we were experiencing the intelligibility problem of AI (LSA at the time). The trained model was making its decision on term weights that lived within unimaginably vast matrices that were not even remotely comprehensible to the human brain. Pointing to a certain spot and finding a problem was impossible. If you cannot understand it, how can you fix it?
Intelligibility Is the Next Frontier in AI
Without labelling intelligibility as the final frontier, we can confidently assume that it is the next frontier we need to overcome to get to the next level of AI. Without intelligibility we can never fully rely on AI-systems. This often leads to the escalation of human monitoring requirements to continuously test AI models and the validity of their predictions. Without this continuous level of babying models we always run the risk of critical failure and there are many examples to prove it. This chart (I plotted it yesterday to show the quality of my trained classification model) offers a very basic idea of why this problem is so important, if we want to start using AI for mission critical tasks. You can see that our predictions (blue line) are nicely fitted through the middle of the actual test results (dots). However, a few dots are quite far off and you do not want to be that dot when the AI model overlooks your cancer or T-bones you on your way to work.
But How Can Waymo Cars Drive without Human Drivers at the Wheel?
So did Waymo crack the code of creating intelligible AI models, as they are having their cars drive around in Arizona, all without "safety humans" inside? I am not an insider, but based on what I can tell from publicly available sources, I do not think that they cracked that code. What they very successfully did is to create a large set of learning models, each with the task to classify specific aspects of the environment that is relevant for their cars and then tune the heck out of these models by feeding them with millions of hours of correlated streaming data from Lidars, cameras, radars, and probably a bunch of other sensors and external input data. When you then limit the area of operations of the car (Phoenix, I believe), connect all operating cars for additional contextual data input, and constantly monitor the fleet so that you can "tune out" undesired behavior, you can build a safe self-driving car, without being able to explain why each individual model works.
Is Intelligibility Possible?
For centuries, humans have used mathematical formulas to explain how the world (and to some degree the universe) works. Before, a lot of our believes was based on assumptions, such as "the earth is flat" or "the sun rotates around the earth." But then, based on a lot of data, some clever humans figured things out. The same should be possible in AI, but the question becomes if there are any shortcut that prevent us from having to do hundreds of years of research. There are plenty promising experimental technologies we might be able to employ to condense this process to only a few years, but this is a topic for my next post.