Simple, Elegant, Convincing, and Wrong: The fallacy of ‘Explainable AI’ and how to fix it, part 5
This week we have a series of blog posts written by our CEO&Founder Luuk van Dijk on the practicalities and impracticalities of certifying “AI”, running up to the?AUVSI panel discussion on April 25.
5. The tragic case that was entirely explainable
______________________________________
5. The tragic case that was entirely explainable
In the previous posts, we went into some detail on how a machine-learned system can never be perfect and how you can’t really fix that. That doesn't sound like a good idea when dealing with systems that guard life and can wreak havoc when they fail. Why bother?
The answer is that somebody has to solve these problems, and currently, it is typically a human operator who is not exactly perfect either. If we want to make progress towards systems that are safer, faster, better, cheaper, and more reliable, we have to deal with the actual tasks that are currently done by people. Whenever it comes to managing risk by keeping the operating parameters of a system, say an aircraft or a car, within safety bounds, systems will have to deal with uncertainty in the environment. The cases where you can engineer all risk away upfront by putting in safeguard upon safeguard and restricting operating conditions cover only a tiny fraction of the set of all problems worth solving. In all the other ones, you will have to make adjustments to the system dynamically, as the situation changes under external influences, subject to imperfect information. The good news is that there’s not really a ceiling on how good we can make these systems, but we have some way to go to get there.
On the evening of March 18, 2018, a vehicle struck and fatally injured 49-year-old Elaine Herzberg crossing N. Mill Avenue, outside a crosswalk, in Tempe, Arizona. This would have been ‘just’ one of 36560 traffic deaths in the USA in 2018 if not for the remarkable fact that it was a test car operated by Uber for their self-driving system. As the system was far from production-ready, it was easy to blame and convict the safety driver who wasn’t paying attention at the time, but the NTSB report goes into quite some detail on how the system worked.?
As it was, the system had multiple fundamental design flaws: to suppress spurious alerts, the engineers had put in logic to wait a bit to see if a detection persisted, which cost precious reaction time. If the system would hesitate between a plastic bag (‘other’) and a pedestrian, it would reset its prediction logic, and for a plastic bag, it would not even try to predict a trajectory.?
The report explains the system design flaws entirely at the classical software level. At no point does the question “why did the neural network not recognize the pedestrian?” arise. We know upfront that false-negative recognition has a non-zero probability, and given the possible consequences, the rest of the system should have been engineered from the ground up to deal with this uncertainty.?
If the safety driver had been paying attention, or if there had not even been a self-driving prototype system on board, the statement “I thought she was a plastic bag, and I did not expect pedestrians on this section of the road” would have been considered a perfectly valid explanation, sadly without a better outcome for the victim.
The report could come to a detailed explanation of what had happened because the system, fortunately, recorded enough data to reconstruct in great detail what had happened. Had the system had any output to the safety driver to say ‘I see an obstacle here, but I’m not sure if it is a pedestrian, so I’m going to hit the brakes now,’ then the driver would have had an explanation even if the system was wrong in its judgment. These forms of ‘explainability’ are very useful to find flaws when the system as a whole is malfunctioning or to gain trust and acceptance when it works as intended, but this is not an ‘explainability’ of the machine learning component.
As a robotics problem, flying is a lot simpler than driving, even though the stakes are typically higher. Where in driving you have to understand the difference between a plastic bag and a pedestrian, and a dog, and a traffic light, and a bike that’s standing there, and one that is being ridden, in flying you can go by the simple maxim: if you can see it, don’t fly into it – unless you are really really sure you want to land on it. But in both driving and flying, it is a misconception to think that the role of the human is merely to operate the controls. Pushing the buttons and handling the steering wheel or yoke to follow a trajectory can be automated easily, but the real and hard-to-replace task of the human in the role of driver or pilot is maintaining a safe state.?
To do this, he or she must have an adequate picture of the current situation to predict the near future accurately enough to steer the vehicle away from danger and towards safety, preferably according to the travel plans. If you ask the driver or the pilot why she took one action vs. another, she’ll probably be able to come up with a plausible explanation, but we also know that humans are terrible at explaining their own behavior, and we almost never can objectively verify that any given explanation is actually true or valid. This hardly ever has any bearing on the ability of the human to deal with risk during flight or drive effectively.?
Risk is inevitably expressed in terms of probabilities, and therefore if we want to build any but the most trivial risk-managing systems, we will have to learn how to use machine learning. A perceived lack of explainability need not stand in the way, but that’s not to say there aren’t any challenges. So one more post tomorrow.
Safe AI for Automated Vehicles
2 年Great, Luuk van Dijk, drive this topic to push industry