Failing Gracefully

Failure is part of any learning system whether human or machine. When researchers tout the ability of their cars to learn "on-the-fly," I think about the other side of that double-edged sword. If the cars can adapt, they can also make mistakes. It is impossible to have one without the other. Autonomous cars today are neither perfect nor reliably imperfect -- the kinds of failures and their consequences remain hard to predict. This means that it is difficult to limit the impact of self-driving failures. Understandably, bureaucratic systems don't like this situation.

A recent study by Cox Automotive shows that as people learn more about the reality of autonomy or experience it directly, their level of trust goes down. The number of people who believe that they would be safer in an autonomous car dropped recently to 45% down 18 percent since the survey of 1,250 people in 2016. I have argued that the field of robotics might be further along if we as roboticists had worked to encourage appropriate distrust. The question is not whether we can make perfect robots, but whether the system as a whole can adapt to individual failures gracefully.

Hava Siegelmann (@havasiegelmann), a DARPA Program Manager, points out that very few companies are working together to address safety issues: "The next program that I've been working on is about the safety of AI. [...] the companies that I've contacted that are working on drones and self-driving cars said they are actually not interested to work together on safety of A.I." Still, Hava insists that a common approach to safety is the key to the advancement of AI.

In my experience, the ancien regime of AI researchers is dedicated to solving navigation and safety at an individual vehicle level. Working collaboratively to solve safety issues would require the companies to each recognize their current failures and then share the solution. Sounds unlikely, right? There are some exceptions. Ralf Speth (@ralphspeth), CEO of Jaguar Landrover make the statement at Beijing's auto-show that to make a safe autonomous car we need to focus on the environment and work together. He went on to say that no one car company can go it alone. I could not agree more. The system should degrade gracefully even if one vehicle is having a bad day.

The Federal Highway Administration also agrees there is a need a common safety framework. We have long known that a single car can't see around corners or anticipate events many cars up on the road. Passing messages up and down the road in real-time has long been in the plans, but it appears that this may not be enough. The problem is that each individual could be wrong about its position or its state. In that case, sharing the information could hurt rather than help. Each car should determine its own position but it also needs an external component that can step in if the individual system fails.

Before the military could embrace any kind of autonomy, the Army Testing and Evaluation Command (ATEC) insisted that we put standardized E-Stops on each vehicle to ensure that independent of our AI system, the vehicle would shut down if someone hit the big red button, regardless of what the AI system had to say about it. There were many times when a human couldn’t get to the button, including one memorable instance where, after a mistaken code download, I ended up sprinting through the woods to dive, arms outstretched for a robot that was careening towards property or people while out of control.

What if in addition to E-stops, we also had “P-Stops” -- external safety regulators based on positioning. I envision a 5G-enabled micro-positioning module that shuts down the vehicle if it moves even one centimeter beyond an authorized boundary or fails to emit the proper messages.

Of course, this requires adoption. I am off to Greenville this week to meet with a host of companies who have agreed to use 5G enabled micro-positioning. Some companies may fight it, but most will realize that without a way to mitigate the risk, full authorization from the government will be a very long time coming. Autonomous systems will always fail but if the roadway can respond appropriately these failures will be minor slow-downs, rather than catastrophic events. Instead of pretending perfection, let's agree to graceful degradation.

#AI #Robotics, #safety, #P-Stop, #DARPA, #self-driving, #ArtificialIntelligence, #DOT, #FHWA, #autonomy, #HavaSiegelmann, #RalfSpeth, #jaguar, #landrover, #waymo