The Automation Conundrum

The concept of 'Automation Conundrum' stems from Mica R. Endsley's - "From Here to Autonomy: Lessons Learned from Human–Automation Research". Endsley says, 'The more automation is added to a system and the more reliable and robust that automation is, the less likely that human operators overseeing the automation will be aware of critical information and able to take over manual control when needed'.

The statement highlights a common problem that arises with the increasing use of automation in complex systems such as industrial plants, transportation networks, and other critical infrastructure. Particular examples include -

• Aviation - Autonomous air vehicles used to deliver cargo surveil remote locations, take over control from pilots if an imminent collision is detected, and act as teammates with manned aircraft.
• Autonomous Cars - Automobiles can automatically park, maintain lane discipline, and control speed in conformance with traffic.
• Cyberattacks - Among many others, AI systems can detect and prevent malware, identify fraudulent payments and respond to threats in real-time, by automatically blocking suspicious IP addresses thus preventing several cyber-attacks.
• Autonomous robotic vehicles are being developed to deliver materials to remote locations, repair runways, and retrieve fallen soldiers from the battlefield.

As automation becomes more prevalent and advanced, the role of human operators in overseeing and controlling the system has begun to diminish. This is also known as Out-of-the-loop (OOTL)* performance. As Endsley described, 'People are both slow to detect that a problem with automation exists and slow to arrive at a sufficient understanding of the problem to intervene effectively'. One reason is that automation can operate at a higher speed and accuracy than humans, performing tasks continuously without getting tired and there is minimal chance of error.

This problem is exacerbated when automation is highly reliable and robust, as it can create a false sense of security that everything is functioning correctly. The automation system may lead human operators to assume that everything is being taken care of, causing them to potentially not pay enough attention to the system.

The relationship between factors creating the automation conundrum is depicted in the Human–Autonomy System Oversight (HASO)* model. The HASO model is a framework for understanding the factors that influence human performance in complex systems.* (Corrected from the previous version of the article)

The risk with this approach lies in the chance that operators may become overly reliant on the automation, and lose their ability to monitor the system and detect critical issues that the automation may not be able to handle. A popular example of automation failing due to less control is the case of the 2018 self-driving car accident in Tempe, Arizona. The self-driving car, which was in autonomous mode, struck and killed a pedestrian who was walking outside of a crosswalk. Investigations revealed that the car's sensors had detected the pedestrian but failed to properly identify her as a pedestrian or predict her path due to a software flaw. The accident highlighted the risks associated with autonomous vehicles and the potential consequences of relying too heavily on automation without adequate human oversight and control.

If the automation is highly reliable and robust, operators may be lulled into a false sense of security and assume that the system will always function perfectly. This can become a serious problem if the automation encounters a situation it cannot resolve. In scenarios like these, the issue will not be detected until it is too late, and operators may not be able to take over manual control of the system.

To mitigate this risk, it is important to ensure that operators receive adequate training and are able to monitor the system effectively, even when automation is performing most of the tasks. It is important to have robust systems and a feedback loop in place for detecting and reporting issues, and for providing operators with clear instructions for taking over manual control when necessary. Finding the right balance between automation and human oversight is a key challenge in designing safe and reliable complex systems

*The next article, will dive deep into technical concepts of OOTL, the HASO model, and situational awareness (SA), concepts that provide the foundation to devise methods to support human–autonomy teams.