The Flip Side of Safety Barriers

The addition of barriers still is one of the most common ways to improve the safety of a system. However, if not adequately and carefully implemented, they may represent an additional source of hazards and generate catastrophic consequences.

“Safety barrier” — or simply “barrier” — has been a widely used term in aviation and other high-risk industries for many years. It became more popular in the 1990s, when psychologist James Reason proposed the Swiss Cheese Model of accident causation, and it is still widely used today as a way to refer to the layers of protection that are added to a system with the goal of fixing a safety issue.?

According to Reason’s model, mishaps are explained as a consequence of the coexistence of active failures (errors and violations of frontline staff — e.g., pilots, air traffic controllers, mechanics, etc.) and latent conditions. Under ideal circumstances, a system contains multiple barriers that prevent one or more active failures from resulting in an accident. Nevertheless, since barriers are a product of human ingenuity, most — if not all — of them contain deficiencies, which Reason illustrates as holes, making them end up resembling slices of Swiss cheese, as illustrated in figure 1. He calls the holes “latent conditions,” and cites as examples poor equipment design, deficient procedures, and inadequate training. The model preaches that, if the holes are allowed to line up in a specific way (i.e., if just the right — or “wrong” — conditions are met), they may create what Reason calls the “accident trajectory,” thus giving way to a mishap.?

Figure 1. The Swiss Cheese Model, adapted from the International Civil Aviation Organization (ICAO), 2018, Doc 9859 Safety Management Manual.

While barriers do, in many cases, increase the safety of systems, they carry a few fundamental problems that, if not acknowledged and properly addressed, may render barriers useless or even turn them into an additional source of hazard. Firstly, barriers may backfire when a system finds itself operating under conditions different from the ones under which the barriers were initially envisioned to function. Secondly, the addition of layers of protection generates an increase in complexity, thus increasing the chances of having potentially dangerous component interactions that are very difficult to foresee — and, for this reason, to prevent — at an early design stage. Lastly, barriers may create a state of risk homeostasis, meaning that they may make system operators feel confident and secure enough to adopt riskier behaviors, in this way bringing a system’s risk exposure back to its original — or an even higher — level.

When barriers backfire

In 2001, the world was shocked by the 9/11 terrorist attacks that took the lives of almost 3,000 people and injured more than 25,000. On that day, four airplanes that were bound to California were hijacked by Al-Qaeda terrorists, who made their way into the airplanes’ flight deck and flew them into the World Trade Center buildings in New York, the Pentagon in Washington, and into a field in Pennsylvania, following the attempt of some of the passengers to regain control of the aircraft by fighting the terrorists. The attacks changed aviation forever. Among the many security measures adopted by the industry to mitigate the risk of similar events was the bulletproofing of cockpit doors and the introduction of advanced locking mechanisms to prevent unauthorized, ill-intended individuals from accessing the flight deck.

14 years later, an Airbus A320 operated by Germanwings, a low-cost carrier owned by the German airline Lufthansa, crashed in the French Alps, killing all of its 150 occupants. The crash was caused by Andreas Lubitz, the co-pilot, who deliberately flew the airplane towards the ground. Andreas suffered from depression and had suicidal tendencies — aspects that had not been reported by him to the company nor were detected during the pilot’s annual medical checks. During the flight, while the Captain, Patrick Sondenheimer, was in the lavatory, Andreas locked the cockpit door, thus preventing Patrick from reentering. The Captain initially tried to knock and use the intercom to ask the co-pilot to open the door. However, as the criticality of the situation became more evident, Sondenheimer began to try to break in, only to realize moments later that the safety barriers installed years before to prevent cockpit break-ins — the bulletproof doors and the improved locking mechanisms — would not allow him to avert the disaster that followed.

The Germanwings accident demonstrates the importance of carefully assessing the impacts of barriers under?various conditions. Nevertheless, such a task bears its own difficulties, since attempting to predict the numerous possible accident scenarios that may result from the implementation of a barrier is an endeavor that requires the efforts of many people, who ideally should come from different professional backgrounds, and time, a resource that is usually scarce in accident prevention. This scarcity may be explained by the context under which most safety barriers are devised, which is in the aftermath of accidents, when the pressure to find causes and solutions is heavily present.

While it is understandable that such pressure puts accident investigators and other stakeholders involved in an accident investigation in a rather difficult position, an effort to not succumb to the multiple demands that emerge following a mishap has to be made. Otherwise, this may result in barriers being implemented haphazardly, thus representing what author Nancy Leveson, Professor of Aeronautics and Astronautics at Massachusetts Institute of Technology, calls a “whack-a-mole” approach to managing safety. Whack-a-mole is a popular arcade game in which the player uses a hammer to hit plastic moles as they come out of their holes in a random fashion. If barriers are not well thought through, they become a way of “fixing” issues as they come up. This approach may not only give way to unintended consequences as seen in the Germanwings example, but may also increase system complexity, which brings about its own set of problems.

When barriers increase complexity

In industries such as aviation and nuclear power generation, it is extremely common to see old designs being adapted and modernized in order to meet demands (regulatory, societal, safety, etc.) that change with time. Since nuclear power plants and aircraft are engineered to last many decades in the way they were originally designed, improvements in safety that may become necessary as time goes by have to be achieved through the addition of layers of protection as opposed to a complete system re-design.?

In 2010, Airbus introduced the A320neo, featuring engines that were considerably more fuel efficient than those of its predecessors. In response to the new model, Boeing, Airbus’ fiercest competitor, had initially planned to come up with a clean-sheet design, but later abandoned it and decided instead to modify the Boeing 737-800 to accommodate a pair of newer, bigger, and more-efficient engines, in this way giving birth to the 737 MAX. Since the 737’s original design dated back to the 1960s, when most airports lacked passenger boarding bridges and conveyor belts, its landing gear legs were shorter than those of today’s airliners, and fitting the larger, more modern turbofan engines required Boeing to make some adaptations to the airplane’s design. These generated some undesirable — and potentially dangerous — handling problems that the original equipment manufacturer (OEM) mitigated by adding a feature called the Maneuvering Characteristics Augmentation System (MCAS). The system consisted of an additional layer of protection to prevent the aircraft from entering a catastrophic stall, and it interacted with some of the components that were already part of the 737’s original design as, for example, the angle of attack indicator.

Some time after the MAX was put into service by many airlines around the world, two accidents that occurred in Indonesia and in Ethiopia only a few months apart from each other revealed significant problems with the MCAS that were pointed out as significant contributing factors to the crashes. In both events, a failure of one of the the angle of attack (AoA) indicators had led the MCAS to function based on incorrect values, thus resulting in the system commanding the two aircraft to pitch down in response to high angle of attack indications that were not true. Erroneous AoA indications are not an uncommon problem in aircraft. In fact, most of them — the 737 included — have two (or even three) AoA indicators, which usually gives pilots or the onboard systems a chance to crosscheck information. In the MAX’s case, however, the MCAS extracted information from only one of the aircraft’s AoA indicators, meaning that a false AoA indication could not be crosschecked in any way, and that the system would command the aircraft according to whatever AoA value it received, regardless of its validity. This dangerous interaction had not been accounted for by the Boeing engineers and was not detected by the Federal Aviation Administration (FAA) during the model’s certification process.

The MCAS’ logic alone made sense from a safety standpoint, as it had an important role in preventing a stall. Nonetheless, it brought problems that stemmed from the way it was integrated in the aircraft. Due to the fact that the system consisted of an accretion (an added safety layer) rather than a re-design, the result was an increase in complexity that allowed for a dangerous interaction to occur when a component — in this case, the AoA indicator — failed to perform as designed. The two MAX accidents attributed to the MCAS’ logic illustrate the Normal Accident Theory (NAT) proposed by sociologist Charles Perrow in the 1980s. The theory not only alludes to mishaps that, given the characteristics of a system, result from multiple and unexpected interactions of failures, but also to the fact that, in complex systems, catastrophes are often the consequence of “normal” failures (i.e., ones that are known to happen commonly) that trigger hazardous interactions. In the MAX case, the AoA indicator problem was the normal failure — as this type of mechanical issue is common in aviation — and the MCAS was the piece of the puzzle responsible for turning what would have been a rather prosaic failure into what Perrow would call a “normal accident.”

When barriers increase risk taking

When car anti-lock braking systems (ABS) and airbags started to become ubiquitous in automobiles, they were thought to have the potential to drastically reduce the number of road accidents and fatalities. However, many studies throughout the years have demonstrated that such devices have actually increased the likelihood of accidents and injuries as they made people feel safer and more confident to drive at faster speeds. Such behavior illustrates the concept of risk homeostasis, proposed in 1982 by Gerald J. S. Wilde, a professor at Queen's University in Canada. Charles Perrow, being himself a supporter of this concept, stated that “[f]ixes, including safety devices, sometimes create new accidents and quite often merely allow those in charge to run the system faster, or in worse weather, or with bigger explosives.”

In aviation, certain technologies and rules that had initially been envisioned as effective ways of preventing accidents have had varying performance in different sectors of the industry as time has gone by. While Ground Proximity Warning Systems (GPWS) have demonstrated to be an important tool in averting Controlled Flight Into Terrain (CFIT) accidents, many operators have been found using GPWS — successfully or not — to guide their way through mountainous terrain under marginal visibility conditions. Similarly, the rule adopted by many countries that imposed commercial pilots to have some hours of simulated Instrument Flight Rules (IFR) flying during their training in an attempt to prevent accidents associated with inadvertent entry into Instrument Meteorological Conditions (IMC) has led some inexperienced, non-IFR-rated commercial pilots to feel confident enough to get themselves into IMC, lose control, and become involved in a fatal accident.

In summary

Barriers have been paramount in creating the incredibly safe systems that support society nowadays, aviation being a particularly remarkable example. However, as time goes by and as technologies and systems become increasingly complex, barriers, if not carefully devised, may result not only in safety issues not being fixed, but also in them being transported to different parts of a system, thus giving way to potentially hazardous emergent properties not previously anticipated.

To prevent barriers from backfiring or, at least, to reduce the chances of that happening, they have to undergo a thorough analysis that involves people from different professional backgrounds who can look at barriers from different standpoints. The more varied they are, the easier it is to find from where dangerous interactions may emerge. Additionally, and equally as important, analysts have to be given sufficient time, since the number of possible interactions in modern complex systems may easily reach the thousands.

Although not always available, an alternative to the accretion of layers of protection into a system and the ensuing increase in complexity is to embed safety into a system’s engineering process starting from its first conceptual stages. According to Nancy Leveson, this increases cost-effectiveness and facilitates the implementation of safety features, since design modifications that are prompted by safety concerns at an advanced developmental stage are often expensive and also result in design trade-offs.

Lastly, barriers have to be monitored from up close. If safety fixes and devices are not coupled with increased monitoring and enforcement of standard operating procedures, operators will very likely come up with means to use safety barriers as a way to increase system output or to more easily accomplish certain tasks. If a risk homeostasis condition is allowed to develop, a system’s risk exposure may revert to its original value or even surpass it, thus completely defeating the purpose of safety barriers.