Learning from Aviation: Ways to Enhance Incident Response in Software Engineering

Looking back at our fireside roundtable chat after the first OOPS - Outage Operations and Incident Response event, our CEO and Co-Founder Hamed Silatani sat with Har-Inder Chandan , a pilot at Virgin Atlantic , and Ivan Merrill , a seasoned professional in monitoring and observability. They discussed the crucial aspects of safety culture in aviation and its parallels with incident response in software engineering.

Their conversation highlighted the critical role of training, preparedness, and organisational culture in managing incidents in both fields. Adopting these principles can help build resilient systems and create environments where teams are ready to handle emergencies efficiently.

We’ve highlighted some of the key takeaways below, but feel free to watch the whole video linked below.

Understanding Aviation Safety Culture

Har-Inder Chandan , with 40 years of flying experience, shared a profound insight: despite never encountering an emergency situation, he was always prepared. This, he noted, underscores the resilience built into aviation operations and the extensive training that pilots undergo. In aviation, pilots train every six months, rehearsing and refining responses to evolving threats. This rigorous training ensures that pilots are always ready to handle emergencies effectively.

The Role of Training

He emphasised that training is not just about repetition but about adapting to new threats. The data from daily operations inform training programs, ensuring that pilots are prepared for both known and emerging risks. This proactive approach helps in identifying and mitigating potential issues before they escalate.

Parallels with Software Engineering

Drawing a parallel with software engineering, Hamed Silatani highlighted how dealing with system incidents can be nerve-wracking. Unlike aviation, where pilots train regularly, software engineers might only handle incidents occasionally, leading to increased anxiety about the next big failure. Regular training and preparedness, as seen in aviation, can be beneficial in managing this stress.

The Importance of Organisational Culture

Ivan Merrill provided insights into how organisational culture impacts incident response capabilities. He stressed that preparation for incidents should start long before they occur. Just as aviation has the Black Box for data collection, tech teams need to build resilience and monitoring into their systems from the design phase. This requires a cultural shift towards prioritising reliability alongside new features.

Building a Culture of Psychological Safety

In high-stress situations, maintaining psychological safety is crucial. Ivan emphasised the importance of a blameless culture where team members feel safe to speak up and take decisive actions without fear of judgment. This culture needs to be nurtured over time and should be reflected in all aspects of incident management, including post-incident analyses.

Mitigating Human Error

Chandan explained how aviation deals with human error through a framework of avoidance, trapping, and mitigation. Pilots aim to avoid threats whenever possible, have plans to trap errors early, and mitigate the impact of unavoidable threats. This structured approach ensures that errors are managed effectively, minimising the risk of escalation.

?

Watch the full fireside chat here: