What do you do if your IT team is facing a critical system failure?

Establish a centralized incident management platform for streamlined communication and collaboration, conduct regular drills and simulations to prepare the team for handling critical system failures, maintain up-to-date documentation of system configurations and dependencies to expedite troubleshooting efforts, leverage automation tools for rapid response and remediation of recurring issues, and foster a culture of continuous learning and improvement to strengthen the team's resilience and capabilities in managing future incidents effectively.

Define the problem clearly but identify not just where the problem is, but also where the problem isn't. Prove or disprove any assumptions that are being made. Prioritising effort into investigations is challenging, especially if you have resources that are sure they know what the issue is and they disappear down a rabbit hole trying to fix it. Pulling people together at regular intervals to share updates frustrates some people as they want to be working the problem, not discussing it. Ensure you clearly articulate the need for collaboration and remind the team that you also have to provide updates to the business/customers.

The first priority is to isolate the root cause of the failure. This may involve analyzing logs, checking system health, reviewing recent configurations, and investigating unusual events that may have led to the failure.

Be careful to not get mixed between the incident and the root cause. The first thing to do is to identify the severity of failure (incident prioritization), this will determine your next steps. Sometimes getting the root cause will took longer time than expected. Your first priority is to have the incident fixed, even with temporary workaround/solution. Check logs, contact vendors, etc is to fix the incident first. Business/our user needs to have our system up and running as soon as possible. Find the incident, isolate it and fix it. Record every works you did to solve the incident. And once the incident is fixed and the system up and running, you can start problem investigation to find the root cause and the permanent solution.

When facing a critical system failure, swift action is crucial. Begin by assessing the situation, notifying stakeholders, and activating the incident response plan if available. Mobilize the IT team to troubleshoot and resolve the issue while implementing temporary workarounds to minimize disruption. Clear and timely communication with stakeholders throughout the process is essential for managing expectations and maintaining transparency.

In the event of a critical system failure, the IT team should immediately initiate incident response procedures, prioritize resolution based on impact and urgency, communicate transparently with stakeholders about the situation/ expected resolution time and collaborate effectively to restore service as quickly as possible while adhering to established protocols and contingency plans.

Immediately inform interested parties about the critical system failure. This may include the technical team, leadership team, affected users, and other relevant parties. Clear and transparent communication is essential to maintain trust and ensure a coordinated response.

Sending communication to your stakeholders is 1st priority. Ensuring that they know the incident from you will shows that you own the issue. Even if you haven't found the root cause, the communication should be sent, as you can mentioned that the incident is still under investigation with the maximum priority. The regular update should be sent whatever the progress, even if you still stuck in finding the cause, just send it. Again it shows that you still put your max attention on it. Once the system is up and running, complete your test and once you sure that the failure is fixed, send another communication. Sometimes you fixed the incident with temporary workaround, just mentioned that the permanent solution is still in progress.

Don't forget to document everything. Keep detailed records of the incident, response actions, communications, and resolution steps. This documentation is crucial for the review process and for any necessary reports to stakeholders or regulatory bodies.

After the failure is resolved, conduct a post-mortem analysis to review the incident, identify lessons learned, and recommend improvements to prevent future similar failures. This will help strengthen system resilience and increase IT team responsiveness.

In ITIL, this steps are included in Problem Management. Runt the process, do root cause analysis to find the main root cause, then you may find some area of improvements. For example, if the root cause of a system failure is due to unnoticed certification expiration, your preventive could be having a TLS/SSL certificate list where your service team regularly check and maintain when needed. Or if you find the incident fixing was done in longer time than expected due to no one know how to run a rollback, then your area of improvement will be ensuring all major changes goes to your production should have a detail rollback plan.

The problem statement will evolve as you learn more. Be mindful that formal statements you may have to issue based on early assumptions may end up being wrong, so you have to the the master of making confident statements that are broad enough at those early stages.

Apply best practices such as ITIL Major Incident Management. This will help you to define what to do in facing major incident/critical failure. Sounds like very theoretical, but applying this means you don't learn hard way from your mistakes but you learn from others' and just apply the best practice with some adjustment based on your business needs.