登录查看更多内容

How do you handle system monitoring emergencies or incidents?

由人工智能和领英社区提供技术支持

System monitoring is a vital part of ensuring the performance, availability, and security of your IT infrastructure. However, sometimes things go wrong and you have to deal with emergencies or incidents that require immediate attention and action. How do you handle system monitoring emergencies or incidents effectively and efficiently? Here are some tips and best practices to follow.

此文章中的业界达人

由社区从 5 条内容中精选。了解更多

1 Assess the situation

The first step is to assess the situation and determine the scope, impact, and urgency of the emergency or incident. You need to gather as much information as possible from your system monitoring tools, logs, alerts, and reports. You also need to communicate with your team, stakeholders, and users to understand the business context and expectations. Based on the assessment, you should prioritize the most critical issues and assign roles and responsibilities to your team members.

添加您的观点

Karl TS Jackson (Yulaw)
举报内容
Before we get here, there has to be SOPs in place. There are a lot of companies implementing monitoring systems to check a box. Not every person capable has been looped in or knows the prescribed steps to follow.

已翻译

赞
Sagar Navroop

? Architect | ??????????-?????????????? | Technologist
举报内容
Swift identification and response are critical to address critical system monitoring emergencies If using an AWS Ecosystem; utilize CloudWatch for proactive monitoring, and set up alarms for key metrics. Implement CloudTrail to track API calls for incident analysis. Deploy AWS Config for real-time configuration compliance monitoring needs. Invoke AWS Lambda for immediate incident isolation and response needs. Follow the incident response plan, involving predefined run-books for routine issues. Use AWS X-Ray for distributed tracing, and root cause analysis. Regularly conduct GameDays to simulate incidents, refining responses. Ensure continuous improvement through post-incident reviews, applying lessons learned for system resilience.

已翻译

赞

2 Follow the incident response plan

The second step is to follow the incident response plan that you have prepared and tested beforehand. The incident response plan should outline the steps, procedures, and resources needed to resolve the emergency or incident. It should also define the escalation paths, communication channels, and documentation standards. You should follow the plan as closely as possible, but be ready to adapt and improvise if needed. You should also update the plan as you learn from the experience and feedback.

添加您的观点

3 Isolate and contain the problem

The third step is to isolate and contain the problem as much as possible to prevent it from spreading or worsening. You should use your system monitoring tools to identify the root cause, isolate the affected components, and apply the appropriate mitigation measures. You should also monitor the impact of your actions on the system and the users. You should avoid making drastic changes or fixes that could introduce new problems or risks.

添加您的观点

Michael Lamb

Systems Engineering | DevOps | Site Reliability Engineering | Network Operations Center Manager||Ex-Amazon|Ex-Disney
举报内容
A good path to follow, is to look at what has changed in the environment. Look for deployments, or recent change records (you do record your changes and approvals, right?). Eliminate those as culprits to the issue as soon as possible. If no changes are evident, then start with standard “paths” through the application and verify each step is functional. Pay attention to databases and other typical bottlenecks. If services are all functional, move on to networking or external dependencies (3rd party integrations, etc), as they may be down.

已翻译

赞

4 Restore and verify the normal operation

The fourth step is to restore and verify the normal operation of the system as soon as possible. You should use your system monitoring tools to confirm that the problem is resolved, the system is stable, and the performance and availability are restored. You should also verify that the users are satisfied and the business functions are resumed. You should communicate the status and outcome of the emergency or incident to your team, stakeholders, and users.

添加您的观点

Michael Lamb

Systems Engineering | DevOps | Site Reliability Engineering | Network Operations Center Manager||Ex-Amazon|Ex-Disney
举报内容
During the investigation and resolution, aim for the simplest and fastest actions to bring resolution (restarting services, adding capacity, etc) Avoid the urge to create “long term solutions” to the issue on the fly. The initial goal is to return it to service, not prevent it from occurring again. After it’s back up, you can spend the proper amount of time to create the necessary changes to ensure it’s permanently fixed.

已翻译

赞

5 Analyze and learn from the incident

The fifth step is to analyze and learn from the incident and improve your system monitoring practices. You should conduct a post-mortem analysis to review what happened, what went well, what went wrong, and what can be improved. You should document the lessons learned, the action items, and the recommendations. You should also implement the changes and improvements to your system monitoring tools, processes, and policies.

添加您的观点

Michael Lamb

Systems Engineering | DevOps | Site Reliability Engineering | Network Operations Center Manager||Ex-Amazon|Ex-Disney
举报内容
I would also suggest having a mechanism to share those lessons broadly across the organization, so that other teams can learn issues to avoid from the start.

已翻译

赞

6 Review and update the incident response plan

The sixth step is to review and update the incident response plan based on the analysis and learning from the incident. You should evaluate the effectiveness and efficiency of the plan, identify the gaps and weaknesses, and make the necessary adjustments. You should also test and validate the updated plan to ensure that it works as expected. You should keep the plan up to date and accessible to your team and stakeholders.

添加您的观点

7 Here’s what else to consider

This is a space to share examples, stories, or insights that don’t fit into any of the previous sections. What else would you like to add?

添加您的观点

System Monitoring

+ 关注

给文章评分

我们借助人工智能创建了此文章。您认为这篇文章怎么样？

很棒不太好

举报此文章

查看全部

How do you handle system monitoring emergencies or incidents?

1

2

3

4

5

6

7

1 Assess the situation

2 Follow the incident response plan

3 Isolate and contain the problem

4 Restore and verify the normal operation

5 Analyze and learn from the incident

6 Review and update the incident response plan

7 Here’s what else to consider

System Monitoring

给文章评分

感谢您的反馈

更多System Monitoring相关文章

更多相关阅读内容