What are the best practices for minimizing the impact of IT incidents and problems?

Incidents are disruption or degradation to IT service. Problem is associated to root cause. Problem ticket would be raised to investigate the Root cause of the incident once it has been restored.

Incident is an unplanned disruption or degradation to an IT service, while a Problem could be underlying and potentially the cause behind a string of Incidents. Another way to clearly differentiate the two are the objectives of each of these processes. The objective of Incident Management is to first and foremost restore the IT service, while Problem management strives to determine the underlying root-cause to ultimately to prevent the incident from re-occurring. In real world example, a car's dead-battery is an Incident, and immediate resolution would need resurrecting it using another battery, or replacing the dead battery. A recurrence of dead battery could however could signify an underlying Problem with the car's electrical system.

Effective problem management is part of the ITIL Continuous Improvement component of the SVS as well as ISO/IEC 20000‐1 (service management system). For effective problem management, you don't need sophisticated Six Sigma tools; rather, you can use a simple technique called 'The 5 Whys'. Why is there a faulty hardware component? Because it's damaged. Why is it damaged? Because it was not replaced in time... You continue in this way until you come up with the root cause. Also, another simple tool to use is the 'Ishikawa' diagram, where you can write down the possible causes of the main problem.

Depending on the maturity of the IT organization, a healthy set of database of known issues, frequent failures etc. could contribute greatly to the Incident Management process. However, for most cases, many Incidents could be altogether new and will need a set of robust best-practices around: Detection, Response, Mitigation, Documentation and Post-Mortem Analysis. Each of the stages has its own broken-down steps, often requiring multiple disciplines across the organization coming together. For example, teams building the Observability/Alerting could be different than the teams communicating to stakeholders. Overall, strong ITSM Governance practices are needed to run Incident Management in large IT organizations with cross-cutting teams.

Incident management process. Goal is to restore the service as quickly as possible. Gauge the impact, analyse, decide which teams are need to restore the service. Follow the runbook associated with the service. Fact gathering starts once the incident is declared with right team. Do you Standard operational checks. E.g Health check, recent changes made to the env Check what is normal and what is now. Start the open ended question. Come up with a plan of action. Implement(if e-change or standard change) is required. Test (BVT, TVT) PIR (Post incident review) Problem ticket and problem task to reduce the future occerence.

The Problem Management process should be more nuanced than the Incident Management process and can often have longer lifecycles due to its inherent nature and the long-term goals of the same. Problem Management can also begin with Detection, Decomposition, Investigation, Diagnosis, Workaround/Known Error Database, Resolution/Release, Closure/Review. The most important areas to consider are the Decomposition of the problem to gather as much information as possible. Its also important to have regular touch-point between the Incident and Problem managers to detect repeat patterns, establish co-relation, share learnings or new observations, and keep the Problem ticket on track for a sound long-term resolution considering all possible scenarios.

Implementing the best ticketing system the market has to offer is one thing, while actually being able to choose or customize the one for your organization's requirements will be another ball-game altogether. Very often ticketing systems are disjointed between ITSM, Agile/Project Tracking, CMDB, Observability Platforms, etc. In an ideal world, a single interface could potentially exist for all of the above as most organizations have some level of interplay between them (or more domains). While the Holy grail may be a few years away, its still important to find the one which has the least amount of disjointed workflows and allows some level of data-pipelines to help with future integration and buildout of a true 'One Operations' department.

ITSM training is a must, but a technical aptitude is ideally required to begin with, as running an Operations organization would mean dealing with APIs, Databases, middleware, Operating systems etc. Beyond the technical skills, some of the lesser discussed but game changing skills are deep analytical skills, data science, Automation, Process Management, Quality Management, Troubleshooting & Root-cause analysis methodologies etc. Soft-skills such as crisp communication, cross-team collaboration, executive presentation can truly metamorphosize any operations unit into a top-notch team operating with high impact outcomes, saving operational dollars, elevating customer experience during issues, and powering informed decision-making.

Its not unusual that processes get formed, and many tend to stick around for ages, not getting refreshed or getting overlooked. Its ideal however to actually treat process making with a broader approach, rather than very bespoke offerings. Broad-based process formulation would focus on giving a set of guidelines with appropriate gaurd-rails, which allows smaller application teams to tailor some of the low-level processes as per their own needs. This maintains fluidity and avoids the central power dogma which could go against frequent process changes. Smaller teams can manage process changes within themselves much more efficiently, while higher-level guidelines tend to be changed less frequently.

ITSM implementation cannot work in isolation in achieving a desired Service Operation quality. It must work in tandem with SDLC, Agile processes delivering the software service, Hardware & Infrastructure Lifecycles, Customer/User Experience design processes etc. If an ideal state is to be painted, it could have all the large pieces working seamlessly. An example (using the AI) is: A low disk-space alert is triggered by monitoring system to the ITSM system, which logs a ticket, an AI-powered bot triggers a cleanup script which safely archives files from the server, checks for service quality, documents & resolves the ticket, sends a notification to IT department about the action performed, gathers feedback, adds use-case to its AI algorithm.

A Problem / incident management process are all core components of ITIL. Embrace it. Accept the importance of system stability on a par with delivery and performance. Delivery requires change which creates incidents and problems that require change (and round and round we go) An often overlooked part of this is a CMDB (configuration management database) The scope of any change should be known before release (What things are you touching). The cost of any incident or problem downtime should be known in terms of monetary and reputational risk. The recovery time objective and recovery point objective should all be known. CMDB can encapsulate all this and gives you a framework to form which to build sustainable stable delivery.

For me, it's pretty simple: A problem incorrectly solved (or not solved at all) becomes an incident. An incident not properly managed then becomes a major incident, and when a MI is not handled correctly, it can become a major disaster for the whole company and the rest, well, you know what comes next, so it's crucial to understand how a problem must be managed and resolved, but unfortunately, many companies (both small, medium and large) are only concerned with the 'golden' metrics of having only green #s without worrying too much in anything else. This is what is called the ‘watermelon SLA’ effect (green on the outside but red on the inside). Effective Risk, Change, and Problem mgmt are key success factors when trying to reduce incidents.