The Evolutionary Phases of IT Ops

IT OPS BY ANY OTHER NAME...

Historically, most companies have deployed their top software engineering talent to deliver the next 'bright, shiny objects' - meaning, new things. Being associated with the next innovative product, upgraded feature or new functionality has always held a great allure for developers. Despite this truism, there has been a change in recent years where more of the top talent is moving into the support of live systems. What I am referring to is the unmistakable rising tide of talent in IT Operations (IT Ops).

'IT Ops' refers to the teams that support the technology in use by a company's customers. IT Ops is also be referred to by other names - 'Break-Fix' or 'Business-As-Usual' or 'Production Support' or the somewhat less-elegant 'Base.' Regardless of what term we use, IT Ops teams do some extremely important things.

IT Ops teams:

• ensure that a company's technology is up and running for its customers
• troubleshoot issues, both inside and outside of a company's control
• manage the delivery of fixes in high-stress environments with tight (or no) timelines for resolution
• communicate problems to stakeholders as varied as external vendors, ovserseas developers, CEOs, the press and regulators
• work with the 'bright, shiny object' teams to improve quality in pre-production
• (often) drink ungodly amounts of caffeine

Simply put, IT Ops teams ensure that companies can make money. It's no wonder that companies are taking them more seriously than ever.

Having seen the inside of IT Ops teams in multiple companies over the last twenty years, I have had a unique opportunity to witness the evolution of IT Ops. For me, the most useful way to look at this evolution is to describe IT Ops in three phases.

A quick caveat before we proceed - Everything described below stems from my personal observations. While I believe that the points made are likely to be extremely common, and are indeed normative, I conceded that they are not immutable laws. Every company is different.

So, back to it... what are the three evolutionary phases of IT Ops?

THE THREE EVOLUTIONARY PHASES OF IT OPS

PHASE I :: The Customer Knows... Worst

In the beginning, IT Ops was largely defined by technologists who had a direct line to the call-centers and learned about problems when customers called in to complain. This meant that those teams inherently learned about issues after customers did. It also meant that smaller issues were harder to discern and troubleshoot because of the (literal) game of telephone at its foundation.

From a metrics perspective, Phase I focused on 'up-time.' Up-time was a binary measurement where technology was either marked as being 'up' or 'down.' Expressed as a percentage, the up-time metric inherently obscured a plethora of smaller problems due to the focus on major calamities. During Phase 1, most IT Ops teams measured their success by the reduction in minutes that they were 'down' for any given time frame.

In summary, Phase I was about responding to customer complaints after their experience was already degraded.

Then came Phase II...

PHASE II :: Proactive Monitoring

Phase II marked a significant point of departure for IT Ops, where dependence on the call-centers was largely replaced with proactive monitoring. In Phase 2, there was a flourishing of both home-grown and off-the-shelf solutions that enabled IT Ops to see issues at the same time as, or even before, customers saw them. In my team's role supporting BofA's online banking environment, we evaluated the health and welfare of that environment with solutions like Gomez, Mercury Interactive, and Webmon. Our relationship to the call-center then changed, focusing more on to the smaller issues that had been obscured during Phase I.

From a metrics perspective, Phase II started to focus on the customer as never before. Paradigms like 'FCIs' - Failed Customer Interactions - arose, which tracked the discreet number of customer failures and successes. With FCIs, companies started to focus on the customer experience in empirical ways. This was a huge improvement over the measurement of up-time.

In summary, Phase II was about using monitoring solutions to see issues earlier. It was a quantum leap forward.

Then came Phase III...

PHASE III :: The Rise of AI

In more recent developments, IT Ops has started to move to Artificial Intelligence. AIs are being deployed by IT Ops to look for small trends that could foretell a coming calamity, to decide when to reboot an ill server, and to develop subtle insights into quality issues that could be overlooked by a human being.

The move in IT Ops from humans-only to supplementary-AI is happening now and it holds great promise. AIs can do things that are either too difficult or too menial for a human to do. With the rise of automation, IT Ops teams will almost certainly continue to move up the value chain by winning back time for higher-value analyses on quality.

These are exciting times to be in IT Ops! But, there is still one nagging question that remains unanswered...

THE NAGGING QUESTION

"When will we start developing solutions that work right the first time?"

Too many companies are their own customers' most dire adversary. In many cases, new technologies are promoted to production and impact customers because of insufficient testing, despite negative testing results, as a result of pressure on tight timelines and short budgets, or simply because of human impatience. While some issues are truly outside of a company's control, I believe that most of the issues that impact customers are self-inflicted.

While not every company is the same, if your experience is anything like mine, you can relate to this challenge... many companies are their own customers' worst enemy because they roll out low-quality technology and expect IT Ops to deal with the fallout.

What can be done about this most serious challenge?

A POSSIBLE ROAD AHEAD

The automation that is part of the Phase III evolution could offer the best opportunity to change this unacceptable status quo. With the increase in automation, companies now have an opportunity to enable their IT Ops teams to improve quality.

Too many IT Ops teams still chase their tails by running from triage event to triage event while their executive management is immersed in minutiae. By getting the IT Ops talent to look away from the fires that they constantly battle, companies may finally be able to do a candid analysis of what is wrong with their tech delivery norms and then make it better.

IT Ops teams are one of the best repositories of knowledge when it comes to a company's technology quality. IT Ops teams could impart a great deal of insights on how to make things better. With the help of AIs, IT Ops may soon spend less time staunching wounds and more time figuring out why the patient keeps coming back into the Intensive Care Unit in the first place.

May you find the will to have the tough discussions and then make the bold decisions. In reality, both your customers and your profits depend on it.

@TriquetraIT on Twitter

www.triquetraIT.com