How Not to IT – Maturity Ain’t Cheap

Toward the beginning of the movie Ghostbusters, an important safety tip is revealed: don’t cross the streams!? In the climactic “boss fight” at the end, the only way to defeat the big bad and win the day is … cross the streams!? The same is very much true for IT frameworks.

?

So, what do I mean by “maturity”?? I’m not specifically talking about CMMI-style maturity models, but they are a fantastic source of inspiration, good/best practices, etc.? What I mean most in this article is that you cannot and must not ignore any part of a service’s lifecycle to save cost, save time, or just because you “don’t wanna”.

?

From ad hoc to repeatable to documented to optimized and beyond, maturing your people, processes, technology, organization, and culture is a journey you cannot skip if you want good results in the end.

?

In recent years, there’s been a mania for cost-cutting in the operation of IT services.? If that doesn’t jive with what you’re seeing/hearing, then perhaps there’s a mania for shifting your spending ratio toward innovation.? Either way, with a positive or negative spin, nobody wants to spend the majority of their IT budget on “keeping the lights on.”? I’m not here to tell you that’s wrong.? Far from it, but I am here to tell you that de-prioritizing that spend too much creates risks your remaining people may hesitate to tell you about.

?

I’m a big fan of holistic/systems thinking about IT in the context of the organization.? Neither “the business” nor IT exists (anymore) without the other, and neither can work effectively or efficiently without 24x7 communication and collaboration at all levels and in all contexts.

?

I’m still mentally wired for ITIL v3 and TOGAF 9 to some extent, but ITIL v4 and TOGAF 10 have some very good things to say in this space.? I don’t agree with some of the terminology, but I didn’t write it, so too bad for me.

?

Business Development

?

“Huzzah!? We just signed a new contract / got a new customer!”? That’s wonderful!? I’m so glad to hear it!? Was IT in the room for all of those discussions?? Did they have an opportunity to estimate cost (changes) required to support that new business?

?

All too often, the answers have been a resounding “no!”? In one case, a CIO defended a business unit’s ability to sign (literally!) anything a new customer put in front of them without IT’s input.? He said they couldn’t wait around for us.? He said they had the right to sign contracts without knowing whether we could make a profit on them or even if they knew we wouldn’t.

?

[picture a small dog, head cocked to the side, hearing something it clearly doesn’t understand]

?

At the time, we were an independently-run, publicly-traded, for-profit, Fortune 500 company.? Exactly how did they think our investors would like us signing multi-year contracts in the 8, 9, or even 10 figure ranges that may not or won’t make a profit?? Oh, wait!? That environment is no longer any of those things.? Activist investors got tired of our crap and did what they usually do.? Lesson learned. ?Again.

?

This is an element of maturity across the spectrum.? It’s definitely an element of business, people, process, and culture maturity to leave people out in the cold when their input could be make-or-break.? It’s also an element of technology maturity.? If IT doesn’t have good cost models, good vendor relationships, a good estimating track record, a holistic view of their internal structure, etc.; then the business has reasons not to trust us to provide good input on a timely basis.

?

Service Strategy

?

“Making these changes will save us a ton of money!”? That’s great!? Can you show us a detailed breakdown of our current costs, and where they’re allocated?? No?? Then, how do you know your proposed changes will save us a lot of money?

?

Again, this involves maturity across the board.? Will the change require more or fewer people?? Is that short-term or long?? Will it generate more and better data (it better!)?? Can we store it, move it, protect it, leverage it, back it up, etc. properly?? What kind of infrastructure and service lifecycles will be needed to make this work “at the speed of business?”

?

Sadly, I’ve seen environments where “the architects” hand down decisions and talk to internal or external customers without a glance toward the rest of IT.

?

“How do you want us to describe this new thing in the build request?”

“We don’t offer that combination of services.”

“We already promised it to your customers by next Friday.”

“Why?”

“Architecture told them it was a good architecture, and they should come ask for it.”

“Oh,” said in a very small voice.

?

Obviously, I’m condensing a few meetings, paraphrasing, and leaving out the naughty words, but that was real life in one IT shop.? Some folks might have already guessed what ensued.? We did what they asked, and … got ourselves and others into a 6-month series of meetings because various tools and technologies completely broke/failed in the presence of that “good architecture.”

?

Worse yet, that was an environment where certain deep-in-the-weeds technical workflows could be done in multiple ways that all worked fine for steady-state operation.? They found out the very very hard way that at least one of those ways catastrophically failed when certain components were rebooted.? I’m not talking about some major service incident.? I mean a simple, ordinary reboot broke things badly and lost internal customers’ data.? Oops!? No strategy, no design, no testing, no runbooks…

?

Honestly, this is just the same as the previous example between “the business” and IT.? If all the people who need to be in the room aren’t, your risk profile is both unknown and likely unpleasant.

?

But, service strategy cannot be ignored for even the oldest and most … seasoned (?) services in your IT portfolio.? Who’s keeping watch on when the next version will be coming out?? Who’s keeping watch on new requirements for what you have?? Who’s keeping watch on versions that are falling out of support?? Who’s keeping up the internal strategy documentation that customers can use to guide their requests? ?

?

“Hi, I need help with a …”? Wait.? Why are you laughing like that?? That’s not very nice!? Been there, done that.

?

Service Design

?

What happens if you have no service design people or artifacts?? This is all too common in IT infrastructure groups.? “We buy it.? You run it.? End of story.”? Really?

?

Going back to both of the examples above, every major component ought to have at least a generic service design that lays out menus of choices (and cost estimates!) before you get started.? Just like in a restaurant, requesting something that’s not on the menu will result in higher costs and longer lead times.? You want another green widget?? Boom!? Done.? You want something we’ve never seen or done before?? Well…

?

Beyond design documents, every service ought to go through some kind of serious proof of concept and have the results written up and stored somewhere searchable.? No technology is an island, so every team that will need to interact with or support your new component doesn’t have a seat at the table / voice in the process, your mileage will vary.

?

Astute readers may sense a Deming reference coming.? Absolutely, this can and probably should drive down variability in IT’s products.? It’s one of the ways to build in quality to both the processes and the products.? We may not build one million new VMware VMs in a year with only three defects a la Six Sigma doctrine, but every one we build should have good documentation, good testing, etc.

?

Spoiler alert:? without service designs, critical parts of service operation just won’t work well, if at all.

?

How does a customer know whether a particular service will meet their needs?? Theoretically, a set of generic service documents should lay out menus of choices, call out which ones will and won’t work together, and enable the customer to make good choices or understand that they’re ordering off-menu.

?

Whether we’re talking about service designs a la ITIL or re-usable building blocks a la TOGAF, they need to exist, they need to be kept up to date, and they need to be searchable.? All of that takes time, money, people, technology, and most importantly, culture.? If the folks at the top ignore this process area, everyone else will assume that all requests are fair game and cheap.

?

Service Transition

?

Unfortunately, this is where the bus tires hit the back in a lot of shops.? “Change management is a quick win we can show people to gain more support for our framework du jour!”? Ummm…? No, it’s not.

?

This is one of the least mature areas in a lot of shops precisely because it’s one of the “quick wins” that folks like to do early.? Sure, you don’t have a configuration management database (CMDB) worthy of the name or configuration items (CIs) for a ton of your critical infrastructure, but go ahead and implement change management early.? Each and every single change record that doesn’t have good links to CIs in a working CMDB is of lower value than it could be, and that lower value will probably remain so forever.? Who’s going to hire an army of data entry clerks to add details to thousands of old change records later on?? If you ever need to mine that information, you’re likely to pray for good typists and do a lot of brute-force text searches.? Maybe AI can help.? O:-)

?

But, what is it you’re changing in the first place?? Is it following your service strategy?? You do have one, right?? Is it a design change that ought to flow back to the generic design doc?? You do have one, don’t you?? Is it an implementation change that ought to be reflected in the specific design doc?? You do have those, right?? Oh, it’s brand new!? Well, I’m sure you have all the documentation for strategy, design, and operation ready to go.? Let’s see it!

?

Maturing service transition processes is perhaps one of the toughest challenges out there.? It’s also a constant source of “ITIL is too slow for Agile!” and other kinds of complaints.? Pick your poison: Lean, Six Sigma, CMMI, Kaizen, whatever.? There are lots of ways to improve your processes, make them faster, or make them dovetail with other processes better so people stop working around them.

?

“Drop it on the floor and see who picks it up.? They’re the new service owner.”? I promise, that really never works and often leads to some really bad outcomes.

?

Of course, it’s not enough to talk about changes to our IT estate.? We have to think about changes to our people.? New hires need on-boarding and help assimilating, since all frameworks are locally adopted and adapted.? No, you didn’t “do” ITIL, and you’re not “done” now.? Exiting staff need replacement, and you need to capture their unique knowledge wherever possible.? Folks moving to new jobs need a bit of both.? It’s no fun to get stuck doing your old job for another 5 years because nobody took it over.

?

So, who’s doing all of that?? This is one of those areas where IT must not go it alone.? Every single time anyone in the organization has an “I need to do this, but I can’t figure out how, whom to ask, or anything” moment is a potential glassdoor.com posting your HR folks won’t like.? Of course, that’s the best case scenario.

?

Service Operation

?

And, here we go!? “Let’s do incident/request management first, since they’re so easy and quick wins!”? Time to look for another job.? Quickly.

?

That’s really not how it works.? Incident management is one of the hardest things to master in all of IT, and it touches every single other part of your service and organization lifecycles.? That’s especially true if you want your service desk to work on single-call closure rates.

?

No strategy documents or can’t find them?? They can’t answer questions about “when will X be available?”

?

No design documents or can’t find them?? They can’t filter requests for “we don’t actually support that.”

?

No CMDB/CIs?? They can’t filter requests about things that don’t exist (anymore) and probably can’t route incidents/requests properly for the things that do.

?

No on-boarding docs or can’t find them?? Sorry, hon, you’re on your own.

?

But, it’s so much worse than that…

?

“We’ve decided to out-source Area X.”? Oh.? Have fun storming the castle!? Every single document/spreadsheet/artifact you don’t have yet is a win for the out-sourcer’s bottom line.? I promise you they’ll be grinning ear to ear about back-filling all of the physical and logical/virtual inventories, runbooks, disaster recovery plans, and documentation you can’t provide them – billed on a time & materials basis, of course.

?

My biases against out-sourcing are probably well known by now.? They didn’t get any better when I was on the other side of that divide…

?

Gartner-style bi-modal IT is one of the things that really makes this tough.? If you’ve got folks who are in “run” groups, not empowered to make substantive change, etc.; the quality of your service operation is likely to go downhill almost as fast as their morale.? You might be one of the lucky shops where the chasm between innovation/change and keep-the-lights-on groups is bridged somehow, but it’s almost guaranteed to be sub-optimal.

?

Adopting some kind of “plan, build, run” organization only makes it worse in most cases.? How many folks out there want to be button-pushers all day?? How many want to keep the lights on?? And, how exactly do you recognize or reward them?? “Kudos to Dave for not messing up again this year!” ?All-in-all, it’s a hell of a thing to do to your IT shop, especially in 2024.

?

Think I’m kidding?? Sit at a table with aligned “build” and “run” people and listen to them laughing their asses off about pinging requests back and forth between their teams.? “You’re the experts!”? “No, you built it!”? “But, you run it!”? And so on until they need another beer…

?

Service Improvement

?

If you’re not interested in mature services or processes, I can just leave this part blank.? You’re not really doing it, or everything you’re doing is likely un-focused and ad hoc.

?

If you don’t have anyone concentrating on service improvement for their services, entropy will take over, and things will slide down the slippery slope to oblivion.? Asking transition / operation (old school admins) to do it is … fraught with peril.? Some just won’t get it.? Some will never get past incremental thinking.? Some may be stuck in ticket-worker mode.? They need support and not just from management.

?

Odd Ducks

?

As ITIL and TOGAF (among others) are frameworks that must be adopted and adapted locally, there are plenty of areas that straddle lines or could drop through the cracks.

?

“Your change request looks great.? Did you attach the benchmarking results?”? Ummm…? What benchmarking results?? Collected in what benchmarking environment?? That straddles the design / transition line, but it’s one of the things a mature service really needs from time to time.

?

“Shucky darn, that new SQL query is causing havoc.”? I guess it wasn’t benchmarked in anything like an apples-to-apples environment.

?

“Holy crap, we’re burning through storage fast!”? I guess it wasn’t benchmarked in anything like an apples-to-apples environment.

?

This is one of those areas where it can be really tough to allocate the money.? I’ve been in exactly one IT shop in my life that not only had benchmarking gear set aside, they actually had benchmarking specialists set aside to help with it.? On the flip side, I’ve been in a place where “the business” signed off on a service level agreement that every single transaction would be handled in 4 seconds or less with no benchmarking gear or people, with no serious load-testing gear, and with neither service strategy nor service design processes in place.? They’d been live in production 4 years before we heard about those “requirements.”

?

Disaster recovery / business continuity is another “neither fish nor fowl” kind of thing.? Every single service needs to have IT service continuity management parameters in its design.? You have designs, right?? Generic designs lay out the menu of options and costs.? Specific designs pick what matches the business’ requirements.? Right?? It’s a place where the “DR Team” may be flopping around at loose ends much of the year if you’re not maturing their processes.? Full-scale testing may be too expensive for your tastes, but small scale?? Tabletop exercises?? Documentation reviews?? For most places, it’s more than a full-time job and an area that never really matures.

?

I’ve been in exactly one IT shop in my life that did planned fail-overs of their entire, core data center regularly.? I was young(er) and less jaded, but it was still an amazing accomplishment.? It was not, of course, cheap.

?

Similarly, you might still have local availability requirements beyond what either a single system or a bunch of independent systems can provide and need some kind of clustering service.? Again, I’ve been exactly one place that did planned “role swaps” on a regular basis.? They had to buy extra hardware, but they were serious about keeping their services up and running.

?

The canonical “test or fail” scenario is backups, but even there, the road map is no longer clear.? Restoring every kind of backup once in a while is a very good idea that takes people, process, technology, and culture assets.? “We can restore any backup in reasonable time!” means exactly bupkis when the ransomware actors just laid waste to your backup infrastructure and the vast majority of your servers in an hour.

?

Speaking of serious, I’ve been exactly one place (not an out-sourcer) that had IT vendor engineers on-site at least 40 hours per week with their own cache of spare parts.? A memory DIMM failed in a critical, database server while some of us were on-site.? By the time we gathered up and walked to the data center, those engineers had gotten out their crane (looking at you, HP V-series), pulled the cover off the box, identified the failed component, raided their stash/cache, and were already close to done.? At the same time, their clustering software had done its job properly, and that critical database was up and running on another physical server.? It was extremely impressive, but again, not cheap.

?

But, The Cloud?

?

Yes.? I can hear you out there.? Your mental voices are coming in quite clearly.

?

“Doesn’t the cloud change all of that?”? Oh, waiter!? Check, please!? No.? The cloud changes literally none of that.? The names may change and roles may change here and there, but overall, the need to implement mature services with the right combination of people, process, technology, and culture doesn’t change at all.? In fact, there are places where it’s more critical.

?

In much the same way that out-sourcers will happily charge you to back-fill all of the documentation you can’t give them, cloud providers will generally be quite happy to start charging you by the second to run your workload(s) no matter how immature they or your processes are.? No documentation?? That’s on your side of the shared-responsibility model.? O:-)

?

You may currently rely on some “keep the lights on” people you plan to lay off when you move everything they support to the cloud.? It’s okay.? It’s natural.? Communicate and lead well, and they may not even hate you for it.? However, think about this for a moment.? Does everyone you’re about to enable with “as a service” / self-service functionality understand what those “keep the lights on” people were doing for them and why?? If not, maybe think again…

?

“But, they do reviews!”? [pat on the head]? Sure, they do.? Do those reviews drill down into how application stacks A and B for the same business unit are completely different, managed completely differently, installed completely differently, etc.?? Or, are they just making sure some cloud tenets are being followed?

?

This is, of course, another time when everyone who needs to voice an opinion must be in the room.? Sure, they may be leaving the old, slow, tiresome, on-premises world behind, but do they know everything they need to know to go it alone with little or no safety net?? I’d bet against it, but I can count up several P1 incidents with significant monetary penalties on my side of the ledger already.? If you feel like taking that sucker bet, I’ll take your money with nary a qualm.

?

Much like Agile + DevOps + Infrastructure-as-Code is (yes, singular – all for one and one for all) the on-ramp to the Public Cloud, mature services and a mature culture and organization are absolutely critical to running a 21st century business.? You’re either taking a holistic / systematic approach to integrating IT throughout the business and across all areas within IT, or … you’re not.