The biggest challenge in adopting new technologies

I was recently asked by Daniel Blickling “what, in your experience, is the biggest challenge when it comes to finding and integrating new tech that isn't a fad?”

For some reason, rather than just giving a succinct response, the question crystalised some thoughts that have been sitting in the back of my head for a while now,  triggering the following rant, which is a slightly cleaned up version of my reply, along with some pictures to make it look pretty.

IT Doesn't Matter

In short, the biggest challenge is that many organisations don't have a good process for evaluating new technology, mostly because its IT department is a victim of the "IT doesn't matter" mindset that Nicholas Carr exemplified ...which I might summarise as "if IT is non-differentiating for the business, then your primary IT management activity should be squeezing the costs out of it". This becomes self fulfilling because IT is then starved of the resources needed to make good investments in things which aren't immediately productive. If IT was aligned to business strategy, then it would benefit from the same level of discipline and investment that goes into that strategy. However, without that alignment it can't or at least, doesn't spend R&D money effectively. From my point of view, the most promising way of investigating where IT investments should be spent would be via a "Wardley Map".

It's worth going to Simon Wardley's blog https://blog.gardeviance.org which explains the benefits of strategic mapping, and I'm only just beginning to really get my head around the process. Something I'm a little more familiar with is the "3 Horizon" model, which I believe is a McKinsey construct. It doesn't have the sophistication or the navigational utility of a Wardley map, but it's pretty easy to understand.

Narrow Horizons

Because IT practitioners, especially in the Infrastructure and Operations part of the business are for the most part stuck in Horizon 1, which is just "extending the core", rarely is it aligned with the future plans business, and tends to invest in a series of cost aligned short-term fixes to cater for what are now unforeseen requirements. Vendors do everything they can to make adoption of these new technologies as easy as possible. Often the most successful products in the market are the ones that let me "do exactly what I'm doing now, but faster and cheaper", which then often leads to the Jevons Paradox resulting in increased rather than decreased IT expenditure, with virtual server sprawl being a prime example of that.

Innovation Constipation

That would be OK, if it wasn't combined with something I call "innovation constipation", where we are able to easily adopt new technology, but we are AWFUL about killing our darlings and retiring old stuff sitting in the "Run" parts , or even new stuff that has gone past the "transform" phase and into "grow" phase of Gartner's IT spending categories, even when other options look way more promising.

There's some good evidence that moving budgets from run through into grow and transform spending categories (which are closely aligned to the three horizons from McKinsey) can have a significant impact on the profitability of a business, however, while there is much agreement that it should be done, my experience is that it happens very slowly, if at all. My personal belief is that a lot of that is because of "resume driven development", where a pet project keeps going past the point where it should have been killed off because someone needs to show their success in that for their next job.

Resume Driven Decision Making

One example I've seen are the Rockstar CIOs who do "let's cloudify ALL THE THINGS" and two years into the process, hail their success at conferences and then leave to do the same thing at the next company. Or the developer who decides Docker Swarm is awesome so he can put "DevOps" on his resume and then gets a higher paying job because of it, only to be replaced by the next guy who rips it all apart to put in Kubernettes, and then the next guy who decides that cloud foundry is the right choice etc, with none of them ever quite getting it to the point where the investment turns into productivity.

Inflight Magazines, Distractions and Squirrels

In the absence of a strategic approach to investigating new technologies and a well understood and disciplined methodology for cutting your losses and killing tech that needs to be replaced (refactoring not just code, but infrastructure as well, to get rid of technical debt) then new technology decisions will often come via the "inflight magazine effect" where the CxO sees some article about the next hype on the block and wanting to be seen as innovative, starts asking "why aren't we doing supercool hypey things.next ??" which is another form of resume driven development .. it's more about egos and self promotion than technology adoption.

The trouble is that once this sets itself up as a pattern for innovation decision making, it becomes subject to the "SQUIRREL !!!" effect, where half laid plans are interrupted to pursue whatever that has caught the eye of innovator who's love of novelty and promise of the next big thing makes it impossible for them to focus on delivering pretty much anything.

Of course, all of this applies mostly to the kinds of companies that are generally "early adopters/early majority" on the innovation diffusion curve. Late adopters and laggards have different problems and people at the "innovators" segment either have a really good process already, or are out of business, unless they get really, really lucky (that describes most startups IMHO). Nonetheless, without IT being aligned to the business strategy (however that is conceived, planned or visualised), the problems caused by that lack of alignment will remain the single biggest challenge in adopting new technologies.