Here's why enterprise IT is so complex.

Note: My blog has moved to its own website. You can read this and new posts at ArchitectElevator.com

Enterprise IT is plagued by quite a few things, but excessive complexity must be near the top of the list. Any attempt to depict the average IT landscape ends up in an undecipherable spaghetti of applications, hardware, and inter-dependencies. It's almost like enterprise IT is subject to the Second Law of Thermodynamics, which concludes that the entropy in an (isolated) system can never decrease - at best it can be constant, but usually it increases.

The Price of Complexity

Excessive IT complexity causes a number of serious problems:

• It drives up cost because managing more pieces requires more effort and generally also requires a larger set of specialized skill sets, which come with a higher price tag.
• It drives down reliability. Complex, highly interdependent systems tend to have poorly understood failure states and large failure domains: it's difficult to know what can go wrong; if something does go wrong, it's difficult to know what actually happened; and if one part breaks, the problems cascade in a domino effect. Together, complexity leads to shorter time between failure (MTBF) and longer recovery times (MTTR), both of which lower system availability.
• It drives up vulnerability. Simple systems are generally more secure than complex ones because complex environments are more likely to have a weak link, e.g. an unpatched system or a default password hidden somewhere in a long-forgotten system component.

Understanding that much of modern IT is driven by security, uptime, and cost, complexity is definitely in the running for IT's public enemy number one as it hurts all three factors!

Living with Complexity

Realizing the serious damage complexity does, we should expect IT managers and decision makers to fight it with all their energy. Strangely, though, too many times they seem to be resigned to the fact that complexity is a fact of life in enterprise IT, some sort of karma if you wish. In most cases, though, we find that complexity persists not because the CIO isn't trying. It's due to the fact that most parties surrounding the CIO have little real incentive to making it go away:

Vendors

Most enterprises correctly follow a "buy over build" approach, meaning much of the software and hardware they run is purchased from enterprise vendors as opposed to built in house. This enterprise software isn't cheap - many IT organizations spend about 1/3 of their budget on licenses. The software vendors benefit from complexity in several ways, though:

• They tend to compete with long feature lists, which increase product complexity. This isn't entirely their fault as they know that IT organizations score competing solutions by the presence of features, even though users may actually never use them (this is another great example of a Disablement Cycle).
• Unsurprisingly, successful vendors are good at selling: "You already have an application performance management framework? Ours is better and integrates with your existing solution!" Much of enterprise complexity stems from having multiple products for the same problem.
• Vendors also create complexity in the product strategy. If marketing hasn't coined a new buzzword or positioned a new-and-improved solution within 6 months, the sales pipeline is at risk.
• Lastly, hardware vendors benefit from complexity as they can address many problems with more hardware as opposed to rightsizing or simplifying the software solution.

System Integrators

Software that needs to be built or the integration of purchased software is generally done by system integrators (SIs). They play an important role as they provide specialized skill sets that the internal IT may not have or not in the required quantity. SIs generally don't sell products. They sell a service, provided by consultants. While perhaps packaged in all sorts of agreements, at the end of the day these consultants bill by the hour because the staff hour is the fundamental unit of consulting economics.

Generally, more complexity means more work and thus more revenue. This isn't necessarily intentional or devious - self-preservation is at the very base of Maslow's Hierarchy of Needs.

Consultants

While the SIs are the enterprise's helping hands, consultants are the hired brains. They solve complex problems and devise IT strategies. But they also want to remain employed, so they might solve a problem almost completely, leaving just enough for more work to be done. Self-preservation equally takes hold here. In consultant vernacular this is called scoping or expectation management.

IT Staff

But not all fingers should point outside the organization - quite the contrary. The internal IT folks love complexity. In too many organizations the person with more budget and more moving parts is still considered more important. Ironically, complexity is a career booster!

It's also an ego-boster. Many IT managers love to let you know how sophisticated (read complex) their operations are - consuming huge amounts of hardware gives bragging rights: "One big man (person), one big IT infrastructure". Perhaps they're looking for a bit of appreciation. IT is a tough job, so wanting to let people know how difficult it really is, is more than understandable.

Lastly, internal fiefdoms lead to duplication - a quick path to unnecessary complexity.

Managing Complexity

Complexity won't go away. The current pace of technology evolution adds new moving parts all the time and too many parties surrounding the CIO are quite content with a little bit of complexity. How to deal with it is likely enough material for another post, but to avoid a classic cliff hanger, here a few ideas:

• Transparency. Complexity is double bad if it isn't known. Gaining transparency across the IT estate is a first critical step towards getting control of complexity.
• Architecture. Architecture done right develops models and abstractions that hide irrelevant complexity and allows us to make better decisions. After all, you can't manage what you can't understand.
• Standards. Interface standards localize diversity and reduce complexity.
• Don't outsource thinking. Keep control of your IT in house. More on this in the next post... (so we do get a cliff hanger after all).

Eager to know more?

I hope you enjoy my thoughts on large-scale IT management and IT transformation. You can find more real-life stories in my book 37 Things One Architect Knows About IT Transformation:

• DRM-free e-Book: ArchitectElevator.com
• Hard copy via Amazon US, Amazon UK, Amazon Europe