Innovation & Automation: Are they at battle?

We find ourselves in a peculiar age, surrounded by buzzwords and leading the parade is the evergreen pair of Innovation and Automation. Is there a possibility that they are engaged in a tussle? In a ramble that risks oversimplification of the two, I try to find myself an explanation.

Having managed delivery, support and in my current role leading quality engineering organisations - all of them maintaining these as particularly important goals, I find myself constantly balancing priorities and the resulting conflicts between the two - while constantly guarding against the universally dreaded "mindless automation" and "innovations that don't influence outcomes". In a sadistically assuring manner, this battle is not just mine.

Are we losing critical peripheral vision while speeding towards 100% of everything?

Consider this paradox for a moment: We have found faster ways of producing food now and yet find ourselves with lesser time to consume it. Effective, but not wholesome!

There seems to be a paradox in our buzzwords under consideration too.

Innovation and Automation share an endearingly simple but amusing relationship. They feed each other and feed on each other at the same time.

Higher automation → rapid innovation

Rapid innovation → disrupts automation (both positively and otherwise)

And this lands engineers at cross-roads especially within IT organisations that support the backbone of products and services.

If we loosely define Innovation as finding newer ways of doing things, and Automation as finding faster ways of doing things - we find that both have a common goal represented very simply as:

Lesser Cost, Higher Velocity, Higher Quality → Better Developer Experience → Better Customer Experience

If they have common goals and hold great synergy, why do organisations struggle to achieve both at higher measures? Let's look at two of many reasons.

Firstly, architecture! We haven't completely crossed the bridge between classic and modern architecture in the same landscape. And add to this the fact that organisations seem to be in a continuous state of rapid transformation. In an oversimplified comparison:

• Classic Architecture: Consolidates infrastructure and applications, tightly coupled business processes boxed into packaged solutions
• Modern Architecture: Fragments infrastructure and componentizes application features into loosely coupled solutions

Secondly, generalising measures for both. In an organisation that is stepping out of being a startup and moving towards tighter structure and maturity of processes and stepping into the zone of higher impacts of changes, the common trap is to define blanket goals for automation percentages and focus for innovation, and failure to dig a bit deeper to associate value to outcomes. Is the holy grail of 100% a real need?

• Assumption: Mid-sized organisations often find themselves with an application landscape that is a combination of classic and modern architecture as well as a host of third party applications to support features that are not core to the organisation's product or service.
• Outcome: Lets simply consider binary outcomes: lesser cost, higher quality, higher velocity
• Weightage: This should be a focus area. Not all outcomes demand the same weightage. Breaking down weightage into following parameters for consideration, we find ourselves in need to define optimal automation and considered innovation:

1. Speed to reach outcome
2. Scale of impact of the outcome
3. Longevity of impact of the outcome

In a typical IT setup that has to service broadly Finance, Sales, Marketing, Billing & Revenue, Reporting, Mobility and to an extent Customer-facing portals & applications, there is a need to also categorise desired outcomes based on the speed of change in processes and technology.

Typically the questions that need to be asked: What should be our innovation and automation approach for processes that don't change rapidly such as in Legal, Finance, Compliance and Audit constrained flows etc. and for processes that change rapidly such as in Sales, Marketing, Billing & Revenue, Customer Experience etc.

It is not surprising that we have a predominant average of 45-55% automation in IT development and quality engineering processes in most mid-sized/large organisations but this is not necessarily a bad number. To bring leadership expectations and engineers' effort to an agreement, it is necessary to define optimal automation as a function of outcome and weightage for each business process and tech stack. And in my humble opinion, hard decisions towards well considered compromises are not necessarily bad in the larger scheme of things.

Simply put, we are at a time where classic architecture and classic processes put us in a situation where "we find faster ways of doing things more rapidly than we find more things to do"; modern architecture and dynamic processes lead us towards the situation where "we find more things to do faster than we find ways to do the same things fast".

The recognition of this dual reality within the same landscape could lead to un-burdening of engineers and clarity of where their efforts make most sense and derive most value.