Rule 5: Software and data - it takes the two to tango

This is the fifth rule for digital business renewal. For full context, see Introduction:

Software and data are fundamental enablers for strategic agility. Although they are conceptually different, they share many commonalities in the way they are used for value creation.

Software development has gone thru decades-long evolution resulting in quantum leap in productivity. Data utilisation is now about to follow the suit with basically the same target: High volumes of data products while maintaining good quality and combined with short lead times. In this respect, software development has reached maturity but its data analytics sibling is just coming of age. It would be foolish not to take heed of older sister’s hard-earned wisdom.

“Every company is now a software company”

It is almost impossible to talk about strategic agility without software. In the digital technology defined business environment, no new customer solution becomes a reality without software. In the overall business landscape, own software development is needed to differentiate, see description of the three application domains in earlier article on architectures:

Software development to differentiate basically applies to all product and service based businesses. Only in pure manufacturing industries may standard enterprise applications provide the necessary operational efficiency without any software tailoring.

There is no agility in 18-month lead time and the fundamental assumption that nothing changes in between.

However, the way software is developed determines how nimble and flexible company can become. There is no agility in the paradigm of 18-month lead time and the fundamental assumption that nothing changes in between. Still, this is how it used to be not so long time ago. Software in itself does not imply agility. Rather, agility comes from operating model, technology platform and architecture deployed for software development. Let’s explore each of them.

Operating model for software based agility

To improve responsiveness and shorten time-to-market, software industry has taken many decisive steps. The primary change has been adoption of Agile development practises. Agile is an umbrella term for multiple iterative and incremental development methods that have delivered signicant gains in speed and flexibility. Instead of organising software development as massive projects with multiple stages and with gates in between, Agile is based on short development cycles run as sprints. In this way, development can react to new understanding and changes in customer needs. In the world of uncertainty and unpredictability, nothing beats this in-built learning ability.

However, software development is not only about coding but also about testing and deployment. This creates need to have dedicated environments for design, testing and production. Each of these environments translate to separate IT infrastructure – consisting of things like servers, network, storage and operating system. Two significant problems emerge. First, repetitive IT environment creation becomes serious bottleneck for fast-moving development cycles. Second, quality assurance becomes difficult if IT environments are not identical – code that worked fine in previous development environment suddenly crashes when taken to production. What’s worse, the two problems make each other worse. Clearly something beyond Agile is needed.

In terms of operating model, DevOps is the answer. DevOps signifies merging of software development and IT operations. It is a way to integrate sw development and IT as single tight-knit team. That goal is achieved by deploying all operating model components: organisational structure, processes and methods, and governance.

Still, this is not enough. For the final leap in software development productivity, we need help from technology platform.

Technology platform for software based agility

Cloud computing as agility cornerstone was discussed in earlier article:

In that article we learned that service model versatility makes significant contribution to software development productivity. Specifically, Platform-as-a-Service and serverless computing were identified as welcome additions to developer’s toolbox. Furthermore, containers provide very powerful way way to share physical computing resources and, perhaps more importantly, enable easy code portability.

From DevOps based agility perspective, there is one more key ingredient: Infrastructure as Code. With IaC, the IT environment is managed basically like software code is being managed. This is possible because cloud provides all computing and storage resources as on-demand service. Now developers and DevOps engineers can write scripts that provision prosessing resources as strictly defined environment – for development, testing and production alike. Furthermore, scripts allow all that to become automated. This is truly significant. When routine provisioning and configuration work is automated, DevOps productivity reaches new level.

Together, Agile, DevOps and IaC constitute product creation paradigm shift that is often referred as Continuous Integration and Delivery (CI/CD). Combined with automated testing, release cycle that used to be counted in weeks is now reduced to minutes. From strategic agility perspective, this is a game changer.

Architecture for software based agility

As discussed in Rule 2, for software development microservice architecture is the dominant game in town. When applications are created as suite of small services communicating thru well-defined APIs, architectual design contribututes to strategic agility – alongside with operating model and technology platform.

As a reminder, the key architectural principles to follow include:

• Each microservice is about business capability like Order rather than technical transaction like Database query
• Endpoints are made smart, pipes are kept dumb; that is, role of centralised resources is minimised
• Bounded Context is about allowing domain specific definitions for e.g. customer; Domain-Driven Design is the underlying design principle
• Architectural and operating model design go hand-in-hand with DevOps teams having ownership over handful of microservices

“Data is the new oil”

For many millenia, oil in the ground was mere nuisance or completely ignored. Today – for good or bad – world economy is dependent on oil. However, for oil to become the essential commodity for modern lifestyle, it took massive investments in the value chain from oil exploration to multitude of industries and applications. It took time, effort and human talent. Making data into powerful commodity is similar undertaking. The ambition level, effort and patience needed are about the same.

Technology evolution shapes customer value creation but it is data that leverages technology into productivity booster. Without data, technologies from IoT to AI/ML and from AR/VR to Edge Computing would be useless. The source of productivity boost is clear: Combined with data, technology increases customer value, improves customer experience and enhances operational efficiency.

However, the fact that data is everywhere and connects to everything is also its undoing – unless managed with strong vision combined with systemic and persistent data capability build-up. This constitutes significant challenge. Any company not willing or able to face the challenge can lead to only one outcome: Becoming marginalised and eventually fade away. That is the nature of data-driven disruption in the competitive landscape.

What makes data strategy good?

To make data into value source similar to oil, data strategy needs to be solid. Such strategy covers three complementary aspects:

• Diffrentiation thru products, services and solutions – Customer value and experience depend on connected product and digital service characteristics. Combined with new technologies, data has central role in making company offering a bit more lucrative than alternatives.
• Operational efficiency thru business processes – Innovative use of data for control and insight and for diagnostics and prediction brings quantum leaps in process efficiency across all industries.
• Data operations to organise and deploy data capabilities for maximum impact – Choices on operating model, technology platform and architecture have decisive impact on data’s value potential.

"What you measure is what you get" applies to data capabilities too

Strategic measures are used to determine whether data strategy is good or bad. More importantly, measures are used as strategic targets. “What you measure is what you get” applies to data and data capabilities too. Strategic measures fall into same three categories discussed above:

• Differentiation based on competitive profile viz-a-viz main competitors. The assesment needs to go beyond product and service portfolio to cover all customer journey touchpoints from brand recognition to purchase and delivery and customer care and relationship. The set measures need to focus on data’s impact in identified key areas.
• Measures on gains in operational efficiency drill into data-driven improvements in key business processes. Industry benchmarks are used to determine relative merits of each improvement.
• Strategic measures on data operations focus on three key themes: quantity, quality and time. Quantity is about sheer volume of usable data products (data solutions) emerging from data pipeline. Quality is primarily about low error rates but also more generically about usefulness and trust. Time is about average lead time. That is, how long does it take from opportinity identification to new data solution in production.

When setting strategic targets, current state in terms of organisational maturity must be observed. Setting the bar too high by attempting too much too soon is destined to fail. Regardless, data vision must be formulated keeping the end-game in mind:

The winners in data-driven disruption are the ones being able to organise their data operations for high volume, high quality and short time-to-market data solutions that differentiate and bring operational efficiencies.

Organising data operations

Virtually everything above concerning software development applies to data operations as well. After all, analytics is fundamentally about software code. Consequently, software learnings and practises are applicable to a large extent. For example, Agility is needed to deal with uncertainty and changes in customer needs and wants. Automation is needed to speed up integration and delivery. Infrastructure as Code is the tool for quality assurance when moving between IT environments. Data Architecture needs to enable parallel development, testing and production environments and smooth transition in between. And so on.

However, there are some data specific capability needs that are not necessary found on software development side. These include:

• Data Governance – Operating model practises to ensure data sourcing, availability and quality and to define data ownership.
• Center of Excellence – As discussed in the context of Rule 4: Operating Model (see below),?CoE as a centralised hub can be very useful in supporting distributed organisational units in business lines and functions and in various geographical locations. Early stages of data capability build-up benefit greatly from dedicated group of data specialists.
• Data Modelling – Business-driven data modelling, also known as Concept Modelling, provides foundation for alignment across various stakeholders. It excels in catalysting and facilating effective dialogue between business representatives (business and domain owners, subject matter experts) and the data team.

Strategic measures are used on evaluate the outcome of data operations build-up. Success is not defined by whether we have organisational functions or governance practises in place. Rather, the ultimate measure for success boils down to profitable growth – determined by differentiation for market share capture and by operational efficiency for low cost value creation.

Summary - What matters now

Software development and data operations are key sources for agility and differentiation. Although conceptually different, they share many similarities. In terms of productivity, software development has reached maturity but its data analytics sibling is just coming of age. It would be foolish not to take heed of older sister’s hard-earned wisdom.

The End Game in data-driven disruption is about speed, volume and quality. To reach this target - no matter how meager initial circumstances are - data capability build-up needs to be holistic, systematic and persistent. This takes solid change management skills. And it requires strong leadership. Without those, the risk of becoming marginalised or even obsolete increases rapidly.