Every company is (or will become) a data-driven company!

In the sixth edition of the newsletter, I shared my views on how digital transformation can help enhance the operating model of an organization. This included an approach to identify good use cases of digital interventions - for automating, augmenting, and advising various tasks/ activities.

In this edition, we delve deeper into how digital transformation can help 'advise' companies in making the significant shift to data-driven decision-making.

A key consequence of increasing digital adoption is the proliferation of data across the enterprise. An oft-repeated quote is that "Data is the new Oil!". In the chart below, we zoom into the analogy, and consider the parallels.

The oil and gas value chain

The oil and gas value chain is broadly divided into - upstream (exploration and production), midstream (transportation, storage, and trading), and downstream (refining and retailing). Each stage of the value chain has its unique nuances:

• Upstream companies: Upstream companies deal with significant fragmentation - there is a very large landmass (and seabed) with possible potential, but relatively few fields that can be viably developed. This presents a very high degree of risk due to the very low probability of success. However, proven oil and gas fields generate significant value. So, these companies spend significant amounts in exploring and developing oil and gas fields and are richly rewarded for the risk and reward potential. Some of the most valuable companies in the world are upstream oil and gas companies including Saudi Aramco, ExxonMobil, Chevron, Royal Dutch Shell, and BP.
• Midstream companies: These are intermediaries involved in the storage, transport and trade of oil and gas between global spread-out producers and consumers. Midstream companies are relatively insulated from market risk such as demand/ supply and pricing as they deal with long-term contracts. However, the need to manage multiple operational parts including ships, trucks, ports, storage depot, containers - means they must contend with significant complexity and operational risk.
• Downstream companies: Downstream companies have an interesting mix of refining and retailing companies. Refining companies transform raw crude into usable products like gasoline. They have long term contracts and predictable operations, leading to stable refining margins. Retailing companies package and sell the finished products. They need to manage customer preferences/ demand variations as well as competition - and consequent impact on profitability. Retailing companies can use a combination of product, distribution and branding to generate significant value - and capture data and insights to help this objective.

Transforming Data to Value: The data value chain

The data value chain in most organizations can be considered along the same lines as the oil and gas value chain discussed above.

• Collecting Data - Most organizations collect data across their operations. In some cases, this is structured data captured in ERP, CRM, and other such tools implemented in the organizations. In many cases, they also have semi-structured data generated in daily operations which is available in emails/ attachments/ external sources. Finally, organizations also have access to immense unstructured data from channels/ operations/ customer service. The challenge at this stage is like that faced by upstream companies - significant fragmentation. The sheer variety, volume, and velocity of available data (frequently called big data) - can be daunting to manage. While several organizations are implementing databases/ data lakes - they usually only consider structured data. While this is a good start, not harnessing all available and useful data in the age of Artificial Intelligence and Generative AI will start to impose a significant opportunity cost in terms of potential value creation. Leading companies that recognize the challenge and opportunity start the data journey with a Data Strategy exercise. Data strategy creates a 360-degree view of data for an organization, to ensure future readiness and alignment to the business strategy. A Data Strategy is supported by Data Engineering, which operationalizes the data strategy by designing appropriate data storage and connectors.
• Organizing Data - As previously mentioned, organizations grapple with managing large volumes (size of data), variety (structured, semi-structured, unstructured), and velocity (always-on/ continuous data streams) of data. While Data Engineering can consolidate all this into a single data lake, most of this is not ready to be used meaningfully. The first step companies undertake is Data Governance. Most organizations struggle with poor quality data - this is due to lax checks/ controls at the time of data capture. A Data Governance exercise reviews the quality of available data. This is then followed by establishing processes and ownership for all new data creation/ capture, ensuring clean data on a go-forward basis. Cleansing of existing data is a tricky exercise, which is performed manually (for complex/ variable data) or through algorithms/ automation (for cases of missing/ minor deviations in data). The next step is Data Modeling, which identifies available data elements, defines data tables/ hierarchies, and establishes data relationships. Data Modeling helps consolidate the data expanse into manageable and meaningful data sets. Like midstream companies, this is a crucial intermediary stage that deals with significant complexity.
• Thinking (using Data) - The next stage after data modeling is Data Science. Data Science like the downstream refining activity, which transforms raw data into a useful form. Essentially from "Data to Insights". Data Science identifies trends, patterns, and opportunities to unlock/ create value for organizations. This is usually performed through statistical analysis of data, combined with industry and functional expertise to distill meaningful insights. The toolkits available for statistical analysis are predictable (like refining companies) and are being democratized as well as accelerated thanks to Artificial Intelligence and Generative Artificial Intelligence.
• Using Data - The final stage is to use the data to generate value, where we translate "Insights to Action". This stage is like the downstream retailing activity where the value chain culminates, in the hands of customers, which is decision-makers in the case of data. The key enabler here is Data Visualization. Visualization spans the spectrum from descriptive (post facto) to predictive (forecasts/ suggested actions), and is delivered through reports/ dashboards/ alerts/ triggers. With a variety of modern tools available, leaders and decision makers are increasingly getting actionable insights on demand.

Getting Started

In this newsletter, we introduced several important concepts in the Data value chain - Data Strategy, Data Engineering, Data Governance, Data Science and Data Visualization. Together, these stages transform information overload into actional insights, unveiling hidden gems from data. In future newsletters, we will unpack each of these elements in greater detail. There's a popular Chinese proverb that says: “The best time to plant a tree was 20 years ago. The second-best time is now. So is the case with organizations and data!

"Those who rule data will rule the entire world" - Masayoshi Son, CEO, Softbank

#genai #artificialintelligence #technology #digital #TechTonicThursday