The Power and Paradox of Data

No one would argue that data has become the world’s most valuable resource. For businesses, data offers a direct path to understanding customers better, refining operations, and outpacing competitors. But beyond the value of data itself lies a deeper question: What data does a company really need? And, how should it be collected, interpreted, and ultimately used?

A historical perspective

A vivid example of data’s transformative power goes back 30 years. In a LinkedIn post, Bill Gates shared a memorable image from the early days of Microsoft when his team sought a unique way to showcase the power of data storage. Back then, a single CD-ROM could hold the contents of an entire encyclopedia. "Thirty years ago, we wanted to show just how much information a single CD-ROM could hold. The team decided a visual demonstration was necessary!" Gates recalled in the post. See Gates' LinkedIn post and picture here.

Today, in just a few seconds, a modern server can process data thousands of times more complex and vast than Gates' CD-ROM could hold. According to Forbes and IDC, global data volume is expected to grow 5 times from 2018 by 2025, underscoring the accelerating role data plays in shaping the future of industries worldwide. IDC says that China’s Datasphere is expected to grow 30% on average and will be the largest Datasphere of all regions by 2025.?

Neil M. Richards and Jonathan H. King, in their paper Three Paradoxes of Big Data, noted that big data analytics rely on “small data inputs, including information about people, places, and things collected by sensors, cell phones, click patterns, and the like.” They explained that these small data inputs are aggregated to create large datasets, which are then analyzed for insights. “This data collection is only accelerating.”?

The business of data: insights and strategies

According to IoT Analytics, there were 16.6 billion connected Internet of Things (IoT) devices by the end of 2023, marking a 15% increase from 2022. Experts from IoT Analytics anticipate this figure will grow by 13% to reach 18.8 billion by the end of 2024. This forecast is lower than in 2023 due to continued cautious enterprise spending, as inflation and interest rates remain high.

The Big Data and Business Analytics Market was valued at USD 245.9 billion in 2023 and is expected to grow at a compound annual growth rate (CAGR) of 15% from 2024 to 2032, according to a report by Global Market Insights. Experts estimate that the market value will reach USD 861.7 billion by 2032.

Data has become the lifeblood of digital transformation. Paul Shanahan, EMEA Digital Sales Director at Microsoft, captures the sentiment well, stating, “New technology has given us the ability to access and interpret data in new ways. The trick is to identify strategic goals and the right infrastructure investments to make it work for your business.” This mindset underscores the importance of aligning data initiatives with overall business objectives, as harnessing data effectively can lead to stronger strategic insights and more agile decision-making.

McKinsey offers actionable insights into this challenge, suggesting that businesses should treat data as a product to unlock its full potential. As McKinsey outlines, “Data products incorporate the wiring necessary for different business systems, such as digital apps or reporting systems, to ‘consume’ the data. Each type of business system has its own set of requirements for how data is stored, processed, and managed; we call these ‘consumption archetypes.’” By tailoring data products to these archetypes, businesses can streamline the use of data across applications, ultimately maximizing its utility. According to McKinsey experts, the benefits of this approach can be significant: new use cases can emerge 90% faster, technology and maintenance costs can be slashed by as much as 30%, and companies can reduce both governance burdens and data security risks.?

Challenges in the era of data-driven economies

While data holds transformative potential, it also presents a series of challenges. Businesses differ significantly in how they leverage data, often influenced by the economic and regulatory environments they operate within. Issues such as the ethical collection and use of personal data are highly sensitive topics. Consumers deserve transparency and safeguards; they want to know not only how their data is collected but also how it is protected. Moreover, the question of whether companies should share certain types of data with competitors raises complex debates around both ethics and innovation.

Data is undeniably the cornerstone of the digital economy. But with data’s great power comes the responsibility to use it thoughtfully, balancing privacy with progress. A Norton survey reveals that nearly 80% of internet users are concerned about the misuse of their personal information. This highlights the need for secure and anonymous data collection practices. While more than three in five consumers are willing to accept certain risks to their online privacy for greater convenience, four out of five still express concerns about data privacy. Additionally, seven in ten individuals have taken steps to protect their online privacy.

As we look ahead, innovations like digital avatars may allow us to interact in a more secure, yet data-rich environment, protecting personal identities while enhancing digital experiences. But this evolution poses new questions: Will the authenticity of human experiences be preserved, or will we be guided primarily by predictive algorithms? The answers lie in how businesses choose to handle the data-driven revolution responsibly, with a balance of innovation, security, and ethics at its core.

Daniel Griffin from Cisco marking a modern industry issue: most data driven companies aren’t as data driven as they think they are. He describes what ‘The Big 3’ questions companies should care about.?

Griffin observes that “nearly all data science in industry today focuses solely on answering the initial question: what is happening, or what will happen?” As noted in the Cisco review, “Most data scientists understand supervised, unsupervised, and semi-supervised learning revolves around answering what is happening or what will happen.” Even with applications like product recommendation systems which may sound prescriptive because of terms like ‘recommend’ all we actually know is what products a customer is interested in. This indicates interest, but not the reason behind it. We’re left with uncertainty about the most effective way to act on this information. “Should we send an ad? Should we call them? Do certain engagements with them cause a decrease in their chances of purchase?” Griffin argues that answering these ‘why’ questions requires foundational work in causal inference, a field developed by researchers specifically to understand the causes behind observed behaviors.

And as business and data science are still only at the first stage of understanding, the threat to personal privacy remains relatively limited for now.?