Why Is It So Hard to Become a Data-Driven Company?

Thriving as a mainstream company today means being data driven. Companies that have lagged on this front have observed their data-driven competitors seize market share and make inroads into their customer base over the course of the past decade and pioneers like Amazon, Facebook, and Google develop dominant market valuations. Now, mainstream Fortune 1000 companies are fighting back by investing heavily in data and AI initiatives to narrow the gap. For the third consecutive year, investment in data and AI initiatives has been nearly universal, with 99.0% of firms reporting investment in data and AI according to findings from a newly released executive survey from NewVantage Partners, a strategic advisory firm that I founded in 2001 to advise Fortune 1000 companies on data leadership issues. But this year, despite growing investment, it appears most companies are struggling to maintain momentum.

This is the ninth annual survey of C-suite executives representing mainstream companies, that is focused on the progress of Big Data and Artificial Intelligence (AI) initiatives within these firms. For the 2021 report, we surveyed 85 Fortune 1000 and industry leading firms — the highest rate of participation and representation since the survey was first conducted in 2012. Participating firms encompassed blue-chip leaders in financial services, life sciences, healthcare, and retailing, including American Express, Anthem, Bank of America, Bristol-Myer Squibb, Capital One, Cigna, CVS Health, Eli Lilly, Glaxo Smith Kline, JP Morgan Chase, Liberty Mutual, Mastercard, McDonalds, Merck, Pfizer, Sanofi, Starbucks, United Health, VISA, and Walmart. This year, a record 76.0% of respondents held the role of Chief Data Officer or Chief Analytics Officer. Survey respondents comprised the most senior corporate executives with oversight and responsibility for data within their firms.

After nearly a decade after of performing this survey, we’ve noted two significant trends. First, mainstream companies have steadily invested in Big Data and AI initiatives in efforts to become more data-driven: 91.9% of firms report that the pace of investment in these projects is accelerating, and 62.0% of firms reporting data and AI investments of greater than $50 million. Second, findings from this year’s survey suggest that even with record levels of committed investment, firms are continuing to struggle to derive value from their Big Data and AI investments and to become data-driven organizations. Often saddled with legacy data environments, business processes, skill sets, and traditional cultures that can be reluctant to change, mainstream companies appear to be confronting greater challenges as demands increase, data volumes grow, and companies seek to mature their data capabilities.

In the 2021 survey, Fortune 1000 companies reported a decline in the leading metrics which are used for measuring the success of their data and AI investments. Companies reported struggling to make progress — and in many cases even losing ground — on managing data as a business asset, forging a data culture, competing on data and analytics, and using data to drive innovation. Only 29.2% report achieving transformational business outcomes, and just 30% report having developed a well-articulated data strategy. Perhaps most tellingly, just 24% of respondents said that they thought their organization was data-driven this past year, a decline from 37.8% the year before — a figure that suggests that efforts to incorporate data into decision-making processes were not as successful as leaders had previously believed.

What’s at the root of this slow progress? For the fifth consecutive year, executives report that cultural challenges — not technological ones — represent the biggest impediment around data initiatives. In the 2021 survey, 92.2% of mainstream companies report that they continue to struggle with cultural challenges relating to organizational alignment, business processes, change management, communication, people skill sets, and resistance or lack of understanding to enable change. This represents an increase from an already high percentage of 80.9% of firms that named cultural challenges as the greatest impediment to success just four years ago. As management guru Peter Drucker once said, “Culture eats strategy for breakfast.”

This cultural resistance was not inevitable, but it was perhaps foreseeable. Nearly a decade ago in 2012, Eric Brynjolfsson and Andrew McAfee wrote in Harvard Business Review that Big Data would be a management revolution. So why, after nearly a decade of investment in data initiatives, are firms continuing to struggle in their efforts to become data-driven? One answer is that becoming data-driven takes time, focus, commitment, and persistence. Too many organizations minimize the effort or fail to correctly estimate the time which these kinds of wholesale business transformations require.

Given these findings, Chief Data Officers and corporate data leaders should consider three pragmatic recommendations:

1. Organizations can benefit by focusing their data initiatives on clearly identified high-impact business problems or use cases. By starting where there is a critical business need, executives can demonstrate value quickly through “quick wins” that help a company realize value, build credibility for their investments in data, and use this credibility to identify additional high-impact use cases to build business momentum. We see firms that invest in data capabilities and technology without a clearly defined business demand failing time and time again.
2. Companies must reexamine they ways that they think about data as a business asset of their organizations. Data flows like a river through any organization. It must be managed from capture and production through its consumption and utilization at many points along the way.
3. Data-driven business transformation is a long-term process that requires patience and fortitude. Investments in data governance, data literacy, programs that build awareness of the value and impact of data within an organization, may represent an eventual step in the right direction, but organizations must show that they are in it for the long haul and stick with these investments and not lose patience or abandon efforts when results are not immediately forthcoming.

While there were apparent setbacks in the last year, it’s worth noting that companies have made great progress over the course of the past decade. When this survey was first launched nearly a decade ago, mainstream companies had just begun to embrace Big Data and its expected transformational impact: Big Data and AI were nascent capabilities, which received minimal investment; few companies had developed formalized programs and articulated a corporate commitment; and the function of a Chief Data Officer was non-existent, except within a small handful of companies. In the intervening years, the percentage of organizations reporting the appointment of a Chief Data Officer has increased from 12.0% in 2012 to 65.0% in 2021. Interestingly, 81.0% of executives surveyed this year indicated that they an optimistic outlook for the future of data and AI within their firms. There is no question that Big Data has been absorbed into the mainstream.

It may also be true that as companies gain in sophistication, they have come to appreciate that they had initially overestimated the degree to which they were achieving data transformation milestones. Today, armed with greater understanding and sophistication, resulting from a decade of investment, many companies are taking a more self-critical view, realizing that to successfully compete on data and analytics, they must raise their game even higher. It is no longer sufficient to be good enough. The bar continues to be raised by their data-driven competitors.

What this year’s survey results tell us is that even with the progress that has been made in terms of investment and the establishment of data leadership within mainstream companies, significant challenges remain. What has been underappreciated is how much time it will take for these data-transformation efforts to take effect at mainstream companies, as they remain saddled with legacy systems, legacy cultures, and in some cases, legacy skill sets. Becoming a data-driven organization does not happen overnight. Building a data culture is a process. These efforts unfold over time. Today, Big Data and AI are mainstream, but there is still much work to be done.