Is The Big Data Bubble Deflating?

The picture I chose to illustrate this article comes from the #CDAO conference in Berlin 2019. I still had pictures of The Bourne Supremacy and Victoria floating in my head, two reasonably recent films partly filmed in the same venue, The Westin Grand, when the slide appeared on the screen. I did not have the pleasure of having a room at this hotel, but at a more modest AirBnB, staying with local people on Warschauer Stra?e, not quite as swanky as the conference venue, as I attended #MeasureCamp Berlin a few days later.

Reports of Big Data not delivering value were not new back then. NewVantage Partners reported every year in the Harvard Business Review similar stories: a minority of companies are data-driven, but they keep spending on data. Meanwhile, Big Data underwent many name changes, such as Analytics, Data Science and Decision Science. Companies started hiring Chief Data Officers (CDOs) due to the 2008 subprime crisis. Meanwhile, the average tenure of a CDO was 18 months, and even only 12 months in financial services.

Companies hired CDOs with a more defensive bent for regulatory compliance. GDPR came into force, and other countries followed with local legislation. The practice turned data into a cost centre and prompted a reactive shift towards CDOs able to deliver business value. A recent post by Eddie Short suggests that, although he has succeeded as one such CDO 2.0, companies either still prefer the original defensive style of CDO or are returning to them.

AI is prompting the hiring of defensive CDOs because of the increased risk exposure of companies. Regulators are catching up with AI and are adding additional regulatory burdens. Many companies want CDOs with hands-on Python experience. The issue with these CDOs is the inability to deliver business value; they are too focused on tactics and not enough on strategy.

But what is best? CDOs who cannot communicate with their data analysts, data scientists, and data engineers, like cavalry officers who have never seen a horse that Elon Musk lambasted, or CDOs with P&L experience, able to deliver on strategy and business value? What if the issue was splitting the tactical and strategic aspects of the role, and the ideal should be someone with both skill sets?

As I read the #Harnham salary surveys, a UK recruitment company specialised in data roles active in France, the Netherlands and the US, I discovered interesting findings. In the top 3 reasons why data practitioners leave their current role, independent of speciality or country, we find that the roles offer competitive salaries and no career progression. Year after year, it is always the same.

I have lost count of the times recruiters have told me that, despite my 15 years of work experience in data, data leadership roles are a field in their own right, so as a subject matter expert (SME), and I am white and male, so probably falling short of the diversity points that the company is looking for; I am automatically disqualified. I am not even surprised.

I have said before how all other data disciplines treat Digital Analytics as if in a jar on a shelf. Now, I am under the impression that the companies recruiting in data treat every data discipline as distinct jars, side by side. Within each jar, we see others in other jars across the glass and feel that there should be transferrable skills. But there seems to be no upward but no lateral mobility across data disciplines, keeping the recruiting under the impression that there is a talent shortage, but not really if they considered candidates from these adjacent data fields.

Right after the end of the pandemic, we saw how companies were playing catch up and suddenly had a hiring budget. With the latest Harnham survey, we discover how companies feel they might have overdone it and are now asking companies to put their data capabilities in maintenance mode. That does not sound too good.

So, it seems that companies are losing trust in data's promise to deliver business value and regulatory compliance at the same time. Companies are shunning CDOs with a proven track record of delivering both. However, they persist in hiring CDOs focusing on regulatory compliance and tactical experience. Could this be because companies prefer the status quo instead of letting data practitioners deliver business value? Cui bono??

I believe the C-suites fear letting data take the role it deserves in decision-making. At too many companies, letting this happen would result in a redistribution of power, influence and budgets. Some key people might leave rather than lose their current standing. In other words, companies view letting data influence decision-making as high risk, the status quo of data as a cost centre prevails, and a lack of career progression for data practitioners endures. IT has career progression because they do not challenge the power structure.

"The rumours of my death are greatly exaggerated", Mark Twain

At times, everybody cited Kodak and Blockbuster as examples of what awaits companies ignoring data and digital transformation. But today, most companies are certain that these were outliers who hit challenging times due to bad management rather than innocent victims of disruption. Most startups banking on data rather than brand recognition have failed like most startups do. There's Amazon, but it disrupted mom-and-pop shops rather than big companies.

As the famous quote attributed to Mark Twain goes, not the one about the internet, that's Abraham Lincoln, but the other one: "The rumours of my death are greatly exaggerated." I have been wondering how long the Big Data bubble would last. But that bubble may be popping when companies announce hiring freezes in Big Data, switching to maintenance mode rather than spending as they have in recent years.

After hearing how we cannot leverage our experience in Digital Analytics to work in Big Data, I wonder whether the deflating Big Data bubble would result in hiring companies tarring us with the same brush. Vendors promised companies that a Data Scientist would be able to replace their Digital Analyst and analyse data of greater variety, resulting in richer insight because their Digital Analyst, no matter how brilliant, could only analyse digital data.

Instead, we saw Data Scientists acting like the insecure specialists Steen Rasmussen talked about at MeasureCamp London 3 weeks ago. When Data Scientists focus on making their CV stand out, Digital Analytics does not cut. But sadly for them, so does business value, and now companies are holding them accountable. Meanwhile, Digital Analysts have signed up for inexpensive GCP, SQL, AWS, Python, R and many more courses.

We might witness what I described in articles I published earlier this year: a?donutification?of Big Data with companies using and needing fewer data scientists and perhaps even no CDOs but thriving in adjacent fields such as Digital Analytics, SEO, CRO and many more. Wouldn't it be ironic to see companies turning down data scientists applying for Digital Analytics because they, too, worked in Analytics?

https://www.dhirubhai.net/posts/eddies_data-and-analytics-leadership-is-at-crisis-activity-7119300214758195201-cZsR

https://www.dhirubhai.net/posts/blakejoubert_ive-spoken-to-3-chief-data-officers-in-the-activity-7117786408114606080-W2RO?utm_source=share&utm_medium=member_ios

https://hbr.org/2023/09/how-the-best-chief-data-officers-create-value

https://hbr.org/2021/08/why-do-chief-data-officers-have-such-short-tenures

https://www.dhirubhai.net/pulse/role-data-science-10-years-alban-g%25C3%25A9r%25C3%25B4me?trackingId=XX29OdAaTyyk4ClrsppCNA%3D%3D&lipi=urn%3Ali%3Apage%3Ad_flagship3_profile_view_base_recent_activity_content_view%3BWO6mkI2VQ46Et34tmQk0%2FQ%3D%3D

https://www.dhirubhai.net/pulse/ceos-secretly-against-letting-data-influence-decisions-alban-g%25C3%25A9r%25C3%25B4me?trackingId=eCF87GCXQ9K55JAE977C%2FQ%3D%3D&lipi=urn%3Ali%3Apage%3Ad_flagship3_profile_view_base_recent_activity_content_view%3BWO6mkI2VQ46Et34tmQk0%2FQ%3D%3D

For more articles like these, follow me on LinkedIn or X: @albangerome

#DigitalAnalytics #DataScience #CDO #WAWCPH #CBUSWAW