What’s Up with Big Data? Let’s Look at the Trends

While there’s no question that the buzz of “big data” is still going strong, how well is big data actually catching on? To answer this question, we’ll review some of the recent research to see what’s trending in the world of big data.

To summarize, here are the four trends I’m seeing:

• The big data market is growing.
• The success rate of big data implementations is meh.
• People issues are trumping technology issues.
• Analytics is what keeps them coming back.

Let’s take a closer look at each of these trends.

Trend #1: The big data market is growing.

Wikibon, a professional community focused on solving technology and business problems, began analyzing the big data market in 2011. At that time, the worldwide market was estimated to be a little over $7 billion. Today, Wikibon estimates that the big data market is going to grow to over $61 billion by 2020 and $84 billion by 2026. This represents a 17% CAGR over the 15-year period starting in 2011.

What’s interesting is how the revenue breaks down by type. In 2014, Wikibon came up with these estimates:

• 40% is for professional services
• 38% is for hardware
• 22% is for software

The professional services revenue is notably higher for two reasons: (1) Many big data technologies are open source, which implies the software licensing is free and the vendor business models include more services; and (2) organizations need additional help with these new technologies—i.e., figuring out if and how to integrate big data into the business.

Now that we’ve established that the big data market is growing, how’s it playing out in organizations?

Trend #2: The success rate of big data implementations is meh.

In Capgemini Consulting’s recent report, Cracking the Data Conundrum: How Successful Companies Make Big Data Operational, guess what percentage of the organizations surveyed considered their big data implementation a success (with “success” being defined as most or all of the objectives of the implementation had been met)? The answer is 27%, or approximately 1 in 4 implementations. Surprised?

The report also showed that:

• 13% of the organizations had big data in full-scale production
• 35% were in partial production
• 29% were doing a proof of concept (POC)
• 24% had not implemented big data yet or had no plans to implement

Some would consider a 27% success rate “good” since we’re dealing with newer technologies, while a 1-in-4 chance of success could send others running. So the next question we need to ask is: Why isn’t the success rate higher? Or, in other words, what are the current barriers to implementation?

Trend #3: People issues are trumping technology issues.

This trend is interesting. For this one, I looked at two reports – the Capgemini Consulting report (referenced in trend #2) and TDWI’s just-released Best Practice Report, Hadoop for the Enterprise: Making Data Management Massively Scalable, Agile, Feature-Rich and Cost-Effective.

Both reports told the same story: About 2/3rds of the issues cited as barriers to success were focused on people (and process), and not necessarily the technology. Some of the common issues reported were:

• Inadequate skills or difficulty of finding skilled staff
• Lack of compelling business case – for funding and implementation
• Lack of business sponsorship
• Ineffective coordination of big data and analytics teams across the organization
• Lack of or ineffective data governance models for big data and analytics

You will probably agree that none of these issues are new to big data. We’ve been dealing with these data-related issues for a very long time. Big data is just amplifying these discussions.

So if the success rate isn’t that great (yet) and the barriers are becoming even more challenging, what’s drawing organizations to big data?

Trend #4: Analytics is what keeps them coming back.

The simple answer is big data analytics. Sure, how we store and process all this “big” data is important and it was a crucial nut to crack (thank you, Hadoop). But at the end of the day, if we’re not able to analyze this data and discover new and better insights that help our organizations increase revenues, reduce costs, improve operations, and/or better differentiate us from the competition, then we might as well pick up our ball and go home. 

In the TDWI report mentioned earlier, when survey respondents were asked about the benefits of implementing big data in their organizations, the top 5 responses were:

• Advanced analytics (data mining, statistics, complex SQL, etc.) – 48%
• Big data source for analytics – 45%
• Exploratory analytics with big data – 44%
• Good complement of our data warehouse – 37%
• Information exploration and discovery – 36%

So there you have it: Despite the challenges organizations are facing in this growing big data market, they’re (hopefully) in it for the right reasons—i.e., to discover new insights that will keep them in the game for many years to come.

Originally written for and published on Smart Data Collective as part of the Big Data MOPS Series.