Big Data - I like You! but it just isn't enough.

Big Data has quickly become a focal point fact for Fortune 1000 firms — such was the conclusion of a Big Data executive survey that I was fortunate enough to conduct and analyze with the help of my colleagues.

The objective of the survey was to gathers views from a small but prominent group of business executives such as chief information officers, chief data officers, and senior business directors and technology leaders of Fortune 1000 firms. Key industry segments represented are financial services, government, healthcare, retail, telecom and energy and utilities. Findings suggest that:

• Around 65% of firms now report having Big Data in production in 2015, 3% YoY growth
• Around 73% of firms acknowledge that Big Data is of prime importance to their firms
• Around 72% of firms expect to invest greater than $12 million in Big Data by 2017
• Around 24% of firms say they will invest greater than $50 million in Big Data by 2017
• Around 51% of firms say they have appointed a Chief Data Office

A few years ago, organizations and managers were struggling to understand and anticipate the opportunity and business impact of Big Data. While some executives thought data-driven analysis could transform business decision-making, other thought it to be the usual hype associated with any new technology. But now as I speak, Big Data is here to stay. Now, the focus is rapidly moving towards investments in Big Data technologies, Big Data adoption, the results it produces and the business capabilities it enables. When internet was a new phenomenon, we would ‘feel proud to get a chance to surf the World Wide Web’. Now we everybody does it! We will soon arrive at that same phase of adoption and maturity with Big Data.

So, how can organizations monetize and realize value from their Big Data investments?

Come up with the right set of metrics for your organization. While many of Fortune 1000 firms acknowledge investing into developing Big Data capabilities, few are clear on how to derive business value substantial investments over time. This is primarily because of the lack of the right set of metrics which, in turn, is a direct function of the relative immaturity of Big Data implementations and origination of the sponsorship of Big Data investments.

Define and isolate Big Data implementation costs. While Big Data has been acclaimed for enabling organizations to handle volume, velocity and variety, many firms struggle to measure the effectiveness of Big Data implementations owing to lack of clarity in defining and isolating their costs. This is because, usually firms follow an iterative process of loading data, identify associations and patterns, and then on board more relevant data which they think is beneficial to their business. This enables organizations to learn through trial and error as most organizations have faced a few false starts while trying to develop Big Data ecosystem to suit their business needs. Due to inefficiencies and immature processes, the initial investments of effort and time have sometimes been larger than anticipated. These costs can be expected to level off as experience and efficiencies grow over a period of time.

Identify opportunities for innovation. Big Data continues to promise Innovation. The speed and agility it permits lend themselves to discovery environments such as life sciences R&D and target marketing activities within financial services. Success stories of Big-Data-enabled innovation remain relatively few at this stage. To date, most Big Data accomplishments have involved operational cost savings or allowing the analysis of larger and more diverse sets of data.

For example, financial firms have been able to enhance credit risk capabilities through the ability to process seven years of customer credit transactions in the same amount of time that it previously took to process a single year, resulting in much greater credit precision and lower risk of credit fraud. Yet, these remain largely back-office operations; they don’t change the customer experience or disrupt traditional ways of doing business. A few forward-thinking financial services firms have made a commitment to funding Big Data Labs and Centers of Excellence. Companies across industry segments would benefit from making similar investments. But funding won’t be enough; innovating with Big Data will require boldness and imagination as well.

Cultivate cultural and business change. Though some large firms have invested in optimizing existing infrastructure to match the speed and cost benefits offered by Big Data, new tools and approaches are displacing whole data ecosystems. A new generation of data professionals is now emerging. They have grown up using statistical techniques and languages like Hadoop and R, and as they enter the workplace in greater numbers, traditional approaches to data management and analytics will give way to these new techniques.

The many challenges faced by organizations as they operationalize Big Data are more related to people, not technology: issues like organizational alignment, business process and adoption, and change management. Companies must come to terms with the fact that businesses cannot effectively adopt Big Data without cultural transformation.