Looking for Value in Unstructured Fresh Data ?

I recently came across more industry figures regarding the proliferation and growth of data. Most companies today are busy amassing huge lumps of data, whose primary objectives typically fit into a short list.

1. Understand customer purchasing to optimise sales activities: How do we know what incentives are driving our customers to purchase new products or services from us? What marketing efforts and activities are driving the best conversion rates and when? Similarly, how do we know what is driving up-sell and cross-sell?
2. Building a holistic view of our customers (360’): To build an understanding of each customer’s profile, behavioural attributes, preferences, pain points, and lifestyles in order to optimize our product or service experience?
3. Creating a rounded view of our products, ecosystems and employees: To ensure that the greatest visibility into the performance or productivity of our products, infrastructure, and/or employees is optimised and maximised.

Standard things really, where CRM , Multi-channel, HR, Supply Chain, Distribution, Finance and many other systems & teams are busy beavering away clusters of (mainly) structured data into data warehouses, data banks and clouds, respectively. To date, the Big Data challenge for most executives has been about organizing and managing greater varieties of historical data – understand the past in trying to predict future actions. Historical data is, however, rather static data.

Then along comes another big source of data, which should make us think it is going to get trickier. Because all evidence points to it becoming more established & more integrated. Of course, I am referring to Internet of Things (IoT), generating a plethora of additional data from sensors and smart devices as they become more cost effective and prevalent.

According to the EMC/IDC Digital Universe Report, data is doubling in size every two years. From 2013 to 2020, the digital universe is projected to grow from 4.4 trillion gigabytes to 44 trillion gigabytes. For executives faced with the age-old challenge of how to organize and learn from the data, to drive differentiation and ultimately improved customer experiences, the clock is running.

Catch Customers in the moment

Let’s consider a traditional example of the personalized app from Starbucks. Progressing from emails offers based on preferences and tastes, the strategy has been to extend the mobile app and provide relevant suggestions and recommendations, while in store. The aim is to catch customers in the moment so that messaging becomes more targeted and more personalized. The plan then evolves to add more features to the service, such as the ability to store customers’ favourite stores, orders, recommendations and redeem rewards. All of this is often referred to as “situational awareness,” which means that by having real-time sensitivity to customer preferences at the moment of purchase, a business may be able to tailor, adapt or customize an offer to best fit a customer’s needs at the moment of purchase.

With 280K recently added new Starbucks customers and nearly 10 million transactions per quarter on this service, we can imagine the heightened 2-way data volumes here. Such digital investments resulted in moving the Quarterly Revenue needle up 9% on a multi-billion-dollar business. Pretty good going.

As devices, apps, sensors, scanners, smart phones, become more intelligent, it fuels data proliferation. Today, according to the EMC/IDG study, consumers and workers generate two-thirds of all new data. This is about to change. Some say that within this decade, most new data will be generated not by people — by connected devices. Health care providers are employing sensors to monitor patients in real time, enabling physicians to more effectively diagnose disease and prescribe treatments. Financial services firms are able to monitor and manage risk and detect fraud. Likewise, professionals are using basketballs with 200 embedded sensors to monitor performance, and so are rugby players with wearable GPS trackers and cycling teams with pedal sensors.

Value-Add with Fresh Data

It is becoming now possible for companies to know what is happening as it is happening, and take the right responsive decision in real time. Data is dynamic and exists on a time continuum. When understood in this context, it is easy to appreciate why data can have great impact when it is fresh. As the retail example above, knowing a customer’s preference during a purchase can be significantly more valuable than knowing a customer’s preference after a purchase.

Amazing new opportunities are unfolding every day. Whether structured (e.g., form fills, surveys) or unstructured (e.g., comments, pins, tweets), marketers now have access to more data than ever before. Where marketers were once able to break down the data into smaller datasets and analyze the numbers manually, it’s becoming increasingly more complex. Enterprises are now turning to data scientists to make sense of it all. And data scientists are turning to machines.

With machine learning, data from millions of social media posts can be analyzed in ways that humans simply can’t. Instead of looking at demographic activity (e.g., how many posts, by whom, from where), data scientists can uncover insights into behavior and what’s actually being said in the millions of comments. For example, the tool Twitrratr helps companies with real-time sentiment analysis of positive, negative and neutral tweets. Gatorade and Dell have ‘media command centres’ to track social media developments about their products, and can make real-time changes to promotions and messaging.

MIT Computer Science and Artificial Intelligence Labs believe new data architectures are required to ensure the potential benefits

Industry examples are currently focusing on continuous monitoring, adjusted to automatically course-correct if needed, increased efficiencies, and risk mitigation with cost savings. It is one thing using cloud storage, SQL DB like processing solutions and very large databases. However, the requirements of fresh data imply an ability to ingest, analyze and interact with vast streams of incoming data in real time, requires distinct Hadoop like data architectures, Apache Spark like engines with “in-memory” data management approaches.

Today’s businesses are broadening the data landscape by introducing the newer forms of data — unstructured content, social media data, voice and now sensor data — that can be integrated with traditional forms of historical transaction data to provide a fuller business picture. So, what does the next great march of data and insights look like, feel like, taste like?

How does that affect future plans?

2016 is here and while the last few years have already been laden with heightened analytics, big data mngt and omnichannel processing, the near future will require smarter manoeuvring to address the barrage of even more data. A fuzzy smartish crystal ball is pointing to the following considerations:

1. Medium Data, Big Data, Bigger Data – it does not really matter. With a long list of failed Big Data projects out there and an even larger list of big Data-Warehouse projects before that, the focus has to be on agility with that data - Real Time Processing, in-memory Management, accelerators and so on.
2. Continued Search for more Data analysts and Data Scientists - reporting backlogs are growing. While some technology is catching up, we are still relying on teams of analysts for ‘effective story telling’. According to McKinsey, in 2018 there will be a shortage of 1.5M data experts. Combine this with the growing branches of text, speech and video/image analytics and the delays for typical business managers awaiting their updated reports, mount up. In fact, Gartner says that the number of non-technical folks who want to experiment on their own with data, will grow 5 times faster than the number of expert Data Scientists. Build your teams!
3. Data Sources, Privacy, ownership, Security and IP – with more data, the emphasis on transparency, responsibility and ethics becomes ever greater. Dedicated teams to govern these, plus privacy, consent, security, IP etc. is no longer a nice to have. Of course, anonymization techniques can help address concerns over individual disclosure – but it needs to be baked in from the start of the processes.
4. More Cloud Computing and Cloud Storage – consider this: WAN bandwidth continues to be bought and sold in megabits, while data volumes are approaching petabytes. Many would argue that moving this amount of data to the cloud is not practical, let alone secure.
5. Machine Learning is here to stay – most importantly. ML has been effervescing of late. It is going to get more sophisticated, we need to embrace it, operate it effectively, deliver prompt value. The irony here is that while the volumes of data generated by devices (not people) swells, the ability to process and work with it quickly will be dependent on more machines (not people).

Unstructured data is growing in volumes and significance. Already, some firms are mining social media likes and recommendations to yield insights on user preferences and personalities, which users themselves have not explicitly indicated in their online profiles. Hotel chains augment customer insights by tracking posts on Tripadvisor and tracking certain trends. The examples just keep popping up.

Depending on circumstance/sector, there are naturally a couple more considerations for teams managing these opportunities, as their journeys unfold.