Big Data - Taking Steps to Create Solutions

The amount of data being collected is so massive that it can boggle the mind of a mathematician. Facebook is an easy example to look at due to the amount of traffic and the way the platform is utilized in peoples lives. Here are some of the recent numbers shared by Facebook:

•     2.5 billion content items shared per day (status updates + wall posts + photos + videos + comments)
•     2.7 billion Likes per day
•     300 million photos uploaded per day
•     100+ petabytes of disk space in one of FB’s largest Hadoop (HDFS) clusters
•     105 terabytes of data scanned via Hive, Facebook’s Hadoop query language, every 30 minutes
•     70,000 queries executed on these databases per day
•     500+terabytes of new data ingested into the databases every day

Yes, mind numbing numbers, and as Jay Parikh, VP of infrastructure at Facebook stated, “If you aren’t taking advantage of big data, then you don’t have big data, you have just a pile of data.”

Collecting the data is one side of the equation, data storage companies are growing by leaps and bounds and as a result, the need to house the massive amounts of data (investment opportunities perhaps?).

Discovering ways to use big data can be quite intimidating! And represents the other side of the equation. Especially to business owners whom are working hard, day in and day out, just to keep their business on the growth track.

Digging In

Much of the data can be fairly useless, depending on your business model. Overall, about 22% of big data is useful to businesses today and this number is growing quickly. According to the IDC they expect to see that number jump to 37% by 2020. Furthermore, IDC states that there will be 44 trillion gigabytes of data, ten times the amount there is today, by 2020. In an early post I mentioned Cisco’s new term Zettabytes (1 sextillon bytes), and that big data will be measured by those terms in the coming year or two. 37% of that 44 trillion gigabytes being useful to business owners is a flag to pay close attention to.

The useful percentage of data is what is commonly referred to as big data, and the big companies, like the example of Facebook above, mine big data to better serve (supposedly) their users by generating more relevant advertising. Other examples include:

American Express: Uses big data to forecast customer loyalty.

Macys: Uses big data to adjust pricing in near-real time for over 73 million items (I had no idea they had so many).

Tipp24 AG: Tipp24 a platform for placing bets on European lotteries, and prediction, uses big data to analyze billions of transactions and hundreds of customer attributes to send personalized marketing messages on the fly.

There are many examples of what big data is doing today.

Big data is quickly becoming a staple of business, benefiting organizations in countless ways, opening up a generous amount of engaging possibilities.

The examples above are amazing, however big data is the "horse" pulling the "cart". Analytics and predictive modeling is the "cart". Focusing the data in a manner which is pertinent to your KPI’s is the "reward" in the cart. Businesses big data needs vary and are based on the problems they want to solve. 

We know what the "horse" (big data) is, however the "cart" is variable and based on the problem you are looking to solve. Several companies have been created that facilitate the analysis and generate predictive modeling of big data in different ways. There are several companies that have been born out of this very need. For scale and complex data I would lean towards a third party solution.

Third Party Solution Examples

Fleetmatics: Fleetmatrics is a GPS tracking solution. Their $1 billion (market cap) business provides companies with small boxes that fit under the dashboard of every vehicle in a company’s fleet. The box tracks miles driven, location and speed amongst other items. It works with a fuel card that tracks when and where drivers fuel up. Combining all these different data points allows companies to track their mobile workforce and measure how effectively the fleet is working. Fleetmatics also uses that data to help companies that are congruent in size to a current customer providing cost analysis and guidance.

Saama: Saama has over 17 years experience and a depth in their offerings that is sure to solve problems from companies at the enterprise level to health care to consumer packaged goods. They offer big data analytics solutions for a variety of enterprise level organizations ranging from health care to government agencies. Saama has shown immediate value by helping them fast track product launches, streamline supply chain, optimize forecast demand, improve marketing strategies and transform their businesses into real-time, dynamic, customer and solution-centric enterprises that are driven by predictive modeling and actionable insights. Business Intelligence Center of Excellence and Social Governance, Risk and Compliance are amongst their many other offerings.  Saama is five stars in my book.

Microsoft Power Business Intelligence: which lets businesses analyze data from Excel spreadsheets. Visually explore your data through a free-form drag-and-drop canvas, data visualizations, and a report authoring experience.

Build Your Own

If you are seeking to build out your own solutions, which may be best for small businesses. Software such as Hadoop and databases solutions such as NoSQL put data analyzing at the fingertips of the tech-savvy.

The Approach
The way to approach big data is: Big data is a tool that solves problems.

There are a few fundamental steps that you can follow to assist in the process:

Step One

Know the problem you want to solve.

If you know the problem you want to solve, there is expertise and a solution out there.

Is it lead generation? Creating brand evangelists? Tracking movement? Adjusting Pricing in real time? Tracking movement to improve performance and drive down costs? Driving conversions?

Once you know the problem you can start to use data to find a solution. Once you know the problem you can do research on how best to solve it using data.

Step Two

Think it through.

Think about the use of big data as if you are a scientist creating a hypothesis. It is needfully important that everyone in the company that is involved is on the same page, an agreed upon hypothesis is the foundation for each team member seeing the data in the same way. Otherwise team member may walk away with a different interpretation of the data analysis, once presented.


Step Three

Choosing the right data.

There is an insatiable demand for more detailed data. More detailed data creates more to sift through. There are key data points that will address your problem (in Step One). Ensure that you have scrubbed all the data points that are congruent to your problem.

Step Four

Start small and grow.

In almost every case, conduct a trial analysis of the data to discover if it helps solve your problem. If the trial analysis is a failure you haven’t risked much in terms of time or money. If it’s a success you’ll come out with information you can use and a path for future big data research.

Step Five

Be agile.

Utilizing big data is much more than analyzing the information, it’s about executing action based on your findings. If you cannot act in real time, all you will discover is how much money you are losing. That is a crucial piece through business, from sales to marketing to information technology, get everyone on the same page to get the most out of the data.

See the forest instead of the trees.

As I mentioned earlier, big data can be intimidating. You may thin it’s unaffordable or that it takes too much time. Consider this, big data, once again, the horse, and your analytics team, the cart, will provide you with the information you need to make more effective decisions and execute strategies that move the bar graph up a notch. As case study after case study illustrates, big data and analytics are creating a better customer journey, creating more brand evangelists, and refining spend amongst many others.

Bottom Line

Big data, analytics and predictive modeling present business and organizations with benefits and insights that evolve themselves in ways that our culture has not begun to imagine and is as essential to a company as water is to life.

Those who adapt and adopt, flourish.