CEO Diaries : BIg Data & Predictive Analytics in Retail

"Saal Ke Sabse Saste Din" , " 1947 Independence Day Special , "Black Friday" or "Cyber Monday" have come to signify the biggest"artificially created - non festive" shopping events of the year , driving hordes of excited shoppers to the stores,from 4 am in the morning, waiting in a queue for 5-6 hours for that once in a year mega deal. But what many people may not realize is that this period signifies the approximate time on the calendar when many retail businesses move from operating in the red and start to actually make a profit for the entire year. Despite creating a hype of "unbelievable deals", retailers are able to create a strong contribution to the bottom line , owing to "Big data based predictive analytics".

So what's happening with Big Data and the retail market? Web sites, GPS-enabled tablet devices and smart phones, and embedded sensors -- all increasingly connected using mobile technology -- are generating massive amounts of data about consumer behavior. For the first time, it is also both technically and economically viable to store and analyze this data to reveal new insights and patterns. Big Data-savvy retailers are collecting and mining this data to target customers on a more personal and direct level, especially during during this critical holiday season.

How are retailers using Big Data this year? Analytics that can handle enormous volumes of diverse data are being deployed to perform closed-loop analysis on a wide-range of activities, including effectiveness of marketing campaigns, customer online buying behavior, performance of sales promotions, social-commerce and inventory optimization.

For example, Big Data analytics make it possible for retailers to directly correlate consumer Web activity with promotions and marketing campaigns, and track resulting sales transactions. And as a result, retailers can monitor and tweak promotions and campaigns in near real-time to maximize spend, increase profitability and generate revenue during this short, but critical period of time. They do this by quickly slicing and dicing terabytes of data, including millions of daily emails, every click on Web sites, and every ecommerce and brick and mortar transaction.

These advanced analytics enable retailers to perform deep, precise customer segmentation by demographics, such as age and income, and psychographics such as interest and lifestyle profiles -- segments which are then used to drive highly optimized and personalized offers and campaigns. 

The need for time-sensitive Big Data analytics is leading to the rise of the latest trend of "self-service" Big Data analytics. Data analysts are able to answer their own business questions using Big Data sources in minutes, without needing to wait weeks or even months for their IT department to provide them the data -- too little, too late. This is especially essential when organizations are using the latest and most cost-effective and powerful Big Data platforms such as Hadoop.

An example of Big Data analytics in action is Jewelskart an online seller of high fashion jewelry, clothing and shoes. JK is an extremely savvy social commerce company with personality driven sites that generate massive amounts of data. It tracks all this data, including every click on its Web properties, every click-through on its marketing emails, and every record of each sales transaction. For JK, Big Data analytics has been a game changer for customer acquisition, as they now have easy access to analysis that enables effective cross-selling opportunities and a sophisticated referral engine.

Ideally, a retailer’s customer data reflects the company’s success in reaching and nurturing its customers. Retailers built reports summarizing customer behavior using metrics such as conversion rate, average order value, recency of purchase and total amount spent in recent transactions. These measurements provided general insight into the behavioral tendencies of customers.

However, reports summarizing average behavior don’t provide the useful insights needed to determine how individual customers are likely to behave because general behavior tendencies are simply too broad. In order for retailers to create a meaningful dialogue with customers that honors the shopper’s preferred level and mode of engagement, it takes more than summarized reports, which is why customer intelligence and predictive analytics provide the opportunity to significantly change the retail marketing industry.

Customer intelligence is the practice of determining and delivering data-driven insights into past and predicted future customer behavior. To be effective, customer intelligence must combine raw transactional and behavioral data to generate derived measures.

But how do data scientists and predictive modelers determine which derived attributes are relevant? Usually data scientists lack the deep domain expertise needed to clarify and prioritize their efforts. Therefore, a collaboration with domain experts is essential. This collaboration is like a three-legged stool. Each leg is critical to the stool remaining stable and fulfilling its intended purpose. When it comes to generating customer intelligence, the three legs of the stool are retail experts, data geeks and coders, and predictive modelers or data scientists.

Retail experts have domain expertise and can best frame the problem customer intelligence is aiming to solve. They suggest derived attributes that will provide value to both the brand and the company’s marketing campaign. Data geeks are needed to program these ideas and store them in a suitable database, which can often lead to greatly increased data storage requirements for the retailer. However, if the data can only be used to create solutions or make key marketing decisions if it’s properly stored and accessed. Inaccessible data means useless data and a wasted opportunity.

Predictive modelers and data scientists are then needed to use the stored data to build models that achieve those business objectives originally set by the retail expert. Predictive models find relationships between historic data and subsequent outcomes so that near-term and long-term customer behavior can be predicted. This leg of the stool aims to answer problems such as the likelihood of when a shopper will make their next purchase and what the value of that purchase will be. Sometimes, these relationships are so complex that only machine learning techniques will find them.

In a real world example, consider a retailer that would like to appropriately message high-valued, loyal shoppers who appear to be disengaging from the brand. A predictive model built from stored data could identify which shoppers are likely to purchase again with seven days, allowing the retailer to let them be the loyal customers they truly are. The predictive model can also show if certain shoppers are unlikely to purchase within seven days but have a high average order value. For these shoppers, the retailer could provide an incentive to bring the shoppers back to the brand. In either case, predicting what shoppers are likely to do is critical to understanding how best to complete the dialogue with them.

From healthcare to finance to professional sports, data is being collected and analyzed like never before — but much of it goes on behind the scenes where the average person may not even notice. The retail sector, however, is different. By definition it interacts with average folks in a way that few other industries do, and retailers are interested in learning as much about their customers as they can. In the process, they are radically altering the buying experience for customers — both online and, increasingly, also in the world of bricks and mortar stores.

If you’ve done any shopping online recently – you’ve probably already seen Big Data in action. You go shopping for air tickets, put them in your virtual shopping cart, but then for some reason change our mind. Afterwards, seemingly every site you visit features an travel portal, best rates and the deals for that destination, along with hotel and cab deals. The reason? The travel site can give you a virtual identification number and track you as you go from site to site, and purchase targeted ads for products they already know you’re strongly interested in.

Based on a user’s behavior, sites like Amazon can present special offers or alert users of products they might not have otherwise been aware. Amazon has had tremendous success by using data it has collected to discover what additional products its users are likely to buy. As the study noted, “Amazon reported that 30 percent of sales were due to its recommendation engine.”

But the proliferation and maturation of information technology hasn’t only aided business. It has in fact given consumers more bargaining power than ever. Consumers are no longer bound by geography — the handful of stores offering a specific product within driving radius — when searching for the best price or service. Even when they are physically shopping in a store, smartphones allow them to quickly and easily check for better prices elsewhere.

In the era of Big Data, it’s those physical stores that seemed destined to be left out in the cold. But the vast majority of purchases – somewhere around 90% — still occur in a traditional retail setting. And brick and mortar retailers are looking towards big data to help them stay relevant.

Retailnext has developed a computer program that uses a store’s security cameras to give managers all kinds of information about how consumers interact with the store. Using this program, it can show exactly how many customers are in a given store at a time, which parts of the store they explore, which specific items customers spend a lot of time perusing — and which they do not. It merges this information with other variables like staffing levels, weather, product assortment and placement to determine what does and doesn’t boost sales. Luxury retailer Mont Blanc has used a similar services to improve its staffing levels and its product arrangement within its stores, increasing same-store sales 20% in the process.

The most successful retailers today are increasing response rates to their offers and driving profitability by using Big Data and predictive analytics to make relevant, personalized, and precisely timed offers to customers. Predictive analytics provides a concrete means of realizing the long-standing exhortation to “know your customer.” analytic tools have sufficient information to enable retailers to treat every customer as an individual based on insights into their preferences and future behavior.

What Problem does Analytics Solve for Retailers?

For retailers, one of the greatest values of analytics is providing decision points for determining how to treat each customer. For example, will it be profitable to offer Customer X free delivery? Or, would that offer be wasted because the customer is going to buy the product anyway?

Consider what happens when consumers visit an online or brick-and-mortar store for the first time. The retailer knows nothing about these potential customers and thus treats them all the same. At some point, the consumer may click on a product or category, or make a purchase. This consumer behavior is likely to trigger a business rule that initiates an action by the retailer. Coupons might be printed at checkout. An e-mail might be sent offering a discount on a product in an abandoned online shopping cart.

In such instances, consumers are differentiating themselves by their behavior. The retailer, however, is still treating them all the same – everyone who exhibits the same behavior receives the same offers. Inevitably the offers will be more relevant to some recipients than to others, and responses will vary accordingly. Because the retailer doesn’t know anything more about these consumers beyond their click stream or the product they purchased, there’s no reliable basis on which to make a more relevant offer

Understanding When and Why to use Various Analytic Approaches

Let’s look at what different types of analytics can tell retailers about their customers, giving them various decision points to consider when determining which actions to take.

Collaborative filtering enables retailers to take a degree of targeted action even with first-time customers. This type of analytics is often behind the product recommendations offered on e-commerce sites and the printed coupons generated at in-store checkout. It identifies relationships between customers and items that have either been purchased or viewed, and infers how an individual will behave based on how other individuals who look similar (i.e., share one or more characteristic) have behaved.

Clustering algorithms enable retailers to differentiate between customers in broad ways such as customers who like premium brands or customers who prefer organic food. One of the benefits of painting customers with this kind of broad brush is that it can help direct and justify large-scale expenditures on store design, new merchandising schemes and promotional programs.

Propensity models enable retailers to predict how individual customers are likely to behave. With such specific insights, retailers can differentiate between customers to a much greater degree, further increasing the granularity of segmentation and the relevance of offers. This analytic approach is particularly effective in Big Data environments. In fact, large retailers who serve tens of millions of customers, each of whom has many attributes and preferences, can go as far as to essentially create segments of one.

Here are examples of how leading retailers are applying big data.

• A large retailer is using this type of analytic approach to increase the ROI from promotional campaigns. When a popular new movie or video game comes out, for example, the retailer sends offers only to those likely to buy the product within the offer redemption period. Response rates are two to three times higher than when the same offer is sent to everyone. And because the retailer is not wasting customer time with irrelevant offers, future promotions are likely to be considered by the customer.
• Another large retailer is improving its ability to predict when customers are about to make a big purchase by incorporating customer clickstream data into propensity models. Many consumers do extensive online research before making a major purchase. By analyzing billions of clicks across millions of customers, along with each customer’s purchase histories and historical behavior patterns, the retailer can pinpoint the right moment to make an offer.

Uplift Models

If a propensity model predicts when a customer is likely to buy a given product, why should the retailer go to the expense of sending a promotional offer? Uplift models help retailers determine if an investment is likely to be worth the result.

Often used in conjunction with propensity models, uplift models predict the amount of change likely to occur in customer behavior as a direct result of a particular retailer action.

The future of big data and the retail industry is very promising with technology taking a strategic lead for maximizing competitive advantage. Let’s consider a couple of chief inflection points that the future might hold. First, the Internet of Things (IoT) will play an important role in terms of how sensor data is writing the next chapter in the retail & big data story. As one example, say you’re walking down the aisle at a retail store and an embedded sensor detects that you have the retailer’s app installed on your cell phone and it offers you the immediate gratification of a coupon.

Next, consider the increasing importance of realtime analytics through use of Spark streaming. This is new technology and still a bit cumbersome to deploy so it’s not being as widely adopted as many might think but it is coming on strong. Real-time analytics represents a tremendous opportunity for retailers who are building their business and retaining their customers.

Case Studies: Dell Focused Customer Use Cases

Some early Dell customers started using big data technology solutions back in 2011 when they came looking for recommendation engine solutions, e.g. the kind of system pioneered by Amazon. The recommendation engine was a seamless way for retailers to start seeing the benefits of big data. Once they saw the value and the ease of Hadoop once the platform had been implemented, the use of the big data technology stack grew from there.

Case Study: Large Retailer

One example is a large retailer who came to Dell for assistance with a recommendation engine project. They collaborated closely on their proof of concept with the Dell Big Data Specialists in the Dell Solution Center, and proceeded to build an 8 node cluster that grew to over 300 nodes in a 3 year period. Once they got their feet wet and saw the results, they moved on to other solutions like an ETL offload mainframe replacement. The company’s next steps with Hadoop were:

• Taking all their data sources and started to use them for their analytics
• Supply chain analysis
• Central data repository
• Price setting
• Logistics planning

Case Study: Staples

Fortune 500 office and school supplies retailer Staples increased its brand recognition and boosted staff efficiency by improving social media listening and analysis, and reducing “noise” (i.e. irrelevant data) by 75 percent. Staples needed to reduce the noise it collected from social media channels to understand customers’ likes and dislikes more quickly, and improve its offerings. As more people engage on social media channels such as Twitter, Facebook, LinkedIn and others, Staples realized that it needed a better solution to sift through the increasing volume of public data to find tactical.

For further discussions reach me at : [email protected]

Reference : Bigdata / Mokleef's PAD / Retail Forum NRF2015-16 / SAS BU 2014