Data-driven Decision Making in Retail

What is the real challenge to make it work?

The benefits of data driven decision making are widely known. Many studies in academic literature demonstrate the benefits of using data for marketing decisions. This applies to the retail sector too. For example, by using algorithms for shelf planning, retailers can gain 6% to 20% more profits compared to the application of rules of thumb when assigning shelf space to products (1). The retail sector is naturally rich of data. Shoppers leave their footprints during the complete customer journey: search behaviour is tracked on retailer websites, checkout tickets provide information about the type of purchases, and loyalty systems link transactions to individual shoppers. Moreover, the good (or bad?) news is that the amount of data is expected to increase significantly: as more shoppers connect to more devices, use more devices, while they also expect to be able to connect real-time the complete shopping journey. In this article I seek to answer the question of why retailers struggle to make data-driven decisions even though information technology (IT) and data are available to them. In search for answers, I focus on four levers: IT, data, organisational culture and people.

?

Investing in Information Technology (IT)

Traditionally, retailers - like most organisations - travelled the path towards data-driven decision making via investments in IT: the hardware, software and people to store, analyse and transmit data. It makes sense, given the evidence of the positive relationship between IT investments and organisation performance (2). However, savvy retailers soon realized that IT investments were not sufficient to get the job done. Having access to the data is one thing, knowing how and when to apply it to decision making is another. Lidl in Germany stopped their ERP project in 2017 after spending €500 million euro in seven years. The problems for Lidl started when the software was rewritten to value their inventory against retail prices instead of purchasing prices. Similarly, Woolworths in Australia gave up in 2015 after losing more than €700 million euro in six years. At Woolworths processes were not documented, and knowledge about how things were done, was gone when key people left the organisation. For some it is tempting to harshly judge the two retailers, but it is not a fair tribute to the serious attempt of thousands of employees and consultants to make things work. We can draw a few lessons from these two examples. First of all, any major change project needs to be adaptive. If the outcome and process are defined from the get-go, the projects are doomed to lose relevance in changing contexts over the course of the years it takes to implement them. Rigidity in the original blueprint prevents the organisation from leveraging technological innovations during the deployment of the project. Secondly, processes and tools must be aligned to get maximum return on investments. The implementation of standard software solutions would require organisational processes and metrics to be adapted. This might be viable if there are little or no pre-existing metrics in place. On the other hand, the development of custom-made solutions necessitates a higher investment in both time and money. In my own consultancy practice I worked with a non-food retailer that asked a vendor to develop and implement tooling for commercial decision making without documenting the current or desirable processes. They ended up with project extensions, more cost, and users calling software developers directly explaining how they worked.

?

Democratisation of Data

Thanks to the growing use of technology, tools and data are available and accessible across the board delivering a plethora of information. The statistical principles and methods have remained, but are now applied to datasets that are larger and fresher. Even unstructured data such as YouTube or TikTok videos that typically constitute 95% of a large dataset, are increasingly made ready for statistical analysis. This democratisation process happened in steps, or, acknowledging the speed of change, in leaps and bounds. Initially, the amount of data was a challenge, and organisations started collecting it in a structured manner data about their customers. Not so long ago concepts such as Direct Marketing, Database Marketing and Customer Relationship Marketing (CRM) were hot topics in the marketing community. But now Big Data has become the norm. Therefore, organisations can zoom in on the quality of data and describe datasets in 9Vs: volume, velocity, variety, veracity, variability, visualization, validity, volatility and value (3). Organisational focus has shifted from data collection and data quality to data extraction and actual use. From my consultancy practice with large organisations I have learnt the biggest challenge is drifting downstream: The analytics teams develop so many business ideas that the marketing and other departments do not have the time nor capacity to implement them. And hidden within this trend is an existential threat for the marketing function: The analytics function and their algorithms are becoming so strong and fast, they might make their human colleagues redundant.

?

Learning Culture

Since the introduction of scanning technology at their checkouts retailers also have access to vast amounts of data. Because they are in direct contact with shoppers every day and connect transactions to shoppers through loyalty systems, the retail sector has the natural head start on their suppliers to become truly customer centric. One example of how this took off is when Tesco let dunnhumby, the market research agency they later acquired, dig into their loyalty card data. When leading the shopper insights function at PepsiCo Europe some ten years ago I encouraged the country organisations to work with the data and analytical tools provided, or better said, sold by retailers. The adoption was instantaneous: Not only because the new type of data created opportunities for innovation, but also because the set of analytical tools were connected with the category management way of working that was already in place. In addition, suppliers such as PepsiCo started to speak the retailer’s language by analysing the categories with help of retailer specific shopper groups. The adoption of the new data was not always successful. For example, a French grocery retailer stopped sharing their loyalty card data when they noticed their buyers did not rely on the analytical tools for their discussions with suppliers. The culture of (conflict) negotiation was stronger than the incentives from working in a data-driven manner.

?

People

Retailers should not only consider the transformation towards data-driven decision making from organisational and inter-departmental levels but also on individual level. What are the needs of individuals and how do each of them define data-driven way of working? Conceptually, individuals may agree to new ways of working, but it may be harder in practice. It is important for organisations to assess if the required skills have changed as a result of the transformation. Often the installation of self-service applications transfers analytical tasks from the data analytics department to the marketing team.?As a result, marketing professionals have to possess the technical skills to operate analytical tools and also be more self-reliant and curious. On the other hand, the analytics team gets less requests from marketing and may decide to reallocate that amount of time to the development of new algorithms requiring deeper software writing skills. Most organisations pay attention to managerial and technical skills through training but forget that the cognitive style and personal preferences are not aligned with a data-driven way of decision making. Management may need to intervene carefully for example when they observe that new employees hold on to data to avoid any risks while employees with a long tenure sail on intuition and more innovative ideas. When working with category managers from one retail organisation, each time I am surprised to see the diversity in styles and preferences: whereas one category manager loves to explore the numbers with analytical tools and is hungry for more data, their colleague at the opposite desk is not. Maybe I shouldn’t be so surprised: People are different, and each has his or her strengths. Being data-driven is not a requirement for success in each role and for each individual. And, admittedly, on an anecdotal level, people that make decisions without data, certainly seem happier! This reminds me of a transformation project I led at the Gillette Group (now P&G). The skills and personality of account managers were matched with the culture of retailers so that some relationships were built on strategic management and analytics and others thrived on relationship and intuition.

?

Conclusion

This is not an exhaustive list of stimuli and barriers for data-driven decision making. I reflected on aspects that are often associated immediately (IT and data) and the ones that are overlooked more easily (organisational culture and individual). We learnt from the IT case studies that set-up of IT-projects should be based on the way people work and should be open to technological innovations. The democratization of data disperses so fast that we have become less worried about the amount and quality of data but more about the selection of the decision maker: humans or machines, the marketing team or the analytics team. Retailers may have the IT and data in place but lack the learning culture and the right mix of people. In addition to analytical skills, retail organisations need to encourage curiosity and intuition in their culture.

?

In summary, there is more to data-driven decision making than getting systems and people in place. As retail starts embracing data in a shopper centric way and leveraging it towards a better service level, happier shoppers and more brand relevance: the adoption of data-driven decision making will become more widespread. Shoppers certainly have become more open to sharing their data. There is an opportunity for retailers to do more with the data. Yet, there is no simple answer, first there are many small hurdles to be overcome.

?

TL;DR

Why do retailers struggle in the adoption of data-driven decision making, even when data are readily available? I searched for answers in four domains: IT, data, organisational culture and people. IT requires huge investments, and the return on investment is linked to the actual use of the metrics and KPI that are obtained from the systems. Just getting the IT in place is not enough. Big Data has become the norm. Therefore, organisations can zoom in on the quality of data and describe datasets in 9Vs: volume, velocity, variety, veracity, variability, visualization, validity, volatility and value (3). Culture is critical for the adoption of a data-driven mindset. If not, the inertia of decision-making as usual overrides the data: no matter how accurate or brilliant. People may either love or hate data and thus either use it too much as if all answers are in the data, leading to risk avoidance. Or they may use it too little or not at all, arguing that things have been fine without it. The struggle in the adoption is across many levels. Not just in the data itself, but in how to give it meaning towards a more shopper-centric way of driving retail business.

??

About the author

Constant Berkhout works at the crossroads of Shopper Psychology, Data Analytics and Retail Strategy and is author of three books:

• Retail Marketing Strategy, Delivering shopper delight
• Assortment and Merchandising Strategy, Building a retail plan to improve shopper experience
• The Retail Innovation Toolkit, 42 category management tools for growth

?

References

1. Van Nierop, Fok and Franses, 2008.
2. Kohli and Devaraj, 2003; Mithas, Tafti, Bardhan and Goh, 2012; Sabherwal and Jeyaraj, 2015.
3. Lane, 2001; Ducange, Pecori and Mezzina, 2018.