The (Data) Sword of Damocles

The “sword of Damocles” is an ancient moral parable that dates back to around 45 B.C., popularized by Cicero. The tale focuses on Dionysius II, a tyrannical king who once ruled over the Sicilian city of Syracuse during the fourth and fifth centuries B.C.

“Though rich and powerful, Dionysius was supremely unhappy. His iron-fisted rule had made him many enemies, and he was tormented by fears of assassination—so much so that he slept in a bedchamber surrounded by a moat and only trusted his daughters to shave his beard with a razor.

As Cicero tells it, the king’s dissatisfaction came to a head one day after a court flatterer named Damocles showered him with compliments and remarked how blissful his life must be. “Since this life delights you,” an annoyed Dionysius replied, “do you wish to taste it yourself and make a trial of my good fortune?” When Damocles agreed, Dionysius seated him on a golden couch and ordered a host of servants to wait on him. He was treated to succulent cuts of meat and lavished with scented perfumes and ointments.

Damocles couldn’t believe his luck, but just as he was starting to enjoy the life of a king, he noticed that Dionysius had also hung a razor-sharp sword from the ceiling. It was positioned over Damocles’ head, suspended only by a single strand of horsehair. From then on, the courtier’s fear for his life made it impossible for him to savor the opulence of the feast or enjoy the servants. After casting several nervous glances at the blade dangling above him, he asked to be excused, saying he no longer wished to be so fortunate.” (Source: history.com)

The Thread is Growing Thinner

For as long as I can remember, the retail industry has wrestled with data quality. And no wonder, given the millions of products that are constantly churning as some products are discontinued and new products introduced. All that product data, along with rapidly growing product attributes, having to be created, updated, and managed across a global supply chain, from product manufacturers to wholesalers and distributors to retailers.

But data related issues extend far beyond data quality.

Diversity of data is quickly growing in importance, especially as it strengthens marketing personalization and data’s use in small and large language models used to power up ever-expanding AI applications. The acceleration of data growth is forcing a growing number of companies to hire specialists in data management. Realtime data is increasingly valuable in customer experience and operations. And compounding these issues are data silos, which have become a near-permanent fixture in retail enterprises, that impede the rapid deployment of new AI capabilities.

Data has become retail’s Damocles’ sword, and the thread holding the sword is becoming ever thinner as we accelerate up and out the exponential growth curve of tech-driven change. A day of data reckoning is fast approaching, driven by the explosive growth of AI applications across the retail enterprise and across the industry, as the efficacy of AI capabilities is fully dependent on the veracity of the data feeding the algorithms.

There is work to do… and not much time to do it.

The use of artificial intelligence in the retail industry is growing fast. According to one study, the amount of computer processing power devoted to AI is doubling about every 3.4 months - a rate 50-60 times that of Moore’s Law.

Within all the hype about generative AI and large language models, an important fact is often obscured: AI can be most effective when trained and used against a company’s internal data; small language models. Retailers are well positioned to benefit from generative AI run against their own stores of data - if their data is clean, organized, and readily available.

Retailers are increasingly forced to keep pace with the leaders, companies like Amazon, Walmart, Kroger, and others. These companies are increasingly deploying and leveraging AI capabilities to create a growing performance gap between themselves and slower moving retailers.

Business process automation is taking over a fast-growing number of processes within retail organizations; the result being better, faster, and more efficient decisions and performance. Every part of the retail industry is increasingly becoming digitized and driven by data.

Yet extricating themselves from increasingly precarious, and risky, data practices is no small task for retailers.

Why is Getting Data Right Important?

There are the usual reasons for getting data right: Lower costs, improved revenue, more effective operations and marketing, happier customers, and more.

But what has elevated the risk around data is the fact we are now past the inflection point on technology’s exponential growth curve and the expanding use of AI, and growing performance gains fueled by AI, are accelerating in lockstep.

So retailers need to realize that the gains promised by AI are fundamentally dependent on the quality of their data and that the way a retailer organizes and structures their data dictates how quickly and efficiently new AI applications can be deployed.

Noted professor and author Tom Davenport has bluntly called out that leaders in deploying and using AI are opening up a growing performance gap vs slower moving competitors… and that we are approaching the point where slower movers will be unable to catch up.

The Lifeblood of Retail

Data is the lifeblood of retail, made even more important by the development and spread of new AI capabilities that are transforming retail. But bad data is like a malignant blood cell, wreaking havoc as it flows across the enterprise, corrupting the operation.

Imagine a food retailer providing guidance to a shopper to relevant food products beneficial or harmful to the shopper’s health condition. We can all relate to a recent example of a well known retailer caught up in a mislabeling controversy, a product containing nuts but failing to include that on the label, leading to a shopper becoming ill or even dying.

And there are countless more mundane examples of products being incorrectly categorized, corrupting the output of AI powered product assortment systems or promotion recommendations. Like a cancer quietly spreading, bad data is insidious.

The clock is ticking for retailers to address their multi-dimensional data challenge. To clean up their data, bring in additional data that can aid both human and AI powered insights and systems, move the enterprise to collecting and using data in realtime, and to breaking down the silos.

Drilling Deeper: 5 Things to Understand

1. Impact of Poor Quality Data

As the graphic illustrates, the impact of poor quality data is pervasive, extending across the enterprise.

Source: medium.com

Many retail executives fail to take a macro look at the impact of poor quality data; they tend to be most fixated on if sales reporting is accurate, inventory numbers, and margin calculations - all the things that key executives are held accountable for.

But imagine how those key metrics could improve if data were to be cleaned up: improved customer experience = better sales, improved efficiency, better insights, etc. The list is significant.

2. The Growing Importance of Data Diversity

A dearth of data diversity is also a looming danger for many retailers.

Too many retailers have only product data; and many times even that is not accurate or well organized. Some retailers have customer data, often gathered through loyalty programs. Shopper identified transaction data is a prerequisite for personalization and other relevancy marketing.

Larger retailers often avail themselves of third-party data to append additional information to their customer households. Things like income, household size, media preferences, and more are readily available and can assist the retailer in understanding their shoppers and market area.

Location data is becoming more important, both in-store and in the retailer’s market area.

In-store, location data gleaned through smart carts, computer vision systems, or other means, provides insight to true shopper behavior in the store. Department, aisle, and category conversion become new measures of merchandising effectiveness. Knowing what products the shopper is in front of in that moment can power realtime marketing in the aisle.

Shopper location data outside the store can provide powerful, actionable insights. Knowing what other retail competitors your shoppers visit, and how frequently, can be used in ad planning and personalization efforts. Likewise, knowing what shoppers are frequent visitors to competing stores and not to your stores, can power sophisticated, targeted shopper acquisition initiatives. Companies like Refinition and LiveRamp are able to provide this kind of data.

And data diversity extends to products too. There are two facets of product data that I believe retailers should be focused on:

1. Product cost: Maintaining accurate product cost data enables a retailer to accurately measure customer profitability. Retailers having shopper data can build customer profitability from the item level purchase up. Measuring, understanding, and using customer profitability can be a powerful new tool.
2. Health & nutrition attributes: The growing ‘food as medicine’ movement is dependent? on accurate and extensive attributes attached to each food product.

While health & nutrition related attributes can help shoppers filter a retailer’s products to find sought-after foods, the power is magnified when the retailer encourages the shopper to build a ‘health’ profile that can include food allergies, diets, health conditions, and even prescription medications.

For example, Sifter Solutions works with retailers and healthcare to power up these kinds of capabilities, being instrumental in identifying what products qualify for Healthy Benefit Cards and shopper relevancy. Sifter is also working with two of the largest retailers helping ensure the accuracy of their food product data.

3. The Accelerating Quantity of Data

“According to a global survey last year, 33% of business leaders said they can’t generate meaningful insights from their data and 30% said they were overwhelmed by the sheer volume.” (Source: WSJ) And poor quality data is costly. “Juergen Mueller, chief technology officer of SAP SE, said, that it’s typical for 80% of any given data project to be spent simply cleaning data and re-creating the context.” (Source: WSJ)

As the above graphic shows, the amount of data being created is also growing exponentially, keeping pace with the leaps in computer processing power. Retail executives need to develop a strategy and plan for capturing an ever-wider diversity of data, organizing it, and storing it so it is available to the entire enterprise.

Arthur Cooper, CRO of Armeta, says this about data: “Executives typically know what they want, but they and their team can't bring the data together correctly.? This will continue to block the ability to do anything innovative like AI, ML and using 3rd?party data.”

4. Why a Unified Data Foundation is Core to Future Success

As retail organizations grew over the decades, companies would add departments and functions - merchandising, marketing, operations, HR, etc - each of those departments becoming silos having their own applications and data. Inevitably, data held within one silo became needed by another, leading to a point-to-point integration. And those integrations have multiplied over time, creating a giant hairball.

As discussed in Bionic Retail, retail is evolving to a digital operating and business model, one built on a unified data foundation. Amazon has led the way, with retailers like Walmart and Kroger pushing hard to transform. A digital operating model with a unified data foundation makes it far easier and faster to deploy new AI solutions as a growing number of business processes are automated through software.

Retailers need to focus on breaking down their organizational and technical silos, and unifying their data. This includes creating a comprehensive, and accurate, data dictionary along with standard (API) calls so that your valuable data can be made available to any user across the enterprise.

5. Realtime Data for a Realtime World

“The world has changed,” says Mindy Ferguson, vice president of messaging and streaming at Amazon Web Services. “Think about how we used to sit down to watch the nightly news once a day. Now we have access to news around the clock. We expect to know when things happen in real time.”

Data is made even more valuable when it’s available in realtime. Understanding where the shopper is in the store at that moment makes it possible to message with relevant offers and information. Knowing that the swordfish in the seafood department is going to spoil if not sold within the next few hours can trigger a promotion to shoppers in the store.

Just think about the growing number of IoT devices in the store. As a store manager, I want to know within seconds if a frozen food case has gone down. Or Kroger’s checkout system, monitoring the number of shoppers entering the store and projecting how many checkout lanes need to be open in the next 15 minutes.

Realtime data can enrich the shopping experience. Conversely, a lack of realtime response - especially when expected by the customer - can push shoppers elsewhere.