Lies, Damned Lies...you know the rest

We’ve all heard the quote:?There are three kinds of lies:?lies, damned lies, and statistics. Wikipedia, which can never be wrong, tells me that the quote I thought was from Mark Twain was indeed popularized by him, but he credited it to Benjamin Disraeli. I’ve always liked the progression in severity: lies are a grievous thing, which produce anger and demand a response. Damned lies, I gather, are something that leads to an altercation of sorts (a brouhaha, a kerfuffle, fisticuffs, or similar rarely used words). Beyond this realm somewhere exists statistics (for which I am going to substitute data).?

Data sin number one:?Just making stuff up. I’m trying to tell a story, there’s a point that I think is intuitive or have heard often enough that I take it as a fact. This is so unsophisticated and unethical that I don’t want to spend time on it. No professional would engage in this, would they?

“Today’s consumer is time-starved and demands convenience.” Consumer of what? What is the condition of being time-starved and what is the benefit of its being mitigated? What in God’s name does ‘convenience’ mean? To whom? Under what circumstances? Just say it—it sounds good, no one will contradict you, and it’s a great substitute for thought.

Defensive strategy: “You know, I hear that a lot. Do you know where that comes from?”

Data sin number two:?Cherrypicking! Cherrypicking is something other people do. They scour data looking for something that confirms their bias or supports the story they want to tell (the thing they want to sell). This isn’t necessarily a misrepresentation of fact—after all, they are citing a legitimate data point, but it’s obviously a ‘curation’ of the data that is maybe guilty of sins of omission rather than sins of commission.?

This is common and shows up in the best of organizations. Part of it, I think, is the need to make a nice PowerPoint and move on. Senior executives bear some of the responsibility for this, as well. Confronted with a great deal of information, and faced with pressing decisions, senior management needs concise information (do you suppose they’re time-starved? ). The downside of ‘concise’ is that nuance is lost. As a point of humor, I was once presenting to senior management and explained that the data I was presenting ran counter to conventional wisdom because it was more nuanced. A chuckle spread across the room, not because of the word itself, but because it was so unusual to be discussing nuance. I hasten to add that these were very bright and talented executives. I think they were just time-starved.

Defensive strategy:?“What else did your study say about x?”

Data sin number three:?Misinterpreting indexes. This is so common that it might be funny, were it not for the fact that targeting decisions are sometimes made on the basis of this very simple error. It goes like this:?Among a segmented group, attribute x is far more likely to purchase than any other segment—thus, they should be the target. Nobody bothered to check that attribute x is only 0.6% of the population. Before I care about a group of people who are more likely to buy, I’d want to know how many of them were out there, how often they buy, and how much they buy.?

Admittedly, this should not happen with any competent analyst. Everyone knows better. But it can still turn up in surprising places.?

Defensive strategy:?“How large is that segment (percentage of my customer?) How often do they shop my store? How often do they buy? How much do they buy?

Data sin number four:?Counting the wrong noses. The data that is easiest to find and easiest to collect is often general population data. As long as whatever you’re measuring lines up well with that population, then this might be fine. For example, if my population is supermarket shoppers, a highly shopped category might be a good fit for using gen pop numbers. What about when it’s not? I once worked on a product where somewhere around 1% of supermarket shoppers accounted for 80% or so of product sales. If you want to understand packaging, form, etc., for this product, why would you ever ask gen pop or total shopper population? What I cared about was what the 1% thought about any prospective change. That’s basic, right? Any competent researcher will consider the right population for a study. Keep your eyes peeled and you’ll see this sin in surprising places, too.?

Defensive strategy:?“Who was in this study? Why did you choose that group for the study?”

Data sin number five:?Not being curious. This sin is, I believe, the most common and is the reason that most of the value of any research study or syndicated data is lost (according to me). Even if no one is just making stuff up, cherry-picking, misinterpreting the data, or getting the population wrong, the greatest sin is not asking the why question enough times.

Sales of a given product or category are down. Why isn’t the consumer buying as much as they were?

• I wonder how the sales change varies by retailer? By region? By price relative to competitors in market? By in-store execution??
• How does the change in sales at a given retailer correlate with the satisfaction of shoppers in that category at that retailer?
• Is fuel a factor? Higher gas prices may result in fewer trips, leading to fewer purchase occasions. If this is so, how else is fuel affecting consumer decision-making?
• Is there a reason to believe that whatever change is taking place is permanent? Since COVID there has been an endless stream of predictions relative to “the new normal.” How many of those predictions can be substantiated by data over time?
• What factors may cause future changes in market performance:?are there input price increases that have not been brought to market yet? Do we expect those input cost increases to be temporary, and if so, of what duration?

Is it possible to get analysis paralysis by going down the rabbit hole? Sure. Is it important to your business? Probably worth a deeper look. What is the business decision that will be driven by the data??If that decision may yield a significant competitive advantage, it’s probably worth a longer look.

Defensive strategy:?My Marketing Godfather, Bob Salegna, taught me many years ago to ask two questions:?“Why?” and “How do you know that?”

This list of data sins is certainly not exhaustive but maybe it provides food for thought, whether you’re conducting research or consuming it. We like to say that knowledge is power, and yet, we often fail to extract the value of research that we have on hand to do the one thing we intend for it to do:?to give us information that no one else has so we can drive business decisions that transform our business.?

To put it another way, An insight is not an insight until somebody makes a buck off of it.?