The case for small data

When my senior partner decided he was going to buy a Ford Explorer, he had a clear, logical reason: safety. He wanted something big, with a muscular build, something that said, “I’m going to handle whatever the road throws at me.” He’d done the research, seen the ads, and could practically hear the roar of that engine protecting him from everything from high-speed traffic to late-night thoughts about aging.

But here’s the twist: he didn’t buy the Explorer. No, he rolled up with a brand-new Mercedes sedan, the classic choice of success, elegance, and—dare I say—a touch of nostalgia. When I asked him what happened to that towering Explorer, he shrugged and said, “I realized I was old, and you know, I’ve always wanted a Mercedes.” Just like that, all his talk of rugged resilience and safety gave way to a very different, very personal dream. He was getting the car he’d always wanted, Explorer be damned. It’s a powerful little parable of consumer behavior that even the most finely-tuned datasets would struggle to capture.

It got me thinking about data and consumers, or rather, how even "big data" often falls short in answering the more delicate question: why do we buy? It’s easy to count clicks and log locations, but when someone like my partner shifts gears at the very last minute, driven by nostalgia, pride, or even just a reminder of what they wanted when they were younger—where’s the data on that?

Jeff Bezos once put it simply: when big data clashes with a good anecdote, he trusts the anecdote. Think about that. One of the biggest minds in tech, whose career is built on data, chooses to believe in small, human stories. Why? Because he understands that while data points reveal patterns, only human stories reveal motivations.

So, here’s the case for small data: it’s not about scrapping big data or even ignoring it, but about pairing it with the granular, qualitative details that big data can’t easily touch. It’s about understanding that buying patterns are more than patterns; they’re impulses, nostalgic pulls, even contradictions. And for all its algorithms and AI, data alone can’t measure the dreams of a man who wants to drive a muscle car but remembers he’s always wanted a Mercedes.

If we’re going to try to make sense of consumer behavior, we have to respect its irrationality, embrace the randomness that comes when people shift from reason to memory, or from logic to desire. Big data tells us what we did; small data might actually tell us why.