Ask Mr. Research: How can a cup represent a cauldron?

How can a small sample represent a much larger population?

This is one of the questions I’m asked most often. People see a national survey sample of 1,000 and ask: “How in the world could such a small percent of the population possibly represent a universe of 350 million Americans?” After all, 1,000 is just 0.00000286 percent of the population!

The answer is random sampling. The theory of random sampling states that if we take our universe and interview 1,000 names at random from that list, the results we get from our poll are repeatable. In other words, if we poll another 1,000 randomly-selected names, we can expect the same results, to within a predictable margin of error (+/-3 percent for that sample size).

Why does this work?

My favorite analogy is to imagine a giant cauldron of stew. The cauldron may contain meat, various vegetables, flavorings and spices. Now imagine that we puree’ that cauldron of stew to randomize the contents. Once the stew is “randomized” we can draw a single cup and effectively analyze the contents of the entire cauldron.

That cup of puree’ will contain all of the ingredients and in exactly the same proportions as the stew in the cauldron. This is measurable and repeatable for every cup of the pureed stew.

What’s the practical application?

While random samples are not usually easy to obtain, reasonable efforts to build an inclusive sample, while acknowledging any potential gaps, typically produce reasonably representative samples. That said, using Census data and other available information allows the analyst to draw conclusions about the quality of the sample, and additional efforts can be made to fill gaps or mathematically weight the sample to ensure that the final sample accurately reflects the target audience.

It’s not a magic spell: it’s part science and part art.?This practice has provided a reasonable and practical way for organizations to better understand their audiences for decades. It's never perfect, but it beats guesswork and ‘trusting your gut’ every time.