When Random is Good

2024 is the year of elections. This year, an unprecedented number of voters around the globe are set to cast their ballots. In at least 64 countries, along with the European Union, national elections are scheduled to take place. These nations collectively represent nearly half of the world's population.

Understanding the Pulse of the People

As nations gear up for these critical elections, the spotlight turns to poll predictors, who will be working tirelessly to gauge the political climate. Achieving accuracy in these predictions hinges on a principle that might seem counterintuitive: the power of randomness. For a survey to reflect the true diversity of opinions, each participant's selection must be as likely as any other, embodying the essence of randomness.

The Misconception of Randomness

Our intuitive sense of randomness often misleads us, as evidenced by how we choose lottery numbers and perceive the likelihood of shared birthdays. For example, many avoid sequential lottery numbers, eg 1,2,3,4,5,6,7, mistakenly believing they're less likely to win, not recognizing that all combinations have equal chances.

Well, let me take you back to a pivotal moment in history that perfectly illustrates this point—the 1936 U.S. Presidential election.

The Tale of the Misguided Forecast

In 1936, the United States was deep in the grips of the Great Depression. It was a year of significant economic turmoil, but also a year that would witness one of the most dramatic misfires in the history of electoral predictions. Enter The Literary Digest, a popular magazine of the time, known for its political polls. The Digest was confident in its survey method, having successfully predicted the outcome of the previous five presidential elections. So, when it announced its prediction for the 1936 election, people listened.

The magazine had sent out tens of millions of ballots to its subscribers, people listed in telephone directories, and car owners, collecting a staggering 2.4 million responses. Based on this, The Literary Digest confidently predicted that the Republican candidate, Alf Landon, would win in a landslide over the incumbent, Franklin D. Roosevelt. It had predicted that Landon will get more than two-thirds of the electoral votes,

The Reality Check

Spoiler alert: Roosevelt won. And it wasn't even close. He secured a sweeping victory, carrying 46 of the 48 states, winning 98% of the electoral votes. So, what went so wrong for The Literary Digest?

The answer lies in the selection of their sample. By choosing its subscribers, people in telephone directories, and car owners, the Digest's survey skewed heavily towards the wealthier segments of the population. During the Depression, owning a car or a telephone, let alone subscribing to a magazine, signaled financial well-being far above the average American's.

This sampling bias overlooked the vast majority of the population who were bearing the brunt of the Depression—those who were likely to support Roosevelt's New Deal policies. In essence, The Literary Digest's sample was anything but random.

The Importance of Randomness

This historical blunder beautifully underscores the critical importance of randomness in sample selection. Without randomness, we risk introducing bias into our results, leading to conclusions that may be wildly inaccurate, as was spectacularly demonstrated in 1936. It's a stark reminder that when it comes to statistics, sometimes, random is good—actually, it's essential.

The Takeaway

The 1936 election serves as a powerful case study for anyone interested in the fields of statistics, survey methodology, or political science. It highlights the pitfalls of biased sample selection and the importance of embracing randomness to capture the true sentiment of a population.

Whether we're conducting market research, political polling, or social science studies, the principle remains: a random sample isn't just a good practice; it's the cornerstone of credible, reliable statistical analysis.

Thanks for reading! The views expressed here are my own.

Prompt for Image : Create a minimalist image by pencil sketch showing elections.