Polling Pitfalls: Why the Numbers Don’t Always Add Up!

With election season upon us, a familiar tension is all around: the clash of candidates, the race for votes, and-oh yes-the polls! But before we get swept up in the all the excitement, let's explore a crucial question: What do these polls really tell us? The truth is polling is more art than science and understanding polling intricacies is key to deciphering who might take home the victory.

Navigating the Polling Landscape

Polling isn't merely about asking questions; it's a delicate dance of methodology. However, various errors can muddy the water and lead to misleading conclusions. Let's explore some of these errors:

Non-Response Error:

A significant and ever-increasing challenge in polling is getting people to respond. When certain demographics are less inclined to participate, the results can become skewed. This is especially critical in elections, where the voices of specific communities may be underrepresented. Political pollsters often build models to account for non-response.?Models are often based on previous election results but so many changes happen from election cycle to election cycle and there is no single model of truth. Implementing strategies like targeted outreach can enhance participation and ensure accurate representation.

Margin of Error:

A term often tossed around but important to truly understand. The margin of error can mislead the untrained or unsophisticated. Understanding this concept is vital, especially in dynamic communities where opinions may shift rapidly. Essentially, the margin of error shows the range within which the true opinions of the whole population are likely to fall, based on the sample. For example, if Candidate A has 52% support with a margin of error of +/-3%, their actual support could be anywhere from 49% to 55%. A small margin might seem reassuring, but it's a reminder that results can swing significantly in either direction.

Sampling Error:

Sampling, or how we choose participants, matters. Margin of error can only be calculated on methodologies that offer the opportunity for all people to be included. These types of sampling methodologies are called population-based sampling. Traditional telephone survey methodologies are one of the most popular population-based sampling approaches; however, they have become increasingly cost-prohibitive, time consuming, and are plagued with low response rates (a typical telephone survey has a response rate of less than 1%, which means only 1% of the population that is contacted to participate actually participates in the poll). Address-Based Sampling (ABS) offers another way to target all adult members of the population effectively. ABS relies on physical addresses rather than phone numbers or email lists, using databases that include nearly every residential address, even in hard to-reach areas. This approach allows us to reach a broader cross-section of the population, including those who may not have stable internet or phone access, which is particularly important in achieving a representative sample. However, if ABS is not executed properly, it can lead to inaccuracies, but when done right, it significantly improves response rates and diversity in responses. Unlike traditional telephone surveys, which suffer from low response rates, ABS’s response rates can range from 10%-20% or even higher. With ABS, we can more reliably represent diverse communities by ensuring each resident has a measurable chance of being included in the survey.

A Closer Look Through an ORS Project

In 2023, the City of Seattle conducted their Technology Access and Adoption Survey, a study conducted every five years and designed to understand how Seattle residents use technology and the internet, while uncovering barriers that may prevent full digital access. ORS, the primary research partner for this study, crafted a stratified sampling plan to ensure each community was adequately represented, ultimately sending out a comprehensive survey package in the mail to 19,500 households. Recognizing that certain groups, like low-income populations, are historically more likely to be non-responsive, ORS implemented specific strategies to address these challenges. For instance, we over-sampled low-income residents to capture their perspectives more accurately. ORS also performed mathematical corrections (weighting) to adjust for when particular groups of people responses are in lower levels than they appear in the population.

We used Address-Based Sampling (ABS) which allowed each resident a measurable chance to be included, making it possible to calculate the margin of error. ABS’s population-based approach, combined with multiple touchpoints to encourage participation, contributed to a commendable 15% response rate for mail-in. This thoughtful approach provided valuable insights into the communities’ use of technology, with a particular emphasis on digital equity and inclusion.

This comprehensive effort supports Seattle’s ongoing commitment to digital equity, equipping the city with data to identify and address gaps in access, resources, and services. You can read more here about this important body of work and the City of Seattle’s response to this information as they work to bring digital access and digital literacy to every member of the community. ?

Understanding the Bigger Picture

Polling is a powerful tool, but it’s not infallible. Errors in methodology can lead to misguided conclusions about public sentiment, especially as we approach pivotal elections. The interplay between non-response, margin of error, and sampling error highlights the necessity of diligent survey design and execution.

A Polling Analogy: Surveying the Candy Aisle

Imagine walking down a candy aisle at your local store. If you ask 100 people their favorite candy, you might get a variety of answers. But if only those with a sweet tooth respond, you’ll likely miss out on insights from those who prefer savory snacks. Just like polling, it’s essential to ensure all voices are heard, or you might end up stocking your shelves with chocolate when gummy bears are the true crowd favorite!

As we engage with polling data, let’s remember that it’s not just about numbers; it’s about voices. At Olympic Research and Strategy (ORS), we navigate these complexities to ensure that every survey paints an accurate picture of public opinion, empowering communities to make informed decisions.