Some Thoughts on the Flubbed Election Outcome Prediction

Some fascinating watercooler conversations and social media discussions are happening today about the shocking US Elections outcome that defied all polls. Common question: “How did the polls get it so wrong?”. Most pollsters had predicted a comfortable victory for Clinton.

My own quick take on this is that our polling methods and devices haven’t kept up with the times; and secondly, this was a rare event and we had limited historical information to allow us to make an accurate prediction based on human behavior.

Nevertheless, the election results suggest that pollsters grossly underestimated the number of hidden Trump voters, i.e. people who came out to vote in droves on election day but they were not on the radar of surveyors. A few possible reasons for this:

1. Pollsters didn’t seriously survey those who didn’t vote in 2012 assuming their voter apathy will continue this time. Quite a flawed assumption! We now know that a key demographic that might have been missed was the young white wealthy voters who predominantly voted Trump.

2. Pollsters also admit to having missed out on surveying the evangelicals who were mostly unreachable by phone. This starts sounding like convenience sampling more than random sampling! Polls by Pew Research indicated this segment would vote Trump.

3. Phone polling might not have captured people’s voting intentions accurately. Some voters were sheepish in admitting to a human pollster that they were voting for Trump.

In comparison, online anonymous polls such those conducted by Los Angeles Times & the University of Southern California did a better job. These polls calculated the ratio of a person’s likelihood of voting for a specific candidate to his or her estimated chance of voting. Trump was pegged as the winner. Not only are these polls more reliable by virtue of the questions asked, I would also contend that these anonymous polls are more representative of actual voting scenarios where people cast their ballots privately. Hence they offer a viable complement to traditional polling methods.

4. Lastly, we need to pay more attention to indicators such as social media engagement and buzz, which while not entirely accurate, could still be very useful indicators of election outcomes. These types of indicators were used in an AI prediction model which… you guessed it… predicted a win for Trump.

On the whole, it sounds like pollsters need to review their foundational assumptions about voters, and they also need to adapt to new realities where people use varied media channels to express themselves in different ways.

We need a representative sample, we need to definitely estimate the likelihood of voting in addition to the respondent’s choice, and most of all, we need to avoid biases in sampling and estimation.

Following these basics and turning to advanced analytics and data science tools to include more data points that model human behavior might have given us a better prediction – and ultimately, lesser astonishment.