Polling Failure in US Elections: How to improve elections (and marketing) prediction with Machine Learning?

This is a discussion article and I am sharing my beliefs about what went wrong in our polls. I am very interested in hearing your ideas. Do you believe it was intentional recklessness or misleading? Do you believe it was a mistake? How can we make sure it does not happen again? Please share your ideas in the comments. Thank you.

Regardless of who wins the election, we know who the biggest loser was in the 2020 elections: the polling industry. Now, for four cycles in a row, pollsters have embarrassed themselves to a point that their polls seem more like fantasies than statistics.

If that was not enough, nonstop announcements of average of polls by various leading media organizations seem to add to the absurdity of the situation. Many people are now accusing the industry of influencing and directing the voter behavior, rather than measuring it. In his November 5th, 2020 address, President Trump used the term “suppression” to describe misleading polls and suggested that such exaggerations can affect pre-election donor and supporter behaviors.

The question I will try to answer in this article is: how to use artificial intelligence to improve polling and election predictions?

What went wrong?

I believe the problem is that many pollsters continue to use outdated methods and techniques. Such techniques worked in the twentieth century (or the first decade of twenty first century) setting where voters were less suspicious of the pollsters and were more open with their answers.

Today, we live in an environment where many people have become increasingly suspicious of deep state or other domestic or foreign elements trying to influence or control their thinking.

The same media that claims that Russians influenced the US elections apparently fails to understand that such a narrative also makes people more suspicious of phone calls and texts from strangers. If Russians can reach them via phones, texts and social media, or India or Nigeria based fraudsters can go after their money, or foreign powers can sabotage elections, then why should they trust a text or a call received from a stranger?

This is rational behavior. We cannot scare people (based upon reality or perceived threats) and not expect them to behave differently.

So many people are either dishonest about their preferences or refuse to participate in a poll which makes polls highly skewed.

Also, when political preferences are being equated to disloyalty and lack of patriotism on one hand and racism and bigotry on the other hand, people will not feel comfortable being labeled as racist or disloyal to their country. They will not respond honestly.

Furthermore, the polarized environment in the country makes some expressions more acceptable in certain parts of the country and less tolerable in others. Sometimes that happens within the same state. This polarization has affected deeply held beliefs, cognitive structures, rationalizations, opinions, emotions, linguistic expressions, humor styles, values, and vocabulary choices. It has even impacted people's sense of security. Expecting people to comfortably and openly disclose their voting preferences is unrealistic - especially when doing so risks compromising their emotional or physical safety.

The important thing is that while you can lie in response to a survey question, it is very hard for your overall behavior to lie.

That is why, artificial intelligence centric polling – which I call Behavior Extraction Technology – is far more accurate and powerful because it does not depend upon direct survey responses. 

How to deploy BET (Behavior Extraction Technology)?

BET is used to extract patterns of voter (or buyer) behavior and predict their actions. This technology has the following parts:

1)   Speech: Voter’s choice of words and speech patterns predict which way they are leaning. This is obtained from natural language processing. The interaction that captures the speech does not have to be about political preferences. In addition to words, other related aspects could be speech inflection, cadence, tone, and intonation to study the speech patterns.

2)   Narratives: Derived from ethnographic or netnographic studies, patterns from narratives can indicate voter preferences. Again, the narratives do not have to be directly about voter preferences. For example, simply asking open ended questions about their problems can give clear insights into how voters will vote. 

3)   Actions: Voter actions – for example, displaying flags, stickers, and social media updates – give an understanding of voter preference.

4)   Affect: Facial expressions and body language can capture and indicate emotions. Voters reveal significant information about themselves in normal nonpolitical interactions.

5)   Games and Fun Activities: How people behave in simulated environments – such as video games – and nonvirtual fun activities (such as participating in a fair or festival) can provide information about how they make decisions. It gives clues about how they will vote, their commitment level, and motivation.    

6)   Other Behaviors: Other behaviors such as product and services preferences, consumption patterns, etc. can also give a good understanding of voting behaviors.

Employing the above to study and extract the voter behavior can enhance the prediction ability.

What to watch out for?

The above methods are not bias-free, and they can also lead to bias. They can serve as a barometer for other polling data. The obvious concerns about privacy must be addressed.

Action Steps

Pollsters and marketing departments must improve their models. In today’s world and given the existing technology, there is no excuse for getting polls so wrong. AI can help improve election and other polls. The world needs more empathy – and machines can help make us more empathic.  

Al Naqvi CEO of American Institute of Artificial Intelligence