Predicting Elections and Human Behavior: Stop Asking and Start Observing

In one of the wildest and most contentious elections in U.S. history, many Americans had trouble seeing eye to eye. About the only thing we can all agree on is that the pollsters completely blew it when it came to predicting our next president.

Some of these inaccurate poll results call into question the validity of modern-day statistics and data science.

Political polling is a beast all on its own, but the process has much in common with the AI-powered software being built by some of the world’s most influential tech companies.

The truth is, pollsters aren’t the only ones making poor predictions. There’s a lot of questionable data science masquerading as AI.

Societal implications

If the pollsters can’t accurately call an election, we’d better take a hard look at AI-fueled technologies that could have a profound effect on our lives, such as self-driving cars and heart-monitoring devices.

With all of the AI hype in today’s market, everybody labels themselves data scientists. In fact, simply updating your LinkedIn profile to include the term “data science” could increase your income overnight, with the position earning on average $123,000 per year – 113% higher than the average annual salary. However, this title doesn’t make someone a master of the craft, as even smart data scientists need access to large amounts of accurate and timely data to make reliable predictions.

So, why did the pollsters miss on this election so badly? Did they introduce bias? Or did voters misrepresent their intentions – either on purpose or because they were unsure – making it harder to get reliable data? Or, was it both?

Accurate predictions require accurate data

One major problem with the election polls is that they did not consider a broad enough and representative data set. Insufficient data often leads to bias and inaccurate results.

While most of the pollsters asked Americans who they planned to vote for in the election, the USC/LA Times Daybreak Poll took a different approach.

This poll asked people how likely they were to vote for a particular candidate. By continuously gathering new data each week that captured shifting sentiments over time, they accurately predicted Donald Trump’s victory while others continued to project Hillary Clinton as the winner.

It’s important to remember that not all data is created equal. In this election, telephone polls proved much less reliable than online polls. Perhaps voters didn’t want to acknowledge they were supporting Trump on the phone.

Here’s the thing that most people miss: Observed data is inherently more accurate than self-declared data. People don’t always reveal their true intentions when you ask them. Observing their behaviors is a much more accurate predictor of what they will do next than what they say they will do.

LinkedIn and Facebook updates are a good example of self-declared data, and it’s easy to see why this type of data is often unreliable.

When was the last time you saw somebody post on LinkedIn: “I just got fired for stealing a stapler”?

People really do get fired for theft, but they are likely to spin it on social media as: “I’m excited to report I’m exploring new career opportunities.”

This is why the Daybreak Poll outperformed the others. By paying attention to trends in sentiment over time, these brilliant researchers effectively turned self-declared data into observed data.

A better way to predict behavior

There’s a better way to do polling, and there’s a better way to do AI.

First, you must collect large amounts of timely and accurate data that is observed rather than self-declared. Then, you must analyze it with self-learning systems that continuously grow smarter over time and inject these insights directly into people’s workflows with applications that make them immediately actionable.

This is what real AI looks like, and it’s the only way to produce accurate predictions that drive better business results.

The big lesson from this election is that if we want to predict human behavior, we must stop asking and start observing.

Dave Elkington is the CEO and founder of InsideSales.com, the industry’s leading cloud-based sales acceleration technology company. All of InsideSales’ innovative products are fueled by the predictive insights of Neuralytics, a self-learning engine with more than 100 billion sales interactions and counting.

@DaveElkington