Can we analyse complex human states using computer vision?

As digital signage continues to innovate with AI technologies such as computer vision and deep learning, many retailers consider investing in them, as it can go so far as measuring content performance by tracking behavior in-store.

The benefits of having data are quite clear, but can computer vision alone unveil emotions of complex human states? What data on behavior can it capture for AI to understand, then?

Emotion tracking

Humans are hardwired to perceive emotion at a glance. One can immediately tell someone is sad with a frowning mouth and downward eyebrows.

For years, we’ve been training AI models to do the same. Using computer vision and deep learning, data on emotions have been fed to learn and be able to tell happy from angry. With images or videos, facial cues can be captured, such as:

• Eye and eye corner shapes
• Lip curls
• Eyebrow raises
• Nods

It is said that retailers should acknowledge that cognitive reasoning isn’t the only trigger to making decisions. Emotions hold the same power or even much more that it drives impulses, including purchases.

The case of in-store signage

As I’ve mentioned in my previous write up, digital signage today are now equipped with computer vision to anonymously ID people who view the content shown, tracking demographics, and facial attributes, among other data.

Now with this capability, based on the viewers emotions or moods while viewing the content, ads can be optimised to enhance their in-store experience, which is a gold mine for retailers.

As my team are working on our computer vision model, we are challenged to seek better results of our machine training, and can’t help but wonder whether the computer “seeing” is enough to draw the conclusion on human states.

Data gathering methods

According to emotion AI pioneer Affectiva, they’ve developed a massive database of nearly 6 million faces using computer vision to analyse, test and train their algorithm for accurate emotion tracking. 

To diversify the data, the faces were from 75 countries. So far, the insights are allowing building rich emotional profiles based on different cultures to better understand and identify human emotions or moods.

Personally, I find this already impressive, but I’ve also seen a video of Professor Rosalind W. Picard, ScD of MIT Media Lab, tracking can be done with wearables, too. They tracked the nervous system with gloves that glowed differently for different moods.

Another study said that smartwatches and wristbands are also used to gather heart rates and temperatures, among other data to identify human emotions. Voice recognition is another thing.

Although there are many ways to track human emotions, savvy retailers who invest in technologies like computer vision are already able to unveil the emotional layer of consumers and open up an opportunity to personalise and humanise the experience.

Our computer vision model works well in “seeing” such emotions. However, in terms of more complex human states, all the methods combined (seeing, hearing, and feeling) can probably provide more accurate and detailed results. I’ll continue to watch this space and see how it unfolds.