Empathetic technology and the end of the poker face

As humans, we like to think we keep our internal states, well, internal. We have a “poker face” that helps us shield certain thoughts and feelings from the outside world. Or so we think.

The truth is, we give away our internal states all the time. The way we breathe, the way our bodies distribute heat, the timing dynamics of our speech, the way our pupils dilate — all expose the emotions and feelings we think we are keeping to ourselves.

I recently spoke at TED about how technology that reads and integrates the personal signatures we give off and exchange will be a new part of how we interact with each other and our technology. It will mean that sharing how we are feeling — our authentic emotions — may at times be out of our control, and might even happen without us knowing. I’m sure this sounds scary to most of us. It does to me. But I truly believe there are more ways this type of “empathetic technology” can improve our lives than do harm — and that it is worth the effort to solve the questions and protections that make us uncomfortable. Right now, we need to know this can happen, and think about how we want to shape that future. It can change the stories we have to tell. It will change how we tell our stories.

The signatures our bodies and behaviors give away is a type of personal data, even if it isn’t what comes to mind when we traditionally think about our data. We leave data behind everywhere we go — every interaction. At first this may sound alarming, even overwhelming, because the conversation around personal data has become so fraught. Nonetheless, it is important to recognize that sharing personal data with our technology doesn’t need to mean compromising our privacy. 

But consider this: gathering personal data may actually help our technology protect our personal lives even more. And it definitely can help technology better serve its benefits to each of us with all our unique and wondrous differences.

What is empathetic technology?

Most technology makes decisions for us. Developers decide when and in what situations technology should react or modify a parameter to change its state. We may have some say in when that parameter should change—and how, but it’s basically like setting a rule that says, “If A, then do B.” Empathetic technology will change that. It will let technology learn to do “C” when “A” is only one of the variables driving the decision, and other key weighted inputs are coming directly from the state of the individual.

Imagine a thermostat. Ten years ago, most thermostats let us set thresholds or timelines that controlled the storyline of our spaces. We might have created simple rules like turn on the heat if the inside temperature dipped below 65 degrees, or crafted a tailored script that changed the heating or cooling of the space as a function of the day and time of day. Now our thermostats are remarkably more intelligent. They learn our spaces and our actions. They learn our behaviors and take into account key contextual information about how we are interacting in our environments — or whether we are even there at all.

This has enabled us to save both money and environmental resources. Now, let’s take it one step further. In all those scenarios, the thermostat may be intelligent to the abstract behaviors we have in our spaces, but it does not know whether we are cold or hot. It certainly does not know if we are failing to be cognitively effective at a given moment in time, and what optimal temperature we should be at given the context and activity we are undergoing. Are we sleeping? Working? Trying hard to stay awake to finish a project? The current connectivity of sensors is moving us in this direction. But the best answer will be one in which our technology fluidly adjusts the dynamics of the space to optimize for our needs based on insight to our internal experience. 

Take a look at the interesting efforts from the MIT startup - EMBR.

This is empathetic technology. It tracks a number of biometrics and real-time behaviors — the subtle things our bodies do that give away our emotions, how we feel, how hard our bodies and brains are working — to understand our internal state. Then it uses this information to inform how a device responds or adjusts to more naturally and personally interact with us and our lives. Applied thoughtfully, empathetic technology can help us interact more richly and successfully with each other and our technology.

It enables technology to anticipate and act as a true partner. It can, for example, reduce our stress by enhancing the information our brain might be struggling to hear, see, or remember. It can find authenticity in our human interactions — maybe even new ways of falling in love! It can act as an objective arbiter to the success of information transfer when we interact with each other socially, at work, or in education: a transparent way of knowing whether someone really does get what we mean. I know I could definitely use that. But it also can help us know when someone is feeling or thinking something we ought to pay more attention. In a world where our attention is capacity limited, and possibly our most valuable commodity, empathetic technologies can help us connect with each other in the most human ways when we are feeling the most vulnerable.

I actually used empathetic technology during my TED Talk to gauge the impact of an artist’s intent on the audience and to see what happens when we change that intent. Have you ever tried altering the sound during a super scary movie? Watch what happened when we did that live.

The amalgamation of sensors (e.g., microphones, thermal imaging, measures of exhalant, cameras, etc.) around us and in our environment is more capable and prolific than ever. Between ever-growing processing power and smaller silicon to support this processing, we are no longer limited to gleaning insight from a single sensor’s data that is somewhat noisy and ambiguous in the description of our internal experiences. Instead, we can gather rich stories at high speed by computationally combining data from multiple sensors, creating an insightful and actionable picture of our internal experiences. The capability to understand and react in this way is enabled by machine learning.

Empathetic technology and machine learning

As it pertains to this topic, machine learning will help empathetic technology improve at drawing conclusions and disambiguating the data it gathers on users’ internal states and their contexts. Because empathetic technology will need to be able to draw conclusions from a broad amalgam of data across multiple sensor inputs, it will need to know how to prioritize or weigh different data points over others to reach an effective conclusion.

As an example, a current piece of technology may not be able to differentiate if a lot of background noise means you are running late to catch your flight in a busy terminal, or you are at a loud party having a lot of fun. But machine learning will be critical in two ways. It will help devices develop underlying probabilistic likelihoods for the contextual environment of your “noisy background,” and scene intelligence that is even better disambiguated by including the GPS coordinates of the microphone capture. The airport would be a sure point of helpful resolution. Now join that with machine learning algorithms to help weigh and prioritize that information against, say, your pupil dilation picked up by a pair of glasses you were wearing, and it can tell whether you are under stress or highly engaged and having a good time. The technology has to be able to synthesize and weigh the different data to make a probabilistic decision about how it should react to best improve your experience. Machine learning allows it to get better at doing that over time, and empathetic technology allows it to get better at doing this for you effectively.

Specifically, weighing information to make an inference about how you are feeling is called “probabilistic learning” — as in, making assumptions based on probability. Your brain actually does this too. It makes inferences based on your prior life experiences. For example, if your brain puts a heavier weight on low frequency sounds, you may hear “laurel” in the same sound bite where others hear “yanny.” That is your brain weighing different pieces of information to make a probabilistic decision that uniquely affects your experience of a single piece of content in the world.

The end of “one-size-fits-all” technology

Most technology today is remarkably un-attuned to our differences.

Consider the way we perceive sound. How we experience a sound is a function of three things: our underlying peripheral physiology, our low-level sensitivities — like whether we have any hearing loss (most of us do starting in our early 20s), the unique filter our bodies, heads, and convolutions of our ears impose on every sound we hear — and our past exposures, like our native language and environmental influences. All of these can drive changes in our neural sensitivities that are unique for every one of us. They also are influenced by demographics like gender, genetics, or even the urban or rural aspects of where we grew up. Our individual experiences, needs, and bodies affect how we experience any element of the world at a given moment in time.

More data is not necessarily better. It’s all about the quality, dimensions that are extracted, and how data is combined and weighted to drive a decision. Our brain does this kind of weighting and evaluation all the time. Our technology needs to do the same. The more personal data that technology has, the better it can be informed by our internal state — which inherently captures how well the technology is helping each of us individually, in the best way it can, to customize its behavior to our needs.

Yet headphones are just headphones. They’re designed for everyone, so they’re designed for no one. Through the same one-size-fits-all set of headphones, we each hear sound differently.

The same goes for all kinds of technology. Everything we buy, from cell phones to cars, is one-size-fits-all. We can customize settings, but without understanding our internal states, our devices don’t have the context to make meaningful and timely adjustments to be truly personalized. They may have the capabilities in features, but the interface to reach those features and interact with them is broken — it needs to come from the internal experience of the user. Empathetic technology has the capacity to close this loop. It can transform even the most innocuous technologies to be better optimized agents in providing a successful experience for each of us.

One way empathetic technology has a particularly interesting implication is for entertainment and content delivery.

Imagine you are watching a Warriors game on a TV or display in your living room during a party. If that TV or display contained empathetic technology, it could notice — without you having to tell it — that you’re having difficulty understanding the announcer while everyone is busy having their own conversations. Your pupil gives away your cognitive effort, and the context of you in your specific environment can be understood through microphones and visual behaviors captured through analysis of the scene statistics. When those are combined, your technology can have a fairly accurate and insightful reaction to improving your unique experience with the content.

Because technology like Dolby AC-4, for example, can deliver a variety of audio tracks in a broadcast, your TV, or a device you might personally wear like an earbud or headphone, could automatically increase the contrast between the dialogue and the background sounds to make it more clear to you, and only you, without you even having to ask.

And this is just one example. Empathetic technology could empower content creators to deliver different experiences to different viewers in a variety of ways. Author and philosopher Leo Tolstoy defined his perspective of “what art was” by whether there was a shared experience between the creator and the person on the other end. Imagine today what he might think were he to know his philosophical musings were objectified realities.

At the very least, as developers that create part of the ecosystem that enables translation from the creator to the viewer, we can change and enhance what dimensions we use to know that the intent of the creator has been effectively shared across mediums, spaces, and individuals as closely and accurately as possible.

Empathetic technology has the power to bring us together

If we look toward the near future of promised augmented reality (AR) technologies, we are presented the idea of devices enabling experiences that overlay objects and textual information onto the world around us – a type of virtual “semantic encyclopedia.” Every time we see an old friend, for example, we would also see data overlays with their descriptive relevancies — name, background, details — from our last interaction.

It may sound highly useful in theory, but in reality is probably not the right answer for what technology should provide to help us be more effective at interacting with each other or our surroundings.

Any time we use technology to help us interact in one way it usually comes at a cost for how we interact in another. Consider how good you might be at navigating a new city that you have only ever navigated using a digital mapping app.

Not so good – or at least I’m definitely not. Compare that to a city you’ve walked, run, or driven in without guided direction plotting your every move. I would never part with my digital mapping assistants, but we do have to be careful how we use technology to enhance us. Empathetic technology makes use of what we are actually feeling and responding to — to help us have richer experiences with each other and from our technology. We want to make sure that our augmented worlds of the future are aware and listening to our internal worlds to help us enrich the sentiments and experiences that do enable richer emotions, caring, and empowerment, and strengthen our capacity to engage them.

Consider the high school guidance counselor having a standard check-in with an outwardly positive student. Empathetic technology has the power to make this interaction more meaningful. It may enable the counselor to realize the student, who doesn’t wear their difficult emotions on their sleeve, is actually having a deeply hard time – where reaching out may make a crucial positive difference.

Empathetic technology does not just mean a more emotionally present relationship with our technology. It has the power to enable more emotionally present relationships between each of us.

The question of privacy

With any new technology, empathetic or not, the matter of data privacy is paramount. It is important to define the rules that govern what data is collected, how it is stored, and with whom it is shared.

The additional wrinkle with empathetic technology is that, because it has access to information about our internal state, it’s tracking personal data that users may not be consciously aware they are sharing. For this reason, it’s important that we talk about empathetic technology now, when it’s still in its infancy, so we can create a framework for regulating how this type of data is stored and used.

Lost in the conversation around data privacy is an equally important concept, in my opinion: transparency.

Users understandably become upset when their personal data is misused or shared with third parties without user consent. As a result, users become more protective of their data and more wary of the technology they use.

Every conversation or behavior we share, or overtly do not share, with someone is used by them to learn about us and make decisions about our lives and their own. We are constantly exchanging personal data in ways that we need to have a richer conversation about when it comes to contextual implications.

In the case of empathetic technology, more of the right data is better. Enabling technology to create a richer picture of our internal state means allowing technology to capture personal information that it can use to develop a more accurate probabilistic model for what we’re thinking or how we’re feeling. We don’t need to be afraid of sharing this data with our technology, as long as our technology is perfectly transparent with us about how it’s used and we have trust in how our data is encrypted.

This means it is unambiguously clear to the user what information a device is tracking, how much of that data is stored on the device versus in the cloud, what data is being encrypted and how, and with what third parties that data is being shared and why. Recent legislation in California is taking a proactive approach to protecting consumer privacy in the exchange with technology. Nonetheless, we need to also evolve and reframe how we consider transparency and privacy in a way that doesn't introduce more demographic bias and compromise in the benefits of our human/technology relationship.

Without this level of specificity, users cannot trust their technology. And their trust has been abused in the past, so they’re rightfully wary. While regulatory efforts are moving in these directions, we need to establish a richer conversation between those driving legislation and those evolving the technology that will ultimately be most impacted or abused.

Giving up the poker face

So maybe we’re not as coy as we think we are when it comes to the tells and reveals that give up our internal state, but that’s ok. The authenticity in sharing our emotions and feelings paves the way for new technology that knows us and adapts to our mood and context.

And the promise in this technology is huge. Already, “hearable” technology as simple as the smart speaker in your home could pick up on changes in speech patterns that would allow us to detect diseases like Alzheimer’s and dementia, potentially up to 10 years before a clinical diagnosis.

However, the promise of empathetic technology only comes if we are educated on the benefits. We need to understand how it can already be enabled today, and how we create the right engagement and awareness to determine where, when, and how we will share parts of our inner lives with our devices. This is a first step I wanted to take with my talk at TED — to shine light on a reality that an amalgamation of current and potential technologies paired with the right computational framework and consumer trust in data protection and transparency has a positive human potential that is too great to pass by. And that we need to engage now to assure that this capacity for human connection and insight is developed with knowledge and protections with insight towards the impact of what we are building.