The future of UX is multisensory

The need for more beauty and physical connectedness in our digital lives.

Sensory Design isn't new. One way to describe it is that it's a multi-sensory approach to design practice. An approach that is inclusive and considerate of much more than someone’s vision abilities, but also incorporates touch, sound, and other senses. It's easy to see it as a core part of interior and environmental design, or even gaming design. ??

I had the opportunity to grow up visiting MAM and other museums in Rio de Janeiro and S?o Paulo which often showcased some of these experiences. After all, sensory design is synonymous with art installations. My takeaway when visiting these experiences was that they always felt more vibrant and real than any other form of art I experienced. ???? I felt transported to a little slice of the artist's mind and for a moment I would forget where I was. I loved that feeling and it’s what has kept me chasing new art exhibits and immersive experiences.

Yet, nothing would quite prepare me for what I would experience in Tokyo at the teamLab’s permanent exhibit “Borderless” (photo from my visit above). ?? As the name implies, the spaces they created felt endless and seamless. There, I experienced immersion to the max! I spent four to five hours there with my family, surrounded by an extravaganza of expressive video and light installations. The work felt tactile, even though the majority was limited to light effects and video projections. It was so over-stimulating that my brain had nearly melted towards the end. ??

Olafur Eliasson 's body of work has a similar effect on me, but it is much more cerebral and calming. Although his prolific body of work includes even entire buildings, many of his artworks can often be reduced to one object or room. An iconic memory for me was the hypnotic Your Rainbow Panorama from the ARoS museum in Denmark, seen below.

A common takeaway for me in both of these, and other equally wonderful art experiences, is that I left them feeling uplifted, more inquisitive and aware. ????

The questions that unavoidably follow are: Why can't our everyday apps and the web itself make me feel more like that??Why aren't moments of joy and beauty prioritized more often in UX design?

What I know is that if I were given the choice between a year's worth of new features versus more beauty, I would not hesitate in choosing the latter. Would you?

Entering my forties has been a strong reminder that life is short, and I should seek more pleasurable experiences as opposed to simply looking for ways to achieve more in my career.? In fact, I think it's important we all choose to celebrate small, yet significant moments. These can be forms of achievement too. ??

What are some of the small daily achievements you've come to appreciate?

Can a design language account for all senses?

With today's technical limitations, the most fitting answer is no. After all, we don't have smell-o-vision ???? in our mobile phones, and taste is something you can't experience through the web (yet). ?? Nonetheless, it's worth being self-critical when it comes to accounting for other senses, because it isn't just a way to more accurately communicate an idea, it is also a question of accessibility.? And if that's the case, are we doing our very best to be inclusive?

Having been incredibly close to UX teams creating design language at scale for some of today's most prolific global digital products, I know that designers strive to include guidance for a lot more than just pixels on a screen. But in practice it can be difficult to find radical implementations in the areas like sound design ?? and haptics for instance.

This is true not only for operating systems and their apps, but also the web. The more we have standardized web experiences over the years, the less expressive it has become. It is clear that the internet was far less usable and helpful in the turn of the 21st century, but I recall stumbling upon websites that were far more inspiring than today's lookalikes.

As I imagine the experiences that await us in the next decade, one of the topics that rises in importance is the role of senses beyond sight in shaping them. The last ten years in augmented and virtual reality have pushed us to be more thoughtful in that respect, and inspired us to think about designing for greater connectedness in our digital lives. But when the promise of the Metaverse becomes a reality ??, I hope to see a more sensory approach to immersive design being prioritized. Until then, I like to think there are small steps we can take in today's popular experiences to make them more dynamic, expressive, and ultimately more human. Below are just a few reactions around in touch and sound, but welcome you to share your ideas and interests in comments.

Touch: prioritizing feeling through sterile sheets of glass

I recently learned that touch ???? is thought to be the very first sense that human beings develop. Early in my career, my fascination for textures took me through a multi-year exploration of various art mediums: from exploring photography, collage, ink drawings and acrylic paintings, to more sculptural experiments such as creating mobiles out of found objects, and even painstakingly layering strands of wool and pigment to form organic objects.

As a kid, I found it irresistible to pick up things during a walk in the woods, or play with sand at the beach, and as this little lost bear will attest, some of that obsession for textural scenes still follows me into my photography work.

Today most of us spend hours reading the news, working and interacting with each other through phones, tablets and PCs. And while there is a hand-full of great examples of haptics out there (hello Pinterest??, I see you), for the most part we swipe and tap without any distinction between a plain button and a richly textured photograph. There is just so much more to touch than simply replacing a mouse.

What if we could deliver micro-doses of haptic feedback in precise alignment with the perceived texture of an image as if physically simulating its details. The same can be said about the contour of a button, or the depth of a 3D object as you interact with it. Could this approach help people connect better with images from a past memory and more easily learn how to use a new app? Possibly. But at a minimum, it could help those with impaired vision better experience content, which would already be a major win.

Battery life is probably the main reason we haven't tried something along these lines, but if they could help us enjoy and understand things better, perhaps it would be a good trade off. The truth is, I think the explosion of data and content we have seen in recent years has made us biased towards UX that is optimized for volume, not quality: endless content feeds versus focusing on what matters to us.

Sound: prioritizing joyful connection and memories over noise

When it comes to product sound design, it is easy to prioritize critical notifications such as system start up, ringtones, errors, alarms and so on. These are computer-to-human interactions, sounds made by our apps and devices when they need our undivided attention.

I'm interested in a more comprehensive approach that takes the entire system into account and balances the computer and human needs more intentionally. For one, I think that sounds that aren't likely to be used by everyone, but that contribute to making those experiences more inclusive are just as important. In this more universal approach, human-to-human interactions rise to the top. I imagine this category going beyond the all too important team discussions, 1:1 conversations, informal chats and memorable gatherings with friends and family.

One of the under-developed areas in audio is around memory creation. The time we spend with one another has so much texture and nuance worth capturing, especially when we are out and about, as in this photo I took with my family during a trip to Osaka. Can we capture the laughs, impromptu conversations, music, iconic city sounds and urban noise that make memories truly three-dimensional?

I realize this might sound invasive to most people, but if trust and privacy weren't an issue, then what could we do to enhance those memories with sound? I'd love to see more product-makers putting energy into tools that can make us come even closer to remembering and re-experiencing our adventures. Something I've become more in tune with since reading "Die with zero" by Bill Perkins .

While investments in speech for instance have been a transformative addition to our everyday tools (thank you Apple TV ?? voice search feature), I'd argue that our industry has been so focused on implementing multi-modal interaction models that we diluted our focus away from making the basics of touch, haptics and sound just incredible.

These are just a few of my personal reflections and interests around the experiences we are all collectively building. I find it's important we encourage each other to go deep and do better in specific areas as opposed to just doing more across everything.

Quality over quantity! ??

If you're still with me after this long read ??, thanks for your attention. I look forward to your ideas and feedback in the comments.