Vision-based Interaction: Advantages, Benefits, and Best Practices

The potential for vision-based interfaces is incredibly broad across consumer, enterprise, industrial, medical, transportation, military, and other industry verticals. The size of this opportunity can be gleaned from examining one subsegment of applicable consumer devices – everyday technology devices that consumers interact with via touchscreens or tactile input. Popular devices that utilize touch-centric interaction include smartphones, tablets, smart TVs, and laptops. According to Parks Associates research, US internet households have an average of 16 connected devices, including 11 from the mature CE category, three from smart home, and two from connected health.

• Smartphones – Users can utilize vision-based inputs when touch and voice inputs are not possible or desired (e.g., wet/dirty hands or in a noisy environment). Beyond gaze-based typing, use cases could include answering or declining calls, opening a streaming music app and starting playback, taking a photo, recording video, and more.
• Automobiles – Vision-based interaction can be used in combination with advanced driving assistance systems (ADAS) to monitor the attention level of drivers: if their gaze is distracted away from the road, if they are falling asleep, and if an alarm/alert is necessary. VBI can also enable enhanced maps and navigation features.
• Connected home – Today’s smart homes are dominated by smartphones, smart speakers/displays, and voice control as inputs. Vision-based interfaces can allow lighting to be activated, devices to be turned on, and more, with just a gaze – and no voice or tactile input needed. VBI could even enable device pairing between interoperable devices, linking them with a gaze.
• Virtual reality – The gaming, social media, and entertainment markets are becoming ever more immersive, with tech giants building towards a “metaverse” concept. Vision-based interaction is becoming more common in the VR market, with eye tracking as most practical for foveated rendering and gaze-based control.

Foveated rendering is based on the mechanics of human vision, where resolution is highest in the eye’s fovea and lower in peripheral areas. Using this technique reduces the computational load by only rendering high-resolution images in gaze direction and providing lower resolution elsewhere. Foveated rendering is more realistic to human vision and saves processing power by only rendering in detail what the user is directly looking at. The solution provider can also collect data, via the VBI solution, on eye movements, length and subject of gaze, and the main objects/people of focus by the user. They can leverage that data to improve the experience and optimize content delivery by transmitting high-quality media to areas of focus and saving power through lower-quality rendering in the periphery.

Currently, gaze-based control is used particularly for aiming and target shooting in VR games.

The use of vision-based input and interaction as a control paradigm is a natural expansion of existing modalities of input. Vision-based interaction has the potential to drive several substantial benefits for both end users and industry, and this potential can be maximized with proper considerations during the conceptualization and implementation/execution stages.

Vision-based interaction brings a variety of benefits and advantages due to its ability to utilize a fast, natural input device – the human eye – to accomplish device control and input without having to commit the use of the user’s hands, requiring a tactile input surface to be within reach, or requiring clear, audible voice input.

• Speed and efficiency – Vision-based actions can often be faster and more efficient than a touchscreen press or keypress on a keyboard. The ability to complete a task with merely one’s gaze, if implemented correctly, can bring increased speed to common tasks or workflows.
• Uninterrupted manual focus – Use of the eyes as an additional means of input enables a heads-up mode of operation and allows the user’s hands to remain in place for other tasks rather than disengaging to engage in tactile input, which can enhance safety and focus on important tasks.
• Hygiene and epidemiological safety – When direct contact with control surfaces is to be avoided, vision-based input can enable a higher level of safety for the immune-challenged or within research environments, pharmaceutical or medical device manufacturing, and other hygiene-sensitive use cases.
• Accessibility and expanded choice of interaction - Depending on the use case, vision-based input can serve as an augmentation to other interactions, a fallback/backup input method, and a way to expand input choices. This can be useful when a user has their hands occupied, the user has limited mobility or capability with their limbs, or any other circumstance where a single interaction modality is too limiting or too unaccommodating of user and/or situational variables.
• Broad applicability – In its most basic implementation, the principle of vision-based input can in essence be applied to nearly any device with a camera. Provided there is enough processing power to engage in computer vision processing, that device can potentially be operated without physical touch using vision-based input.
• Authentication and personalization – visual-based control technologies that can recognize unique irises may also be used for authentication, particularly for entertainment personalization. Usage of entertainment devices where multiple users are present differs from usage by a single user. By identifying the users present, features such as recommendations and other profile-oriented functions can be optimized for multiple users, enhancing the user experience.

This list is a limited set of advantages out of the many that are enabled using vision-based interfaces. The continual evolution of computer vision has enabled considerable advances in the quality and capabilities of vision-based interactions – but these capabilities must be teamed with intentional design and execution to deliver their full potential.

A poor user experience, especially early in deployment, could greatly inhibit VBI adoption and usage – it is critical that VBI is paired with a well-designed and intuitive user interface.

This is an excerpt from Parks Associates research white paper, Vision-Based Technology: Next-Gen Control. This whitepaper, written in partnership with Adeia , examines selected consumer technology devices that can potentially be augmented using a vision-based interface, how vision can complement or replace the common touch-based interactions, and the benefits of vision-based interfaces to both consumers and industry. The whitepaper also examines best practices and real-world examples of vision-based interfaces and interactions.