Facial Recognition & Journey Mapping for Improved Usability Testing

A reflection on the application of facial recognition for the development of emotion maps for usability testing analysis.

Abstract

This paper will seek to discuss the application of facial recognition technologies in usability testing and the analysis of these tests. This technology was utilized by the author as an augmentation to traditional usability testing methodologies during a pro-bono assessment of the government website www.OSTI.gov. The author found that the features provided by the facial recognition software assisted in the usability testing as they allowed for the creation of emotion maps from a more empirical source than the researcher’s own judgment and/or perceptions of user’s emotional states during the testing. Application of this technology could be improved further by modifying the testing environment to record user’s faces separately from the usability testing software’s own recordings, establishing baselines of emotion before tests to calibrate the facial recognition software, and holding users to a strict time limit on each task. Overall, it is the opinion of this paper that facial recognition software that focuses on emotion detection can be a useful addition to existing methods of usability testing.

       Usability Testing   

Usability testing is a popular method for inspecting the overall quality of a site’s design and for determining how the system performs with representative users. Usability testing is broadly defined as: “Watching people try to use what you’re creating/designing/building (or something you’ve already created/designed/built), with the intention of (a) making it easier for people to user or (b) proving that it is easy to use” (Krug, 2009). This may sound like a broad definition but, that is intentional as usability studies are adaptable and the methodological foundation permits for broad execution. Traditionally user testing is typically done in a controlled environment. This can either be in a full-fledged UX laboratory or in a small conference room or similar environment. The users are typically given a set of tasks to complete and an observer watches how they accomplish these tasks. The observer can either watch the user accomplish the tasks silently and then ask questions retrospectively or can instruct the user to think out loud as they move through the tasks. Most usability tests also include a pre-test and post-test survey to assist in the gathering of additional data from the user.

During the testing, there are a variety of ways to record users. For virtual interfaces on desktop or laptop computers popular recording software records not only the user’s interactions on the screen (clicks, scrolling, navigation, etc) but also the user’s voice and face. This recording is often reviewed by the researcher for additional insights after the testing is complete. The type of analysis largely depends on the goal of the testing. Most usability tests will look at the amount of time each user takes to complete each task (Time on Task or TOT) and will also investigate a user’s self-reported difficulty rating of the task. These are useful quantitative measures that permit for a general baseline to be established. Qualitative analysis typically takes the form of a researcher watching the recordings and referring to their interviews and notes during the test session in an attempt to empathize with the user and identify pain points or stumbling blocks as the user attempts to complete the tasks.

Journey Mapping

A novel method for assisting in analysis beyond time on task and difficulty ratings that also investigates aspects of the user’s emotions during the test session can be journey mapping. Journey mapping can be defined as a “visualization of the process that a person goes through in order to accomplish a goal tied to a specific business or product” (Gibbons, 2017). Journey mapping can take as many different forms as usability testing but, ultimately it tracks the users as they move through an experience. Typical journey maps will include the user’s emotions, the tasks they seek to accomplish (sometimes divided into phases), the actions they take to accomplish these tasks, and any pain-points or “aha moments” they encounter along the way. An example of a user journey map can be seen below in

Figure 1. - By Connor Esterwood

Journey maps in combination with other results from usability testing are valuable resources that can be provided by researchers that offer insight into how to improve a system or the existing baseline performance of a system. One discrepancy in journey mapping however that leans on researcher’s experience and subjective perceptions however is the emotional state of users as they move through a system. This subjective aspect of journey mapping may be a source for some skepticism on behalf of those to whom a researcher seeks to provide this data. Journey mapping is undoubtedly valuable to many but, the further refinement of this method is worthy of investigation. The application of new technologies, especially in facial recognition, may assist the researcher in providing a less subjective analysis of emotional states.

Facial Recognition

Facial recognition is an emerging technology that has seen rapid development over the last few years. Modern application of facial recognition software and technology vary greatly from authorizing access to financial services to permitting for augmented reality filters on an individual’s smartphones. Research utilizing facial recognition also varies significantly as well. Most of the literature on the subject is focused specifically on improving the technology’s performance. There is however growing application of facial recognition technologies in other research areas beyond the development of the technology itself (Martinez & Valstar, 2016).

 An example of two such studies focused on marketing and communication of branding. The first study presented a method for classifying the success of online video ads based on facial responses and found that there was indeed the potential for such a system (Daniel McDuff, El Kaliouby, Senechal, Demirdjian, & Picard, 2014). The second study also investigated ads but looked more specifically at the role of ads on purchasing intent utilizing facial recognition as a method of predicting the impacts of these ads. The study clearly demonstrated that “a reliable and generalizable system for predicting ad effectiveness automatically from facial responses without a need to elicit self-report responses from the viewers” is not only feasible but effective (D. McDuff, Kaliouby, Cohn, & Picard, 2015).

In addition to studies that utilized facial recognition to focus on marketing and advertisements, facial recognition technology has been utilized to predict movie rankings from audience behaviors (Navarathna et al., 2014) and even to estimate the attentiveness of people as they watch TV(Takahashi, Naemura, Fujii, & Satoh, 2013). These studies in combination with the studies presented earlier combine to make a strong case for the potential application of facial recognition as a means of augmenting existing methods of research. Specifically, this paper sees a strong potential for this technology to augment traditional user journey mapping techniques by providing additional insight into the emotional state of users as they move through an experience.

Using Facial Recognition for Journey Mapping

Journey maps contain a wealth of information and offer useful visual presentations of data gathered from field studies or usability testing sessions. Facial recognition has been shown to be a useful tool for gathering information on emotions as a user experiences or interacts with an interface, product, and/or service. Utilizing facial recognition to assist in the creation of user journey maps is not common practice in user experience research. This may largely be due to either the state of facial recognition as an emerging technology and/or simply the high price of facial recognition software suites. Regardless of these hurdles, facial recognition has the potential to provide a more empirical foundation for the construction of journey maps. The specific segment of the user journey map that can benefit from the application of facial recognition is the emotional state of the user as they move through the tasks. This is currently developed by observation and subjective analysis. Utilizing emotion detection from facial recognition, we can establish a more empirical state of tracking the user’s emotions.

To test the viability of this approach, the author has utilized facial recognition software to analyze face recordings from the usability testing of the website www.OSTI.gov. These recordings were edited from their original to contain just the user’s faces and not the screen recordings. This editing was conducted via the iMovie application’s crop video feature. The resulting edited recording was then uploaded into Noldus Face Reader application at the University of Tennessee’s user experience laboratory. Once these files were uploaded, they were analyzed utilizing the software resulting in data providing emotional states of users over time. This data was then coded in Microsoft Excel where sadness, and anger were given a -1 ranking and happiness and surprise was given a +1 ranking. The subsequent rankings were tabulated and then presented in the form of line graphs where the x axis was time and the y axis was the emotional state. An example of this can be seen below in figure 2. 

Figure 2 – By: Connor Esterwood

Once this data was gathered it was integrated into a user journey mapping structure where tasks and interactions were associated with and graphed according to time when they occurred or were completed. This graphing, once completed, resulted in a completed user journey map. This can be seen below in figure 3. It is worth noting however that figure 3 presents a very condensed version of a journey map and that most journey maps are significantly more extensive and inclusive of more actions over time. This map does however appropriately present the concept of how this data can be utilized in journey mapping.

Figure 3 - By: Connor Esterwood

 Recommendations for Future Study

Utilizing facial recognition for emotion detection is not a common approach in usability testing and journey mapping. As such, there is little written on how to conduct these tests in a way that ensures high quality. The author has found that several aspects of his usability test could have been modified to better facilitate integration of facial recognition. Firstly, ensuring that users are limited to a strict time limit would ensure that averaging the emotional states to combine them into one single emotion map is possible with minimal variance. Second, the installation of a second camera for recording exclusively the face is recommended as cropping and editing form the other software drastically decreases resolution. Third, taking time to calibrate the facial recognition software via pre-recorded emotional states would have permitted for a higher level of accuracy in analysis. Finally, ensuring quality lighting on the user’s face would also permit for a higher level of accuracy in analysis. These recommended changes to the testing conducted by the author are easy to implement and should be applied to future studies seeking to investigate the potential of integrating facial recognition into usability testing.

Closing

The current established methods of journey mapping do not include facial recognition or emotion detection, the integration of this technology holds great potential. A more robust investigation of how one may integrate facial recognition in user experience methodologies in general should also be pursued. It is possible that this technology may hold the potential to create a less subjective assessment of interfaces. An additional area where there is great potential for technological augmentation to traditional UX methodologies is in the field of Brain Computer Interfaces (BCIs). These BCIs can map electrical activity in a user’s brain. This mapping can reveal a multitude of information on what the user is likely feeling as they move throughout an activity.

Overall, there is a lack of literature investigating how new technologies may augment traditional methods of UX testing. More experimentation is needed and as the price of these technologies falls, the potential for integration into existing UX frameworks and methods becomes larger. Truly knowing the user is always a slippery business and UX methods have evolved consistently to better get to know users. It is the opinion of the author that there is ample potential for yet another evolution of methods on the horizon. Using new and emerging technologies in combination with what we have learned and practiced traditionally, we can get to know users at an even deeper level.

References

Gibbons, S. (2017). UX Mapping Methods Compared: A Cheat Sheet.  Retrieved from https://www.nngroup.com/articles/ux-mapping-cheat-sheet/

Krug, S. (2009). Rocket Surgery Made Easy: The Do-It-Yourself Guide to Finding and Fixing Usability Problems: New Riders.

Martinez, B., & Valstar, M. F. (2016). Advances, Challenges, and Opportunities in Automatic Facial Expression Recognition. In M. Kawulok, M. E. Celebi, & B. Smolka (Eds.), Advances in Face Detection and Facial Image Analysis (pp. 63-100). Cham: Springer International Publishing.

McDuff, D., El Kaliouby, R., Senechal, T., Demirdjian, D., & Picard, R. (2014). Automatic measurement of ad preferences from facial responses gathered over the Internet. Image and Vision Computing, 32(10), 630-640. doi:https://doi.org/10.1016/j.imavis.2014.01.004

McDuff, D., Kaliouby, R. E., Cohn, J. F., & Picard, R. W. (2015). Predicting Ad Liking and Purchase Intent: Large-Scale Analysis of Facial Responses to Ads. IEEE Transactions on Affective Computing, 6(3), 223-235. doi:10.1109/TAFFC.2014.2384198

Navarathna, R., Lucey, P., Carr, P., Carter, E., Sridharan, S., & Matthews, I. (2014). Predicting movie ratings from audience behaviors. Paper presented at the Applications of Computer Vision (WACV), 2014 IEEE Winter Conference on.

Takahashi, M., Naemura, M., Fujii, M., & Satoh, S. (2013). Estimation of attentiveness of people watching TV based on their emotional behaviors. Paper presented at the Affective Computing and Intelligent Interaction (ACII), 2013 Humaine Association Conference on.