User Testing In The Age of COVID-19

Charles L. Mauro CHFP / President / Founder MAURO Usability Science

What We Hear From The Field and Recommend To Clients: Today, our quarterly Usability Testing Research Insights Newsletter is focused on helping our clients and professional colleagues navigate this complex and challenging time of COVID-19. The following is our technical assessment and immediate actions for framing and dealing with this complex new problem. The world of user testing has changed in ways no one could have predicted even two months ago. But it is not all bad news. Here is what we hear from clients, professional research colleagues, and related action-taking to be considered.

Clients Are Not Canceling Studies, But Reconfiguring To Meet New Risk Factors: Based on our direct experience and feedback from corporate research colleagues and clients, at this time, development teams are not canceling lab-based user testing studies, but are working to reconfigure studies into 3 primary categories (below is a link to our article that describes how to segment into 3 categories) based on product development priorities. Notably, today our firm has more requests for proposals (RFPs) for testing programs than prior to the onset of COVID-19 in the United States. We have also recently executed a complex lab-based usability study within the time of COVID-19 in NYC. Times have changed, but critical user testing will continue under a new set of risk mitigation standard operating procedures (SOPs) and utilization of only the most experienced user research teams. The following is a shortlist of 4 key steps we recommend for starting the process of dealing with user testing in the age of COVID-19.

Step 1: Segment Your User Study Pipeline Into 3 Categories: Your team should immediately review all planned user studies and segment them into 1 of 3 categories. We recently published an article describing how to proceed with such segmentation. The article describes segmentation and user testing reconfiguration in the context of medical device user testing; however, the same approaches should be applied to user testing for products, websites, or systems of any kind that require user performance validation and related testing.

Step 2: Begin Updating Your User Testing SOPs Immediately: We highly recommend that all product development teams immediately update their SOPs (Standard Operating Procedures) to accommodate user testing in the age of COVID-19. There is no future for user testing without improved planning and risk mitigation processes. If you retain outside user testing expertise it is also important to generate a robust vendor selection criteria checklist to confirm that your user testing partners have in place updated risk mitigation SOPs as well. For those teams undertaking this process remotely, our team has found SLACK to be especially helpful in combination with the Google Docs collaboration framework. We also utilize an online database created using FileMaker Pro.

Step 3: Move Formative Studies to Professional HFE Heuristics Analysis and Other High-Value, Low-Cost Research Methods: Today, we are working with our clients to help them replace lab-based formative research with professional heuristics analysis. Professional heuristics analysis involves highly experienced human factors engineering (HFE) experts auditing a physical product, website, system, etc., for usability issues based on a validated HFE audit structure. Early and aggressive use of professional HFE heuristics analysis can identify HFE problems prior to user testing, which can significantly reduce HFE formative testing costs. We are also helping our colleagues and clients reconfigure lab-based user testing studies already in the pipeline based on the application of our SOPs for risk mitigation. This work includes collaboration with our IRB sites and testing facility vendors, who are also undertaking major updates of their SOPs. There are many moving parts to proceeding with lab-based HFE research in the age of COVID-19.

Step 4: Rethink How HFE Science Is Integrated Into Development Programs: Our clients tell us that this is a time to analyze how studies are executed and to determine if other methods, such as HFE heuristics and/or robust online data capture, can supplant early formative studies. It is clear, based on the execution of hundreds of usability testing studies over the past 30 years, that MANY usability problems with products, websites, systems, etc. uncovered in Formative studies could have been identified and eliminated through early and aggressive use of HFE heuristics analysis or other advanced user testing methods. In a surprising way, a benefit of this very difficult time may be an improvement in how HFE science is utilized to improve usability AND reduce development costs.

For more information on how to manage complex user testing programs in the age of COVID-19 please feel free to email our research team.

About the Author

Charles L. Mauro CHFP is President of MAURO Usability Science (MUS) founded in 1975. Today, MUS specializes in advanced human factors research and optimization for medical devices and a wide range of other complex human-machine systems. Over his 45-year career, Mr. Mauro has managed more than 3,000 HFE research and development projects. He is a certified human factor engineering professional (CHFP) and has received awards and citations for human factors research from Human Factors and Ergonomics Society, Industrial Designers Society of America, NASA, and the Association for Computing Machinery. His clients include many leading corporate entities and startups. Mr. Mauro has testified as an expert in product development and human factors science in more than 75 major cases. He has been an invited speaker at the USPTO, FDA, NASA, MIT, Stanford, UPenn, and many other leading academic and research institutions.