USABILITY TESTING

Summary:

Most usability testing involves finding and fixing problems as part of an iterative design process to make an interface more usable. It is typically called a Formative Usability Evaluation. In contrast, a Summative Usability Evaluation describes the current usability of an interface—as measured by things like task times, completion rates and satisfaction scores. Summative tells you how usable an interface is and formative tells you what isn't usable.

When you consider the place of usability testing in the overall product development process, then there are two main types of usability test: formative testing and summative testing. Which test you choose will have implications for the number of participants you test, the methodology you use and the way you log, analyse and present the data.

Formative Evaluation

A type of usability evaluation that helps to “form” the design for a product or service. Formative evaluations involve evaluating a product or service during development, often iteratively, with the goal of detecting and eliminating usability problems. Formative usability testing takes the role of a support tool for decision making during the beginning stages of the design process and – if applied early in the design process – provides valuable insights of where users have difficulty reaching their user goals with the product or service. In a typical project, two formative usability tests should be conducted. One of which should be in the early concept stage where the ux design may be captured as paper prototype with no working functionality. The paper prototype would contain wireframes of the ux design concept with several key usage scenarios supported in this version of the concept. The approach allows for a validation of the workflow and validation of initial ux design decisions around navigation, terminology and rough layout. A second formative usability test should be conducted when early versions of the software and hardware are available to catch issues right at the beginning of the implementation phase.

Summative Usability testing

Usability evaluation of a complete or near-complete design under realistic conditions that can be used to determine if the design meets specific measurable performance and/or satisfaction goals, or to establish a usability benchmark or to make comparisons. Summative usability testing is a Quality Assurance (QA) type of test usually performed later in the product development process. A similar usability test protocol is used as in formative usability testing but now this setup is used to do formal user acceptance testing before the product is released to the target audience. Things like think aloud protocol and hints for the user when they get stuck are left out in this kind of testing. The pass/ fail metric is the most important measurement in this test.A summative test uses additional metrics of users’ success to assess whether the product meets those success metrics and can be released from a user experience perspective into the market. 

Thinking aloud versus ‘just do it’

An important second difference between formative and summative testing is the methodology itself. With formative tests, you want participants to think out loud, describe what it is they are trying to do and let you know when they’re confused. That’s the moderator’s key role in a usability test: to listen and to remind the user to keep thinking aloud. 

With summative tests, your main interest is in the statistics of participants’ behaviour: how long do they take on a task? Are they successful? How many errors do they make? So for summative tests, a moderator’s presence is a distraction.

So, in contrast to formative tests, you’re better off running summative tests with participants working alone, either in a lab or remotely, over the Internet.

Local versus remote methodologies

Both formative and summative tests can be run in a lab or they can be run remotely over the Internet. Formative tests really need a moderator alongside the participant (either in a lab or virtually) to ensure they stay on task and keep thinking aloud.

Frequency and timing

A further difference between the two types of test is in the frequency you run them and when in the development process you run them. On most projects, you should be running formative tests monthly (or during each sprint if you’re using an agile-based approach). include paper prototypes, electronic prototypes and other minimum viable products. Formative tests with prototypes will help you evaluate the assumptions you’re making about the system and help you fix usability issues while they are still cheap to fix — rather than waiting until release.

In contrast, summative tests are difficult to carry out unless you have a working system: measuring time on task with a paper prototype is a little meaningless. This means you’ll run fewer summative tests and you’ll probably run them towards the end of development.

Presentation of findings

The final step is to present the results. For formative tests, this could take the form of a highlights video, a workshop, a list of issues in Excel or simply a meeting with observers after the last participant has left. For summative tests, you’ll want to create a user experience dashboard so that you can monitor progress over time or compare the results with the competition. This is important for benchmarking user experience and using your results to influence business decisions.

Take Away: Having these two test frameworks to refer to will help you make decisions about many aspects of planning a usability test. At the very least, it should act as a reminder that each test type is unique and specialised.

Reference Links: https://www.andsoitcontinues.com/writing/usability_reflections_summative_vs_formative_testing.html

https://akendi.com/blog/formative-and-summative-usability-testing/