What’s the difference between formative and summative usability testing?

Usability tests are used to discover the effectiveness of a design and evaluate the ease-of-use of a product. There are a wide variety of usability testing tools to help you get started. When learning how to conduct usability tests, you'll need to understand the types of evaluations you can use, and what type of data they collect.

When it comes to usability testing, there are two types of evaluations that can be conducted during the product design and development lifecycle: formative and summative usability tests.

Both types of testing are performed to understand and improve the usability of a design, but they're performed at different stages of the development process. In order to know when to apply each method, we need to start by understanding their strengths, weaknesses and differences. 

What is formative usability testing?

Formative usability testing is done early in the product development cycle to help inform the product’s design. It refers to testing that answers the why and how of a design’s usability. Formative usability testing seeks to understand why something isn't working and involves iteratively evaluating a product during design and development.

The goal is to detect any issues and eliminate usability problems before a product is fully developed. Formative usability testing takes the role of a support tool for decision making during the beginning stages of the design process, which helps to discover insights and shape the product’s direction. It tends to be more qualitative in nature. We can see how users actually experience the design, see where and why they get stuck, and hear what they say.

The data collected during formative usability testing is observational in nature. It's crucial to observe and understand the users’ thought processes and their actions resulting from them.

Why conduct formative usability testing?

Formative usability testing heavily influences the design decisions and is considered to be an excellent tool to figure out which design features are useful and which are not.

It’s recommended to conduct at least two formative usability tests in a design cycle.

• The first in the early design concept phase with a wireframe prototype that has no working functionality. This approach allows for a validation of the workflows and the initial decisions around navigation, layout and terminology used.
• The second formative usability test should be conducted on a Minimum Viable Product (MVP) with some functionalities ready just before the development phase.

If we don’t conduct formative usability testing during the design and development lifecycle, designers run the risk of designing something that isn't usable.

Here are some of the defining qualities of formative usability testing:

• Used during the design and development phases
• Seeks to improve the product design
• Can be conducted anywhere using remote, moderated, or unmoderated testing
• Includes tests performed on wireframes and MVP designs
• Typically conducted with 5-8 users

What is summative usability testing?

Summative usability testing is an evaluation of a product with representative users and tasks designed to measure usability (defined as effectiveness, efficiency, and satisfaction) of the complete product.

Summative usability testing is usually performed later in the product development process when a product is fully developed. The main purpose of summative testing is to evaluate a product through defined measures. It uses UX metrics of users’ success to assess whether the product meets defined usability success metrics and can stand up to competition. They act as a final validation where usability issues have been identified and addressed.

Why conduct summative usability testing?

Summative evaluation tells us how usable an interface or a product is. If summative usability testing isn't conducted before a product is released, designers won't know if an interface or a product has any problems or not.

Summative usability testing is used to establish a usability benchmark or to compare results with usability requirements. The metrics recorded in summative usability testing reflect real user interactions rather than perceptions or feelings.

Usability requirements are defined from the beginning. They're task-based, including metrics such as: task completion rate, pass/fail of user tasks, time on task, error rates, and overall user satisfaction.

Summative usability testing is carried out:

• At the end of a development stage before a product is released
• To validate the usability of a product- against usability metrics
• To establish a usability benchmark
• To compare against competitor products
• With fully functioning prototypes
• With 15-20 users

Conclusion

Both formative and summative are forms of evaluative research. With formative and summative testing during the design and development lifecycle, both qualitative and quantitative data are collected. These tests help to determine whether the design elements are working and how they can be improved.

Start formative testing as early and as often as possible in the design and concept phase for a user-centered design approach. Incorporating user feedback through formative and summative usability testing ultimately leads to a better and more usable product.