What are the best practices for designing quality attribute scenarios and tests?

1 Define quality attributes

The first step is to identify and prioritize the quality attributes that are important for your system and its stakeholders. You can use various sources, such as business goals, user needs, domain standards, or regulatory constraints, to elicit and analyze the quality attributes. You should also consider the trade-offs and dependencies among different quality attributes, as they may affect the design decisions and the evaluation criteria.

2 Use a standard template

The second step is to use a standard template for writing quality attribute scenarios and tests. A common template is the SAAM (Software Architecture Analysis Method) scenario template, which consists of six elements: source, stimulus, artifact, environment, response, and response measure. By using this template, you can ensure that your scenarios and tests are consistent, complete, and unambiguous.

3 Specify scenarios and tests at different levels

The third step is to specify scenarios and tests at different levels of abstraction and granularity. You can use three levels: general, concrete, and test. General scenarios and tests are high-level and describe the overall quality goals and expectations. Concrete scenarios and tests are more detailed and specify the exact conditions and inputs for the system. Test scenarios and tests are executable and verify the actual outcomes and measurements of the system.

4 Validate scenarios and tests with stakeholders

The fourth step is to validate your scenarios and tests with the stakeholders who have an interest or influence on the system quality. You can use various techniques, such as reviews, walkthroughs, or prototyping, to solicit feedback and confirm that your scenarios and tests are aligned with the stakeholder needs and expectations. You should also update your scenarios and tests based on the validation results and the changes in the system or the context.

5 Prioritize scenarios and tests based on risks and benefits

The fifth step is to prioritize your scenarios and tests based on the risks and benefits of each quality attribute. You can use various criteria, such as frequency, impact, difficulty, or cost, to rank your scenarios and tests according to their importance and urgency. You should also consider the trade-offs and dependencies among different scenarios and tests, as they may affect the evaluation plan and the resource allocation.

6 Execute scenarios and tests and analyze results

The final step is to execute your scenarios and tests and analyze the results. You can use various tools, such as simulators, monitors, or analyzers, to perform the testing and measurement of the system quality. You should also compare the results with the expected values and identify any gaps or deviations. You should also document and communicate the results and the recommendations for improvement or mitigation.

7 Here’s what else to consider

This is a space to share examples, stories, or insights that don’t fit into any of the previous sections. What else would you like to add?