Basics of test strategy development

Intro

As announced, I will start with the basics. For many people, this will probably be a repetition of familiar things, but perhaps it will provide some food for thought. For all newcomers, hopefully a good overview to get started.?

The starting point for this text is a paper I wrote for SAE in 2017. Since I was doing a lot in the aerospace field at the time, the paper is written with an aerospace focus. I am linking the paper here, if any of you are interested, please just contact me.?In this paper I have also focused more on virtual testing. Here I would like to remain more general.

There are of course many parallels across different industries in terms of test strategy development. However, I want to keep this article general, but still use some examples from the automotive and aerospace sector.

I would like to focus in particular on process-oriented testing, which means that we first think about the strategy and the process and then set up the necessary test steps and test environments that implement this strategy particularly efficiently and successfully. All in the spirit of the guiding principle "Structure follows Strategy".?

Process-oriented Testing

Even though we now talk a lot about agile processes and the phases during development and during the test processes can and should of course no longer always be separated strictly from one another, the V-model is excellently suited as a thought model for defining a test strategy. I would recommend everyone, regardless of whether it is an agile project or not, to use the V-Model to think about the high-level goals of the test strategy.

In order to achieve meaningful test coverage, we need as much information as possible about the system to be tested. The V-model, which is laid out in such a way that from the top left downwards, the system design is derived from the top-level requirements and this is then decomposed piece by piece so that there is information for the various sub-systems and components about which quality criteria these individual parts have to fulfil.

From the bottom to the top right, we then integrate the system piece by piece and determine on how many of these integrations we will carry out tests (test level).

Going through the V-Model, we define the required validation and verification tasks. In general, the right branch of the V-cycle contains the individual test phases and integration levels for testing E/E systems. Integration progresses step by step from individual software unit or components to the entire software running on networked systems. This involves many steps that differ in test object (system under test), test objective, test phase or test level, test environment and the associated work packages. The main challenge is to coordinate these steps and apply them consistently and reliably throughout the verification process. In this way, the large number of tests can be carried out efficiently within the available development time. The input from the left branch of the V-cycle is needed for the execution of the tests. This includes behavioural models for virtual test methods, requirements for test designs, or descritpion of safety mechanisms for doing fault injection testing. This input also needs to be managed. On the one hand, this is a challenge, as extensive planning is required, on the other hand, it is also an opportunity, because managing the input helps to avoid redundant work products in the development process. For example, models are developed once and can be reused throughout the development cycle.

My experience over the last few years shows that consistent planning usually does not take place. Rather, test strategies are based on the reuse of past projects, the test levels are often not defined in a meaningful way and so it happens that testing often becomes a bottleneck at the end of the development projects.?

I see a lot of potential for improvement in the automotive industry and to go into strategy development again at the beginning of the projects instead of saving time by reusing should be worth considering.

Often the test levels and integration stages are not feasible at all because the information from the system specification is missing. I briefly talked about agile development: here, a close interlocking of the left side and the right side of the V-model is all the more important. The definition of a test strategy is mandatory.

Test Objects & Evaluation Criteria

A core aspect of the test strategy is to define what is to be tested, when and with what. Formulating this high-level strategy is enormously helpful when it comes to concrete projects. In the following, I will show you how to define the strategy step by step. Tailoring the strategy for individual projects is then much more straightforward.

An essential part of structuring the testing is the planning of the test level. The planning of the test level is directly linked to the available test objects. Questions such as "When will a software component be ready for testing?" or "When will a first prototype of actuators be available?" must be answered when planning the verification process and test phases along the V-cycle . The test objects can be the systems and subsystems under investigation, e.g. individual electronic control units at the sub-system integration level or an system at the system integration level. The test object is directly connected to the verification environment, e.g. a virtual test environment. Therefore, the test object determines which environments can be used for verification. For example, an ECU cannot be tested in a fully simulated environment, as it requires physical parts for its error-free execution. In contrast, a single software component can be tested in a fully simulated environment. Using the most efficient verification environment for each of the many test objects along the verification process is key to increasing test efficiency. This means that a valid test strategy that considers constraints and opportunities equally is required to adapt and develop the test strategy. The general guideline should be to simulate as much as possible and as little as necessary. The test strategy should be planned for the different tasks and departments involved in the development process. To plan the testing strategy, the following questions should be answered first.

1. Which test object needs to be tested in which test level (adaptation of the V-cycle)?
2. Which test objectives should be tested on which test object?
3. Which test environment must be used to enable error-free operation of the test object and testing of the planned test objectives?
4. What are the possible evaluation criteria for the test environment?

The planning and scheduling is defined in the test strategy.

Once the test levels have been defined, the next step is the test process itself and the execution of the tests. That means what is the exact process that we follow within the test levels, for example how test designs are created or how test evaluation critera can be met. The industries differ a little in how these processes are defined. I will show and explain the example from the aviation sector and then compare it with an example from the automotive industry. You will see that the result will not differ essentially. So feel free to use either approach. You can be sure that the ARP4754A is also a very welcome reference in the automotive sector.

Correlation between test strategy and test process

In my experience, it is a very good approach to deal with the concrete test processes within the respective test levels after defining the test strategy and thus the high level test objectives and the test levels. This means the concrete design of test planning and test execution in the respective test levels, e.g. the concretisation of the test environment for the respective test task and the actual execution of the test and, of course, the achievement of the respective test objectives and approvals.?Having the highy level strategy already in place makes it easier to consider constraints as early as possible.

The following is an example of a test strategy from the aviation sector. Also listed are the test environments that are used at the respective test levels.

With some experience in the automotive sector, it quickly becomes clear that the parallels are very large, even if the test environments are of course not identical.

Next, we want to concretise the test levels. The next picture illustrates how we instantiate the test processes within the individual test levels and thus create a consistent process framework for each test level.

As I said various process standards are available for designing the test processes within the test levels. In the following, I will show two different ones that I have found particularly practical in my work so far.?

So let's start with the ARP4754A

Aerospace Recommended Practice (ARP) ARP4754A , is a guide from SAE International that addresses development processes to support certification of aircraft systems.

The model for the test process introduced by ARP4754A describes the general verification flow recommended for aircraft systems, including the inputs required and outputs generated, as well as the dependencies on the upstream development process. The development tasks preceding the verification process include the formulation of requirements according to the associated development assurance level of the function or item and its design process leading to a testable implementation. The verification process is performed iteratively at different levels of integration, such as the element, system and aircraft integration levels. These are the test levels described before.

The process defines four basic verification methods to prove that the function or object under test, i.e. the test item, meets its functional requirements :

• Inspection or verification
• Analysis (e.g. modelling or coverage analysis).
• Test or demonstration
• Experience with services

According to ARP4754A, testing demonstrates that a test object performs its intended function and that it does not perform unintended functions that compromise safety. This proof is then provided for various test objects at different test levels, so that an overall release argumentation can be derived from their combination. In other words, proof that all requirements, all interfaces and the interaction of the various functions along all functional chains work without errors or at least could not have been tested more intensively.?This can be achieved by using representative software or hardware together with the physical system or appropriate validated models. This allows the use of different (somtimes virtual) test environments, depending on the test object and test phase.

Depending on the development assurance level (DAL) of the test object, one or more verification methods must be used. When aircraft systems are developed to standards such as DO-178C for avionics software, each function or element is assigned a specific DAL that depends on the classification of the function's or element's failure conditions. For example, functions or elements whose failure is classified as catastrophic or dangerous will have DAL A or B. In this case, ARP4754A recommends a test or demonstration and an additional verification method to demonstrate compliance with the applicable airworthiness code. This emphasises the particular importance of testing in the verification process for safety critical functions or elements in aircraft systems.

Each of the above verification methods requires specific inputs and produces outputs that must be managed and linked to the overall development process to ensure traceability between requirements and test results, reproducibility of the verification task, and reusability of associated artefacts, e.g. simulation models and test scripts, for subsequent verification phases. Other deliverables defined in ARP4754A are a specific set of verification data that provide evidence that the verification process has been completed. The standard introduces verification matrices, plans, procedures and summaries that can be used as evidence. This evidence is one of the prerequisites for subsequent certification. If you would like a deep dive into these topics, please feel free to contact me.

A verification plan specifies the verification methods to be used to fulfil the requirements, depending on the associated DAL. It defines the success criteria for the assessment of the verification results. In addition, the plan identifies the configuration of the DUT and specifies the test equipment and environment. This fits right in with what I was talking about before: the verification strategy defines which test environment must be used for specific test objects and how the associated test activities contribute to the verification criteria. The verification matrix links requirements and associated functions from the development process with verification methods, procedures and results of the verification process. The verification summary additionally includes information on significant deviations from the plan, reports on open issues and their impact on safety, supporting test data, and references to the verification plan and matrix. The Test Matrix is described later in the artical.

This blueprint for the development of a test process helps us to always match the requirements for the tests to the test results. It also helps us to structure our activities and not to forget the most important aspects. The process creates comparability, enables us to benchmark and serves as best practice.?

Let's look at a second example of how you fill your test strategy with the necessary (procedural) life.

And continue with a second example - ISTQB

For software development and especially software testing, the ISTQB (International Software Testing Qualifications Board) syllabus gives advice on how such a testing process can be organised. The syllabus calls this the fundamental test process. These aspects will be briefly presented in the following. The organisational advice given in the ISTQB syllabus is linked to familiar roles from classical software development.

If we compare this with the ARP, we see that the focus is different, but in the end they both strive for the same thing: to know what to test, when to test it, what requirements it targets and when we have tested enough.

I recommend that you use both blueprints when defining your processes to see which suits you better. My personal experience is that a combination of both is often useful. At some point, your own best practices will emerge anyway. Let's have a look at the process that the ISTQB proposes.?

The test process begins with "Planning". The central tasks of this test phase are the allocation of resources and the determination of the distribution of roles for the execution of the tests. A test strategy is developed that takes into account the prioritisation of tests based on a risk analysis and the determination of the test intensity of individual system parts. The test strategy includes the test phases in the right branch of the V-model, the integration stages and the methods to be used (Sil, MiL, PiL, HiL, Vehicle etc.) are determined. Here we see an overlap with the strategy development I presented earlier. It gets more exciting in the next phase. I will not go into the definition of roles in the test process here in this article. Depending on the feedback I receive on this, I can go into more detail in a later article.?

During "Analysis and Design", test cases are specified and maybe also simulation models are parameterized. The requirements and specifications, which are the basis for the test design (test creation), must first be analysed with regard to their testability. From this, requirements for the test case creation are derived. I will show you an illustration in the next section.

The test phase "Implementation and Execution" begins with the concretisation of the logical test cases, i.e. the definition of input values. A logical test case is often the basis for several concrete test cases , which are defined by different parameter sets. The parameter sets also place the test cases under a variant and version reference, which enables management and tracebility within the entire verification process. This separation between implementation and parameterisation is also used in model development in order to be able to use generic models for different test phases and systems.

The subsequent step, "Evaluating exit criteria and report", involves the evaluation of the test results on the basis of the previously conducted risk analysis. In this context it must be decided whether the test can be completed or whether changes to the test object and further test activities will follow in an iteration. The retrospective analysis and evaluation forms the conclusion of the test activities and records best practices for further projects. In the "control phase", ongoing test activities are checked against the previously established test plan in order to be able to intervene flexibly.

Maybe in the future I will also do the example of ISO 29119, which is comparable to ISTQB.?Feel free to comment whether this would be interessing for you.

Illustration: Tracebility and Requirements-based testing within the test process

As stated before the requirements are the basis for the test cases. The test cases are used to test the requirements. The requirements are defined long before the test cases, which is why a linking of these two must be carried out over a large period of time in the development and test process. However, linking these two aspects is mandatory to ensure traceability and reproducibility within the verification process. I am often asked about this aspect in the context of safety-critical projects. I can say quite clearly: this is always necessary, no matter what is being developed and according to what standards. If only for the reason of not losing the overview and being able to continuously improve the test strategy.

But Safety is also important of course: regression tests, which are a clear requirement of ISO 26262, can thus be organised more easily and an assessment of the test coverage is only possible conclusively in this way.?However, these links between requirements and test cases are not always unambiguous, as in most cases they are not 1:1 relationships.

Thus, a textually described requirement often becomes several test specifications in e.g. free text, which in turn become several test implementations. By parameterising the individual test implementations, concrete test cases can be created with reference to variants, which are then used for test execution.

This "chain" should be traceable throughout the entire development process.

Linking the individual artefacts to each other is important in order to be able to show semantic relationships between development objects in the entire test process, firstly to keep them traceable and secondly to be able to derive necessary tasks for improving the process.

Important advantages of linking requirements and test cases are traceability and backtracking, but also easier change management and better organised regression testing. In addition, the test coverage can be better assessed.

?The Test Matrix - center peace of our test activities

Just as the test processes must be defined within each test level within the test strategy. Of course, the test methods must also be specified. The following matrix shows an example of a test matrix for a specific test level. In this case, for example, the software integration test.

The necessary test environments are defined in relation to the test methods in the matrix. For each test level an instance of this test matrix can be created to define what is tested with which method. The matrix in the picture above is only one example, which is very closely based on ISO 26262.

The matrix is a procedure model to structure the planned tests in the respective test levels. In the first column of the matrix are the test methods, in the second column examples of the intended test objectives and then in the further columns the test environments in which the tests are to be carried out. The colour code tells whether the combination is possible. Combinations are often possible, but not sufficient on their own. This should also be shown in the matrix. When I discuss the testing of automated driving functions in one of my further contributions, I will take this up again.?

Even this simple example matrix shows that the test methods in combination with the test objectives already determine certain test environments or exclude others. For example, it is easy to understand that the entire embedded software and its interfaces to the control unit hardware cannot be tested in pure SIL environments and accordingly a HIL environment makes sense. I get this question quite often, so I'll answer it here: Instruction set simulations (ISS) do not help either, as we are interested in the real interfaces to the hardware and their responses and not in testing the ISS. Of course, an ISS enables further virtual tests, but one will not be able to fully test the HSI (Hardware/Software Interface Specfication) here. The qualification of the ISS within the framework of ISO 26262 is also not entirely straightforward, which reinforces this. However, I will go into this in more detail in my next article in the context of the proof of suitability of simulators in safety-critical projects.

But first let's continue with the matrix itself: The rows of the matrix are filled with the necessary test methods to be applied in the respective test level. These change depending on the integration level or test focus. Sometimes the focus is on the interfaces, in other cases on the performance of the hardware or the design. In the columns we then list the respective test environments that come into question. Of course, we can use different ones and ensure test coverage by using multiple test environments as well. For example, the purely functional tests can be tested easily and cost-effectively by using a lot of simulation and only for the tests where the hardware is absolutely necessary, such as HW/SW interface tests or performance tests, a hardware-in-the-loop simulator, a mechanical test bench or a vehicle is used.?

The test matrix is such a valuable tool because it gives us an overview and we can plan the test execution according to the best test method in combination with the test environment. We can keep an eye on the test coverage, but also gradually optimize the test execution by using other methods if necessary. The test matrix is as simple as it is ingenious. I recommend everyone to use it. Especially for requirements that come from standards such as ISO 26262 or ISO 21448, the test matrix can help perfectly to get an overview of whether all required methods have already been implemented.?

And last but not least...

The key to success in defining a structured and efficient test strategy. The test processes will fill the strategy with live and help you not to forget some essentials. I have made some suggestions in this article and hope that they will help you in your work.

I am aware that you can go into much more detail about many of the aspects I have mentioned. I would be happy to do this for you if you let me know which topics are of particular interest to you and where you still have too many unanswered questions. Comment directly on my LinkedIn profile or send me a message. I look forward to the discussions with you.?Gradually, I will go deeper into topics, but if you tell me in which direction you would prefer to go first, then I will of course be happy to adapt.

My next article will then describe how to tailor a test strategy described here so that it meets the requirements of ISO 26262. Here I will also discuss how to define a corresponding tool chain with which the test strategy can actually be implemented. In this context, I will also briefly address the tool qualification, which always leads to intensive discussions. I would like to clarify this once and for all.

I look forward to your feedback.?