The art of software tool qualification
Joel Thurlby
Automotive - Functional Safety | Software Architecture | System Architecture
You spend countless hours on making your software safe for use in a critical automotive system, but do the software tools you use provide that same level of safety? How sure are you that your test framework has executed all test cases? Could a code generator introduce memory faults into a software component?
The tool qualification process is there to ensure that the software tools you use do not introduce errors or fail to detect errors in your safety-related code.
In practice, tool qualification requires much more than a mere checklist or assessment of the tool source code. You need to consider the specific requirements of your project, the characteristics of the tools, the potential risks they pose, and their related failure modes.?
In this article I first provide a quick review of the ISO 26262 tool qualification process and highlight the useful and not so useful parts of it. After that, I will show how tool qualification is done in the real-world.
So, what does the tool qualification process look like? Tool qualification begins with the planning of tool qualification as seen below:?
The first five steps are needed to understand how the tool will be used and how the tool will impact the safety-related source code. This information is input for step six: determining the required qualification methods. Step six has two parts, first determine the required tool confidence and then the selection of the tool qualification methods.
Software tools may have no impact on safety-related parts of the code at all.?In other cases, software tools may have an impact on safety-related code, but these errors can be easily detected.?The worst case is when tools can introduce errors that are difficult to impossible to detect.
Unfortunately, the ISO 26262 gives few guidelines on methods for determine the confidence for detecting tool errors.?I recommend performing a FMEA on the tool and to use this as a basis for identifying the likelihood of error detection.
If a tool has no possibility of impacting safety related code, low tool confidence is required, you are in luck.?The evaluation process ends here.?The tool simply needs to be used as previously specified in tool qualification planning.
If a medium or high tool confidence is required, the tool qualification methods must be selected next.?The recommended and highly recommended methods are derived from the safety-level of the software that is impacted by the tool and the previously calculated tool confidence level:
This is where the ISO 26262 tool qualification process becomes problematic.?
So how do real-world projects deal with these problems? The simple answer: they avoid them completely.
For real-world safety projects, the focus is to increase the error detection possibility.? Based on the tool function, its inputs and outputs, and failure analysis, a project must identify methods to avoid or mitigate the possible failure modes.
To tackle potential tool failures, consider using the following methods:
Introducing Redundancy
Making use of redundant tools, especially for open-source software tools, offers an efficient and effective strategy for addressing tool errors. By using multiple tools for the same use-case, failures in one tool can be detected.? Of course, common cause failures must be address, but with enough independence, this could be a suitable method with little overhead.
Audit Trail?
An audit trail provides a mechanism for strengthening verifiability and transparency. Its key components include:
By utilizing an audit trail, verification of necessary actions and the prevention of unwanted actions are possible.
Manual Input and Output Checks
This approach involves implementing manual measures to detect critical issues based on identified tool failure modes. Manual input and output checks must be specified in enough detail to ensure that they are consistently carried out.
领英推荐
For instance, within a testing framework, a failure mode might include the absence of one or more test cases. Addressing this failure mode requires verification of test run results to ensure that all expected tests have been executed.?
Automated Output Checkers
Humans have limits on how much information that they can keep in memory at single moment.?As the number and complexity of input and output checks increases, the transition from manual to automated checks becomes essential for consistent and accurate verification of tool output. In this case, the output checkers must also be qualified.?So, the process is in a way recursive.
Further strategies for efficient and effective tool qualification
Besides the above methods to address software tool failures there are five strategies that are proven to improve the effectiveness and efficiency of the tool qualification process:
Strategy 1 - Evaluate tool chains and not individual tools
Focus efforts on the development and verification tools that have a direct impact on the critical software.? The failure modes of supporting tools are important, but their impact on the tools that have a direct impact on the critical software is where the risk is.
By focusing your tool qualification efforts on the tools that impact the safety-related software, you avoid wasting efforts on irrelevant tools.
Strategy 2 - Output to human readable text; don’t store in a binary format
Storing information in human-readable text formats makes it easier to review, analyze, and verify results manually. In addition, this approach eliminates the need for specialized tools to interpret binary data, reducing potential errors and increasing the likelihood of identifying issues reliably.
Human-readable text formats also facilitate version control, allowing for clear comparisons between different iterations and aiding in the identification of deviations, which leads to strategy 3.
Strategy 3 - Put text under version control
By placing text-based files under version control, such as using version control systems like Git or Subversion, every modification is documented, allowing for easy comparison between different tool qualification iterations. This helps in quickly identifying any deviations or errors that may have occurred during the tool qualification process.
Moreover, if problems arise in the tool qualification process or output, version control helps pinpoint the exact changes that occurred, supporting in troubleshooting and resolution.
Strategy 4 - Perform manual verification of independent format
When output data is converted into an independent format, such as printing Microsoft Word files to PDF, it creates a reference point that is separate from the original tool. This enables manual verification without relying solely on the tool itself.
This approach acts as a double-check mechanism, ensuring that the tool's results align with the expected outcomes. It also provides an additional layer of validation that supplements automated checks, reducing the risk of oversight or false positives.
Strategy 5 - Tool output must be falsifiable
Falsifiability refers to the concept that a hypothesis or statement should be structured in a way that it can be proven false through empirical evidence or observation. By designing tools to provide verifiable evidence of their actions rather than simple pass/fail outcomes, this strategy enables thorough auditing and evaluation at a later stage.
When tool outputs are structured to include an audit trail of actions taken during their execution, it becomes possible to validate and verify the process. This audit trail provides a clear record of the steps the tool has taken, allowing for independent verification of the results. This approach enhances transparency and accountability, ensuring that tool actions can be observed, scrutinized, and even challenged if necessary.
Final thoughts
The reliability of software tools in the automotive industry is a vital concern alongside safe software development. The ISO 26262 tool qualification process, while a foundation, brings challenges that need practical solutions. In real-world projects, error detection is prioritized over the development process used to create a tool.
Through the qualification methods and strategies that I discussed in this article, we uncover alternative paths to tool reliability. Audit trails, human-readable outputs, version control, and manual verification are essential. Redundancy offers a feasible way to tackle tool failures. Providing falsifiable tool outputs ensures verifiability.
My original goal of this article was to show that by merging theory and practice, we create a software tool qualification process where reliability, safety and practicality go hand in hand.
Now it’s your turn. How do you tackle the tool qualification process in your project? Do you rely on tool providers completely, or do you have a unique strategy to managed the risks?
AFSE | CySec | CSFX | CCSP Working on safety argumentation for complex systems.
1 年if we call it software SUPPLY CHAIN safety RISK MANAGEMENT, then we might have a higher chance of buy-in.
I don't want to spoil this.Intelectually, this is easier: Can the tool produce false positives and If so, how do you work around this. Problem solved.