The art of software tool qualification

You spend countless hours on making your software safe for use in a critical automotive system, but do the software tools you use provide that same level of safety? How sure are you that your test framework has executed all test cases? Could a code generator introduce memory faults into a software component?

The tool qualification process is there to ensure that the software tools you use do not introduce errors or fail to detect errors in your safety-related code.

In practice, tool qualification requires much more than a mere checklist or assessment of the tool source code. You need to consider the specific requirements of your project, the characteristics of the tools, the potential risks they pose, and their related failure modes.?

In this article I first provide a quick review of the ISO 26262 tool qualification process and highlight the useful and not so useful parts of it. After that, I will show how tool qualification is done in the real-world.

So, what does the tool qualification process look like? Tool qualification begins with the planning of tool qualification as seen below:?

The first five steps are needed to understand how the tool will be used and how the tool will impact the safety-related source code. This information is input for step six: determining the required qualification methods. Step six has two parts, first determine the required tool confidence and then the selection of the tool qualification methods.

Software tools may have no impact on safety-related parts of the code at all.?In other cases, software tools may have an impact on safety-related code, but these errors can be easily detected.?The worst case is when tools can introduce errors that are difficult to impossible to detect.

Unfortunately, the ISO 26262 gives few guidelines on methods for determine the confidence for detecting tool errors.?I recommend performing a FMEA on the tool and to use this as a basis for identifying the likelihood of error detection.

If a tool has no possibility of impacting safety related code, low tool confidence is required, you are in luck.?The evaluation process ends here.?The tool simply needs to be used as previously specified in tool qualification planning.

If a medium or high tool confidence is required, the tool qualification methods must be selected next.?The recommended and highly recommended methods are derived from the safety-level of the software that is impacted by the tool and the previously calculated tool confidence level:

This is where the ISO 26262 tool qualification process becomes problematic.?

• The method "Increased confidence of use" suggests using widely adopted tools. However, this method demands substantiation with robust data. Can you access your software tool provider's bug reports and usage data to demonstrate the reliability of their tools? Most likely not. If you don't meticulously document failure data for your self-developed code, the missing of evidence also renders this method impractical.
• The method "Evaluation of tool development process" has a similar problem as the "Increased confidence of use." Will your tool provider provide evidence of a comprehensive process assessment? While they might be forthcoming in the automotive industry, this is less likely with general tool providers.
• The method "Validation of tool output" is reasonable and, in my perspective, should be standard practice. Surprisingly, this method is only highly recommended for ASIL-C and ASIL-D safety levels.
• Lastly, the method "Development according to safety standard" is the most stringent approach. In practice, this becomes unfeasible for complex tools. Disregarding the associated development costs, implementing a complete compiler tool chain aligned with ISO 26262 standards would require a decade to accomplish. Imagine reworking 500k lines of code in accordance with ISO 26262 guidelines. For general purpose tool this method ventures into the realm of pure fantasy.

So how do real-world projects deal with these problems? The simple answer: they avoid them completely.

For real-world safety projects, the focus is to increase the error detection possibility.? Based on the tool function, its inputs and outputs, and failure analysis, a project must identify methods to avoid or mitigate the possible failure modes.

To tackle potential tool failures, consider using the following methods:

Introducing Redundancy

Making use of redundant tools, especially for open-source software tools, offers an efficient and effective strategy for addressing tool errors. By using multiple tools for the same use-case, failures in one tool can be detected.? Of course, common cause failures must be address, but with enough independence, this could be a suitable method with little overhead.

Audit Trail?

An audit trail provides a mechanism for strengthening verifiability and transparency. Its key components include:

• Timestamps: Each entry in the audit trail is time-stamped, establishing a clear chronological sequence of events.
• Configuration information: The audit trail captures details about the used configuration values to detect use of wrong configuration data.
• Action descriptions: Descriptive documentation of the actions or events that occurred, encompassing activities like data input, processing steps, setting changes, and more.
• Affected Data: If applicable, the audit trail records data or objects impacted by an action, providing context and understanding of an event's implications.

By utilizing an audit trail, verification of necessary actions and the prevention of unwanted actions are possible.

Manual Input and Output Checks

This approach involves implementing manual measures to detect critical issues based on identified tool failure modes. Manual input and output checks must be specified in enough detail to ensure that they are consistently carried out.

For instance, within a testing framework, a failure mode might include the absence of one or more test cases. Addressing this failure mode requires verification of test run results to ensure that all expected tests have been executed.?

Automated Output Checkers

Humans have limits on how much information that they can keep in memory at single moment.?As the number and complexity of input and output checks increases, the transition from manual to automated checks becomes essential for consistent and accurate verification of tool output. In this case, the output checkers must also be qualified.?So, the process is in a way recursive.

Further strategies for efficient and effective tool qualification

Besides the above methods to address software tool failures there are five strategies that are proven to improve the effectiveness and efficiency of the tool qualification process:

Strategy 1 - Evaluate tool chains and not individual tools

Focus efforts on the development and verification tools that have a direct impact on the critical software.? The failure modes of supporting tools are important, but their impact on the tools that have a direct impact on the critical software is where the risk is.

By focusing your tool qualification efforts on the tools that impact the safety-related software, you avoid wasting efforts on irrelevant tools.

Strategy 2 - Output to human readable text; don’t store in a binary format

Storing information in human-readable text formats makes it easier to review, analyze, and verify results manually. In addition, this approach eliminates the need for specialized tools to interpret binary data, reducing potential errors and increasing the likelihood of identifying issues reliably.

Human-readable text formats also facilitate version control, allowing for clear comparisons between different iterations and aiding in the identification of deviations, which leads to strategy 3.

Strategy 3 - Put text under version control

By placing text-based files under version control, such as using version control systems like Git or Subversion, every modification is documented, allowing for easy comparison between different tool qualification iterations. This helps in quickly identifying any deviations or errors that may have occurred during the tool qualification process.

Moreover, if problems arise in the tool qualification process or output, version control helps pinpoint the exact changes that occurred, supporting in troubleshooting and resolution.

Strategy 4 - Perform manual verification of independent format

When output data is converted into an independent format, such as printing Microsoft Word files to PDF, it creates a reference point that is separate from the original tool. This enables manual verification without relying solely on the tool itself.

This approach acts as a double-check mechanism, ensuring that the tool's results align with the expected outcomes. It also provides an additional layer of validation that supplements automated checks, reducing the risk of oversight or false positives.

Strategy 5 - Tool output must be falsifiable

Falsifiability refers to the concept that a hypothesis or statement should be structured in a way that it can be proven false through empirical evidence or observation. By designing tools to provide verifiable evidence of their actions rather than simple pass/fail outcomes, this strategy enables thorough auditing and evaluation at a later stage.

When tool outputs are structured to include an audit trail of actions taken during their execution, it becomes possible to validate and verify the process. This audit trail provides a clear record of the steps the tool has taken, allowing for independent verification of the results. This approach enhances transparency and accountability, ensuring that tool actions can be observed, scrutinized, and even challenged if necessary.

Final thoughts

The reliability of software tools in the automotive industry is a vital concern alongside safe software development. The ISO 26262 tool qualification process, while a foundation, brings challenges that need practical solutions. In real-world projects, error detection is prioritized over the development process used to create a tool.

Through the qualification methods and strategies that I discussed in this article, we uncover alternative paths to tool reliability. Audit trails, human-readable outputs, version control, and manual verification are essential. Redundancy offers a feasible way to tackle tool failures. Providing falsifiable tool outputs ensures verifiability.

My original goal of this article was to show that by merging theory and practice, we create a software tool qualification process where reliability, safety and practicality go hand in hand.

Now it’s your turn. How do you tackle the tool qualification process in your project? Do you rely on tool providers completely, or do you have a unique strategy to managed the risks?