Safety verification and validation of autonomous driving

Autonomous mobility is one of the most promising technological developments of the 21st century. Autonomous vehicles offer numerous benefits, such as increased road safety, greater efficiency and reduced environmental pollution. However, there are also concerns about safety of autonomous vehicles due to their independent behavior and limited explainability of the decision-making process.

Since the functioning of autonomous vehicles is tied to often intransparent AI systems, a thorough system design and systemic safety analysis are important building blocks to ensure safety during operation. The emerging topic of AI safety is becoming increasingly important, due to profound differences in the development and technical properties of AI systems compared to classical software. Manufacturers can no longer just rely on existing safety standards, but must expand their safety case. The creation of additional, AI-specific safety artifacts can be a suitable measure at this point, as already discussed in our article "Considerations on AI safety for autonomous driving".

Besides increasing efforts for a safe system design and specifying safety requirements, a suitable verification and validation (V&V) strategy must be defined to prove safety and the fulfillment of requirements. In this regard, different V&V methods for vehicle safeguarding are available and can be used depending on the development phase (see Figure 1).

Simulations

Computer-aided simulations are used throughout the development process. Such simulations offer the advantage of providing reproducible, cost-effective results and the ability to test components and systems beyond performance and load limits. Due to their versatility, several hundred simulation methods are used in practice, which simulate e.g. crash calculations, aerodynamics, chassis tuning or electromagnetic compatibility. While such methods are an integral part also in the development of traditional vehicles, simulations of sensor inputs and operation environments, for example, are becoming increasingly important for AD. Since autonomous vehicles operate in complex environments, there can be large variations in data generated by sensors and characteristics of the operating environments. Simulations are therefore an indispensable tool, as they can be performed in parallel and small changes can be tested iteratively and efficiently through parameterizable simulation models. Moreover, simulation models can be reused, e.g. to facilitate testing of adapted system designs or changing operational design domains (ODDs) for which an autonomous vehicle was developed initially.??

X-in-the-Loop methods

In addition to already mentioned computer-aided simulations, X-in-the-Loop methods are used to mirror the complete-vehicle into detailed digital models. Most important in this regard is a virtual vehicle image whose components are integrated - as a model, software or hardware element - depending on the development progress. For testing, it is not important in which development stage other components are, since specifically defined interfaces allow tests between real components and simulation models. Thus, considerable results can already be achieved during development, which then only have to be verified. A major advantage of such virtual methods is that actual maneuvers and test cases to be performed in reality are already available at an early stage. This allows design decisions to be verified as early as possible, before a prototype has been elaborately created. Moreover, X-in-the-Loop methods offer reproducible results and, due to the parameterization of simulation environments, a high degree of flexibility and a reduction of time required for overall V&V. Depending on the development progress, different X-in-the-loop methods can be used, with Model-in-the-Loop (MiL) being used at the very beginning.

Model-in-the-Loop (MiL)

MiL methods allow the verification of customer requirement specifications down to the level of the logical architecture. For this purpose, software models are created that correspond to the functional development objective. These simulation models are validated within a virtual simulation environment, whereby all required components, such as environment, vehicle dynamics, powertrain or driving model, are made available in modular form. By merging different models into a digital prototype in a complete-vehicle context, concrete requirements and test specifications can be defined. Such a digital prototype can be coupled with a driving simulator to make it more tangible. Overall, MiL methods enable manufacturers to draw conclusions about customer acceptance at an early stage, thus reducing the development risk. Moreover, created software models can be further developed and used for more detailed simulations using Software-in-the-Loop (SiL) methods.??

Software-in-the-Loop (SiL)

SiL methods enable early verification down to the component level. For that, defining functional models in advance? can be transferred into more detailed, but hardware independent simulation environments. In SiL environments, the technical conditions are already highly detailed in terms of resolution accuracy and real-time behavior. The combination of MiL and SiL thus allows the creation of a virtual prototype at the end of the descending branch of the V-model (see Figure 1), including all components, interfaces and the overall functionality. This complete-vehicle digital prototype can then be used by Hardware-in-the-Loop (HiL) methods, to iteratively integrate and test real hardware components.?

Hardware-in-the-Loop (HiL)

By using HiL methods, upfront created SiL models are connected to real components in several iterations. At the beginning of HiL testing, individual hardware components are verified independently regarding their specified requirements. To do so, the component in scope is connected to virtual simulation environments that represent interfaces, or other components and systems. After successful verification of the individual components, step-by-step integration takes place and the resulting subsystems are verified again in adapted simulation environments.

Step-by-step, the complete-vehicle is built up and tested against specified requirements on each level of the architecture. While HiL is typically used to test single hardware components or subsystems, Vehicle-in-the-Loop (ViL) constitutes a method to test fully assembled prototypes in virtual environments.???

Vehicle-in-the-Loop (ViL)

ViL methods allow testing of real vehicles in virtual environments. For this purpose, the vehicles are connected to simulation environments via dedicated interfaces, allowing sensors, for example, to be simulated with artificial signals. Depending on the design of the ViL environment, custom-built test benches with mechanical components are required to absorb longitudinal and lateral movement of the vehicle during simulation. Such settings enable testing of critical environmental conditions or driving maneuvers in a reproducible way, while the vehicle's reaction to prevalent conditions can be used to verify a safe system behavior. These characteristics make ViL highly interesting for safeguarding AD, since autonomous vehicles directly depend on captured sensor data and the vehicles reaction to distribution shifts in the data, for example, can be tested in a safe manner.?

Real road testing

Real road testing represents a classical method for vehicle V&V, where the developed function or the overall system is tested in road traffic. However, this type of safeguarding involves risks for the test driver and all other road users, since unexpected malfunctions can lead to dangerous situations. Thus, testing in a real environment should first begin in a controlled manner, for which test sites exist. Such test sites are designed like typical traffic areas, allowing a large number of frequently occurring real traffic scenarios to be tested. Even though advanced simulation environments and methods allow virtual testing of vehicles with high detail, real road testing still is a must to validate system performance and upfront obtained testing results at the end of the development process. In particular, since not all environmental influences can be represented in virtual simulations, further potential sources of risk and unwanted system behavior can be uncovered by means of real road testing. However, starting with real road testing is not economically feasible, especially in the case of autonomous vehicles. Due to the multitude and complexity of possible traffic scenarios, a well-considered combination of first intensive virtual testing and finally real testing is needed to ensure safe AD.?

While the outlined V&V methods are also applicable to the development of traditional vehicles, the development of AD is much more reliant on simulations and scenario-based testing in virtual environments Digital models of the vehicle and the environment (incl. road infrastructure, weather conditions, traffic participants, etc.) become indispensable for facilitating testing activities and time-to-market. However, the definition of relevant scenarios and transferring them into reusable digital models poses challenges for manufacturers. Using existing and standardized processes and formats is a key enabler for successful development, as shown in my article “Driving ≠ Driving: Why operational design domain and scenario definition is the essence for safe autonomous driving systems”.

Key takeaways?

Ensuring safety in AD is a complex matter. In addition to increased efforts for thorough system designs and establishing safety requirements, testing effort also increases. While manufacturers can rely on existing V&V methods, virtual methods and simulations are becoming more important. X-in-the-loop methods are an effective instrument for creating a detailed digital vehicle image and starting tests in virtual environments early on. Together with a scenario-based testing approach, the behavior of the autonomous vehicle can be evaluated in a large number of possible traffic scenarios before moving to real road testing. However, looking at the diversity of the different methods reveals that comprehensive test management is necessary for coordination and documentation of individual testing activities. In particular, as virtual methods become more sophisticated and complex, but still need to be linked to real road testing and other verification methods, this is mandatory for comprehensive safety evidence.