Self-driving cars are a "thinking game" on public roads and safety of intended functionality is still a work in progress.

While autonomous driving has tremendous potential to save lives, its clear that a vehicle that "does its own thing" can exhibit behavior that is not as dependable under unusual conditions. Make no mistake, driving (by a car, truck or human) is very much a thinking game. What a human eye thinks is a good thing to interpret, may not be what a computer vision system expects. For example, a computer vision model may need its input to be explicitly fine tuned where the actual image---and even from frame to frame--is adjusted.

Andy Vikas, Chief Executive at Hybrid Intelligence at Capgemini Engineering clued me into surprising examples of machines that are rather "naive/fragile/unpredictable" when you come to think of it. In the first video below you can see that computer vision system confusing the moon with a yellow traffic light. It is perceiving a traffic light----but in reality what it should see is the moon.

This is another example, where the vehicle is being "thrown" traffic lights---that are in fact sitting inside a pick-up truck.

Whether it's a young child or an autonomous system---a combination of diverse and explicit situations must be successfully learnt, interpreted and acted upon --especially on the open road. Although humans have the knack and unique capability to extrapolate and generalize in ways that task-specific AI systems simply cannot.

What would a self-driving / autonomous vehicle do if it encountered all of these conditions (simultaneously): loose material on roadway, in a construction zone closure, with a barrier, other non-roadway users---and when its fogy on Halloween night....with someone dressed as a chicken.

Trust in AI systems starts with context & common sense

Every person in the automotive industry is familiar with confusion that can vex a computer vision model. The broader question is : "how do we create more trustworthy AI systems?", and in this article---what are some of the processes and approaches for autonomous driving.

It does start with realizing the limitations of any single decision tool--be that a rule, heuristic or some sort of learnt model. Making a good decision (with more uncertainty) needs a richer context ---especially when we have never seen something before, yet we still manage to know what to do. A good and trustworthy decision also needs some sort of humility. So its not that we can just keep on training a machine learning model, adding more complexity to neural networks, or more computations so that we we arrive at a point where we can trust the AI decision implicitly. Its not that straight forward.

"Trust is a complicated issue, it is more than just the ability to get an answer. There other topics that are relevant to trust AI systems. It is the same as the way we trust people, professionals, medics. We tend to trust people that are less confident---and less certain--they know when they know. Trust is fragile--and easily lost." - Andy Vickers

Most computer vision and collision detection systems will find it difficult to make a reasonable judgment call on what exactly it's dealing with when presented with the situations in the image below:

For example, traffic light LED’s can flicker at around 50hz which is unnoticeable to human eye but captured by camera sensors. The flickering can pose challenges to perception system and decision-making models if each frame is considered part of a new reality.

Or in another example, bright sunny days are ideal for driving but sun glare / flares can be unfavorable from safety aspect and can pose a serious hazard. It’s been documented that glares or sun flares if present can be more hazardous than the snowy weather conditions. Unlike other weather phenomena's, glares have not been researched widely as its difficult to record glares by weather stations like other weather events. An empirical study, based on accident data from Arizona concluded that glares could increase the chances of crashes at intersections. Sun glare can impact data quality and hinder in the decision-making process as it limits the visibility for ML (machine learning) algorithms.

Now not every decision has to use machine learning models that come with their own degree of uncertainty. A deterministic algorithm maybe more appropriate. In fact any autonomous system should have a process that starts with an analysis of requirement that require an ML type component into the function. After ensuring that indeed ML is the only relevant way to perform the function, a design strategy for the ML component should be derived.

Getting real with intended behaviors in edge cases

To help make sense of all those rare (but also obvious) edge conditions (see image below) a standard has been put together, aptly (or not) named: Safety Of The Intended Functionality (SOTIF). SOTIF is defined as “the absence of unreasonable risk due to hazards resulting from functional insufficiencies of the intended functionality, or by reasonably foreseeable misuse by persons”.

SOTIF (ISO/PAS 21448) was devised to account for the performance limitations of sensors and unknown scenarios.?The goal of the standard which essentially complements ISO 26262 is to maximize the known safe scenarios in what becomes the operational design domain (ODD) that describes where an autonomous vehicle is going to operate safely. It is left to the autonomous systems developer to articulate the ODD---and ultimately developers to make things happen in software and hardware.

The taxonomy of ODD is based on the categories like Physical Infrastructure, Operational Constraints, Objects, Connectivity, Environmental Conditions, Zones according to the National Highway Traffic Safety Administration (NHTSA). Another ODD specification commissioned by the Centre for Connected and Autonomous Vehicles (PAS 1883:2020) has three three-level taxonomy:

• Scenery: elements in the environment which are non-movable
• Environment: weather and environmental conditions
• Dynamic Elements: movable objects in the environment

If an AV is operating outside of ODD, it should be aware of that so that the control can be passed onto the driver in case of level 2 or level 3 autonomy. The taxonomy of ODD is based on the categories like Physical Infrastructure, Operational Constraints, Objects, Connectivity, Environmental Conditions, Zones as according to the NHTSA.?

Whether these scenario's are captured by real-world driving, generated by an algorithm---they will be necessary to to build, test & validate complex level 4+ ADAS functions. Of course, it is imperative that quality "ground truth" be gathered by real-world driving on the public roads as a primary feed into the simulation environments in order to target those unsafe & unknown conditions which are potentially difficult to emulate.?

Source:

Trusting AI systems: Necessary processes and artifacts for safety evidence

A SOTIF based approach to the development of autonomous vehicles (that are loaded with machine learning models) is one that helps evaluate the ADAS/AD performance---with the necessary indicators and statistics to give a "go" "no go" decision on the function.

The goal by OEM's and technology suppliers is to put in place processes for perception systems, independent driving functions and and automotive driving stacks so they are assured to be up-to-par and better than human judgement when presented with unknown and unsafe conditions.

Some of the challenges:

• Crafting driver assistance functions that has proper "situational awareness". Consider a T-junction or maybe a round-about assist driving function that has been designed with specific constraints. Maybe it only "understands" what best to do with 5 cars in the way. The assisted driving specification needs to be assisted itself. Maybe its simply a process to identify new scenarios and correct the logic. Maybe its more beneficial to arrive at the intersection and take advantage of someone else's situational awareness to avoid vulnerable road users---such as a road-side unit.
• Degradation of system components: System should be able to compensate for sensor failure / degradation and transfer the system to safe state. For example, Sensors are prone to miss-calibrations due to normal usage, or due to accidents, vibrations etc. while on the field. It is a two-step problem of detecting the fault in calibration in real time and then fixing it either online or offline to get the vehicle back to the fleet for data collection or actual driving tasks. Gap in calibration of the Camera and LiDAR sensor can prove to be fatal in the real world driving and would also lead to corrupt data during data collection campaigns.
• Inability to operate efficiently within the intended operational design domain: Simply put---without the right training data, for example, on a what a crowded intersection looks like---when presented with that situation the autonomous vehicle may not perform as intended. If an AV is operating outside of ODD, it should be aware of that so that the control can be passed onto the driver in case of level 2 or level 3 autonomy.?Ineffective data collection campaigns will not cover the adequate scenarios with correct weightage to each of the operational design domain. This can lead to bias which will affect the decisions of the ML models.
• Safety assurance of machine learning models built using data from real world driving versus synthetic data & simulations: Black box testing of ML models is insufficient to label them useful, if the learning itself is flawed. White box testing becomes essential for safety assurance of ML models.
• Failure of AI models to identify edge cases in real time during driving tasks: Scaling supervised models to identify edge cases is only partially effective as edge cases have a very small probability of occurrence. For example, a person dressed up like a bird can trick a machine learning algorithm. The performance of a machine algorithm solely depends on the training data used. Identifying such edge cases is crucial for the AV performance and avoiding potential vehicle faults. Situations underrepresented in the data can be a challenge to create robust perception models which can be disastrous in real driving and pose risk to road users.
• What to do / what not to do -- common sense: What is a bad idea when driving? Accelerating at an intersection. Not paying attention at an "unprotected left". Playing chicken with a road rage driver. What happens when an oncoming car hesitates at broken traffic light signal? There are ways to learn how to behave in extreme uncertainty and when to be submissive or assertive when driving. There is research to share driving intentions with surrounding vehicles---letting everyone know what "you" want to do---especially when there are uncooperative vehicles.

Contributors:

Walid Negm, Gaurav Pahwa