A Multi-layered Approach to AI Safety - Our Viewpoint at SafeAI

In our last article on safety, we discussed the importance of laying the groundwork for a deep dive into AI safety by first developing an intuitive understanding of why and how AI models work. In this article, we will explore SafeAI’s approach to defining AI boundaries and creating domain-specific models.? We will further explore how to integrate safe Artificial Intelligence within off-road autonomous vehicles.?

At SafeAI, the development and integration of AI solutions begin with a clear understanding of the specific domain and operational boundaries within which the AI will function. This process involves defining the AI’s purpose with clarity and precision, ensuring that each model is tailored to meet the unique needs of its application. By establishing clear boundaries, SafeAI ensures that the AI system is optimized for the tasks it will encounter while also being prepared for the unique challenges and constraints of the domain.

We will use heavy industries like mining and construction as an example. In these industries, we design AI models for autonomous equipment, including haul trucks, dozers, and others. This domain-specific approach ensures that the AI models are effective and safe, as they are developed with a thorough understanding of the operational context and potential hazards.

The Significance of Defining the Operational Design Domain (ODD)

Operational Design Domain (ODD) specifies the boundary where the Autonomous Ground Vehicle (AGV) can safely operate. Therefore, ODD provides the design constraints of the AGV.?

‘‘ODD is defined as operating conditions under which a given driving automation system or feature is specifically designed to function, including, but not limited to, environmental, geographical, and time-of-day restrictions, and/or the requisite presence or absence of certain traffic or roadway characteristics.” (SAE J3016, Taxonomy and Definitions for Terms Related to Driving Automation Systems for On-Road Motor Vehicles)

The significance of Defining the Operational Design Domain (ODD) includes:

• An early definition of the ODD in the design process delineates the functional scope of the Autonomous Ground Vehicle (AGV) and the conditions it must safely navigate.
• The ODD definition highlights areas where enhancements to system capabilities are necessary. For example, a vehicle that cannot operate in heavy snow conditions may prompt devising a perception solution for snow operations.
• Establishing the ODD helps sample a number of AGV operational scenarios, generating test cases with varying levels of complexity.?
• The first ODD is defined as the tightest boundary or radius within which the AGV has been designed and tested to operate. By passing a series of milestones, the AGV can operate in increasingly complex scenarios or ODDs.?
• Monitoring ODD compliance during operations ensures that the AGV operates within defined operational domain boundaries.

Defining the ODD is essential in developing and deploying safe AI systems. The ODD specifies the exact conditions the AI system intends to operate, including environmental, geographical, and operational parameters. This clear definition helps focus the AI development process on relevant data and scenarios, significantly enhancing safety by preventing the AI from encountering situations it is not equipped to handle.? For example, the ODD for an autonomous haul truck in a mining operation includes specific routes, speed limits, types of materials transported, and the presence of human workers. By defining these parameters, SafeAI ensures that the AI system is designed and tested to perform safely within its intended operational context.

Assessing the Impact of AI Output on Operational Safety

Once the AI’s purpose is defined, the next critical step is to assess how its output will impact operational safety. This assessment is crucial to ensure that AI systems do not inadvertently compromise safety but enhance operational efficiency and reliability. In autonomous vehicle operations within a mining site, AI technologies are employed in two primary areas: perception decision-making and site and asset planning and management.?

1. AI in the autonomous vehicle perception system

The perception system relies on advanced AI models, including Convolutional Neural Networks (CNNs), Deep Neural Networks (DNNs), and Transformer architectures for Generative AI. These models process data collected by LiDAR, cameras, and radar sensors to understand the truck's surroundings. This includes identifying and classifying terrain types, detecting obstacles, and tracking other vehicles. The AI models are trained to detect, classify, and track objects accurately, ensuring that the autonomous mining truck can navigate complex environments safely. For instance, the CNNs might be used for real-time image recognition from camera feeds, while DNNs could process LiDAR data to create a detailed 3D map of the terrain.

2. AI in the autonomous vehicle decision-making system?

The decision-making system, on the other hand, generates a plan of action for the autonomous vehicle. It intelligently fuses data from multiple sensors and utilizes AI models often optimized using genetic algorithms. These AI models analyze the sensory input to make informed decisions in real-time, such as obstacle avoidance, path planning, and collision mitigation. For example, when an obstacle is detected, the decision-making system evaluates multiple potential paths, taking into account the truck's speed, the terrain, and the location of other vehicles, and then selects the safest and most efficient route.

By integrating these sophisticated AI models and algorithms, autonomous haul trucks can operate autonomously and safely. The perception system ensures that the truck is constantly aware of its environment, while the decision-making system enables it to react appropriately to dynamic changes and potential hazards. This dual approach enhances the operational efficiency of mining activities and significantly reduces the risk of accidents and equipment damage, thereby ensuring a safer working environment.

3. AI in the autonomous system, including site planning and asset? management

In addition to perception and decision-making systems, AI plays a crucial role in site planning and asset management within site operations. AI systems can analyze historical and operational data to optimize site and asset planning. By leveraging machine learning algorithms and advanced data analytics, AI can predict the most efficient ways to allocate resources, manage assets, and plan site layouts to maximize productivity and safety.??

For instance, AI can evaluate historical data on equipment usage, maintenance records, and production outputs to identify patterns and trends that inform future planning. This allows for predictive maintenance scheduling, reducing downtime, and extending the lifespan of critical machinery.?

Additionally, AI can assist inventory management by predicting the demand for spare parts and consumables, ensuring that necessary supplies are available without overstocking.??

Identifying potential failure points

Once the Operational Design Domain (ODD) is established and the AI model selected, two critical safety considerations must be addressed.

The first involves the input data fed into the model. It’s essential to clearly define the data the AI perception system will process, including sensor data from cameras, LiDAR, radar, or other relevant sources based on the application.

Having pertinent and representative training data specific to the defined ODD is crucial to minimize disparities between model predictions and actual outcomes. Thus, bounding the data input into the model becomes imperative to ensure accurate predictions.

Once the model is trained, the second key safety aspect is the inference process, where AI makes predictions or decisions. In scenarios where AI is involved in decision-making, such as at speeds of 45 mph, an erroneous output can lead to severe consequences as per the hazard analysis.

Both accuracy and timing are critical during inference. If the inference process takes too long—for instance, two minutes while the vehicle is moving at 45 mph—it may not be suitable for our application. We need to assess the potential severity of AI prediction errors and determine the maximum acceptable delay in providing accurate outputs.

The Importance of AI Accuracy in Perception

In our system’s perception, misidentifying obstacles on the road can lead to significant consequences. For instance, if the AI system misclassified an obstacle as a truck when it’s a human operator, relying solely on AI output could be risky. Therefore, it’s essential to integrate a deterministic process alongside AI outputs.

A deterministic approach to bounding AI perception predictions involves several critical steps to ensure the reliability, accuracy, and adherence of AI systems to defined boundaries.

In this context, we aim to establish explicit bounds or constraints within the AI Perception system. This may involve limiting object sizes, speeds, distances, or environmental conditions within which the system functions.

We can also incorporate techniques for estimating uncertainty in predictions, such as confidence intervals, probabilistic models, or uncertainty quantification methods. This helps identify situations where the model’s predictions may be less reliable or human intervention may be necessary.

Finally, we can establish a feedback loop to continuously update and improve the AI perception system based on new data, user feedback, and performance evaluations. This iterative process helps refine predictions, mitigate errors, and adapt to evolving conditions.

Multi-layered Approach to AI Safety

This article explored a multi-layered approach to AI safety, including domain-specific checks, robust validation mechanisms, and integrated safety modules. These components collectively bolster the reliability and trustworthiness of AI systems, especially in safety-critical contexts.

The implementation of domain-specific checks involves tailoring safety measures to AI systems’ unique operational environments. Furthermore, integrating safety modules adds a layer of protection by incorporating fail-safe mechanisms, emergency protocols, and risk mitigation strategies into AI systems. These modules are designed to detect and respond to potential hazards, errors, or anomalies, thereby minimizing the likelihood of accidents or operational disruptions.

Our next article in this series will delve further into the mechanisms we employ to ensure AI safety in autonomous vehicles used in construction and mining operations.