Unveiling the Power of Explainable AI: Demystifying the Future of Artificial Intelligence

Imagine a manufacturing company that employs workers on a production line to operate heavy and potentially dangerous equipment in the production of steel tubing. Seeking to enhance efficiency and safety, the company invests heavily in the development of a cutting-edge artificial intelligence (AI) model created by a team of machine learning (ML) experts. The hope is that this AI model will empower frontline workers to make safer decisions, revolutionizing their business. However, when the complex, high-accuracy model is introduced to the production line, it doesn't gain much traction among workers. What went wrong in this scenario?

This fictional tale, inspired by a real-life case study from McKinsey’s The State of AI in 2020, illustrates the critical role explainability plays in the realm of AI. While the AI model may have been safe and highly accurate, it lacked something crucial: transparency. The frontline workers couldn't trust the AI system because they didn't understand how it arrived at its decisions. In high-stakes situations like these, end-users should have insight into the decision-making processes of the systems they rely on. Predictably, McKinsey's research revealed that improving explainability led to greater technology adoption.

Explainable artificial intelligence (XAI) emerges as a powerful tool for addressing essential "How?" and "Why?" questions about AI systems, while also mitigating ethical and legal concerns. AI researchers have recognized XAI as a vital component of trustworthy AI, leading to a surge in interest. However, despite this growing interest and the widespread demand for explainability across various domains, XAI still faces several limitations. This blog post provides an overview of the current state of XAI, highlighting both its strengths and weaknesses.

Understanding Explainable AI (XAI)

While explainability research is abundant, precise definitions of explainable AI remain fluid. For the purposes of this discussion, we define explainable AI as the collection of processes and methods that enables human users to comprehend and trust the outcomes generated by machine learning algorithms. This definition encompasses various explanation types and audiences, acknowledging that explainability techniques can be applied to a system as needed.

Across academia, industry, and government, experts are exploring the benefits of explainability and developing algorithms to cater to diverse contexts. In healthcare, for example, explainability is deemed essential for AI clinical decision support systems. This transparency facilitates shared decision-making between medical professionals and patients while offering transparency into system operations. In finance, explanations of AI systems assist in meeting regulatory requirements and equipping analysts with the information necessary to audit high-risk decisions.

Explanations can take various forms depending on the context and objectives. Figure 1, for instance, displays human language and heat-map explanations of model actions. The model in question identifies hip fractures in frontal pelvic X-rays and targets medical professionals. The "Original report" provides a doctor's assessment based on the X-ray, while the "Generated report" offers an explanation of the model's diagnosis along with a heat map highlighting influential regions in the X-ray. This user-friendly-generated report aids doctors in understanding and validating the model's diagnosis.

Additionally, Figure 2 showcases a technical, interactive visualization of neural network layers. This tool enables users to manipulate the network's architecture and observe how individual neurons evolve during training. Heat-map explanations of ML model structures provide valuable insights into opaque models for ML practitioners.

Figure 3 depicts a graph from Google’s What-If Tool, illustrating the relationship between inference score types. This interactive visualization empowers users to analyze model performance across different data "slices," identify key input attributes affecting decisions, and detect biases or outliers. Such graphical explanations, though most interpretable by ML experts offer valuable insights for non-technical stakeholders.

The purpose of explainability is to address stakeholder inquiries regarding the decision-making processes of AI systems. Developers and ML practitioners use explanations to ensure that project requirements are met during the model's development, debugging, and testing phases. Explanations also aid non-technical audiences, such as end-users, in understanding AI system operations and addressing questions or concerns. This enhanced transparency fosters trust and supports system monitoring and auditability.

Techniques for achieving explainable AI have been devised for all stages of the ML lifecycle. Methods exist for analyzing the data used in model development (pre-modeling), incorporating interpretability into the system's architecture (explainable modeling), and generating post-hoc explanations of system behavior (post-modeling).

Why the Surge in Interest in XAI?

As AI has evolved, increasingly complex and opaque models have been deployed to tackle challenging problems. These models, due to their intricate architecture, are more challenging to understand and oversee than their predecessors. When these models fail or behave unexpectedly, it can be difficult for developers and end-users to discern the cause and rectify it. XAI addresses this challenge by offering insights into the inner workings of opaque models, resulting in significant performance improvements. For example, IBM's research indicates that users of their XAI platform achieved a 15% to 30% increase in model accuracy and a $4.1 to $15.6 million rise in profits.

Transparency becomes crucial as AI systems play a more prominent role in our lives, with decisions carrying substantial consequences. In theory, these systems can eliminate human bias from historically prejudiced decision-making processes, such as bail determinations or home loan eligibility assessments. However, AI systems inadvertently perpetuated discrimination due to biased training data. As reliance on AI systems for critical real-world decisions grows, thorough vetting and adherence to responsible AI (RAI) principles become imperative.

Legal requirements to address ethical concerns and violations are evolving. The European Union's General Data Protection Regulation (GDPR) and the California Consumer Privacy Act (CCPA) require meaningful explanations of automated decisions, further emphasizing the need for transparency.

Current Limitations of XAI

One challenge facing XAI research is the lack of consensus on key terms. Precise definitions of explainable AI vary across papers and contexts. Some use "explainability" and "interpretability" interchangeably to denote making models and their outputs understandable, while others draw nuanced distinctions. Standardizing terminology is essential for discussing and researching XAI effectively.

Moreover, while numerous papers propose new XAI techniques, practical guidance on selecting, implementing, and testing these explanations to meet project needs is scarce. While explanations enhance understanding for many audiences, their ability to build trust among non-experts remains a subject of debate. Research is ongoing on how to leverage explainability effectively to instill trust among non-expert users, with interactive explanations showing promise.

Another point of contention is whether explainability holds more value than other methods for providing transparency. Some argue for the replacement of opaque models with inherently interpretable models, while others advocate rigorous testing, including clinical trials, over explainability, particularly in the medical domain. Human-centered XAI research suggests that XAI must encompass social transparency, going beyond technical transparency.

Why the SEI is Exploring XAI

Explainability has been recognized by the U.S. government as a critical tool for fostering trust and transparency in AI systems. Deputy Defense Secretary Kathleen H. Hicks stressed the importance of trust in AI systems during the Defense Department's Artificial Intelligence Symposium and Tech Exchange. Building a robust responsible AI ecosystem has become a priority, with the adoption of ethical principles for AI. This reflects a growing demand for XAI within the government. The U.S. Department of Health and Human Services also places a focus on promoting ethical, trustworthy AI use and development, including XAI, as part of their AI strategy.