Explainable AI: What, Why and When?

Artificial intelligence (AI) is transforming every aspect of our lives, from healthcare to finance, from education to entertainment. AI systems are making decisions that affect millions of people, such as diagnosing diseases, approving loans, recommending products, and detecting fraud. But how can we trust these systems? How can we ensure that they are fair, reliable, and ethical? How can we understand how they work and why they make certain choices?

These questions are at the heart of explainable AI (XAI), a field of research that aims to make AI systems more transparent, interpretable, and accountable. XAI is not a new concept, but it has gained more attention and urgency in recent years, as AI becomes more pervasive and powerful. In this blog post, I will explain what XAI is, why it is important, and when it is needed.

What is XAI?

XAI is a broad term that encompasses various methods and techniques to make AI systems more understandable and explainable to humans. There is no single definition or standard for XAI, but one possible way to categorize XAI approaches is based on the level of abstraction and the target audience:

• Model-level explanations?aim to provide a global understanding of how an AI system works as a whole, such as its architecture, parameters, features, and learning process. These explanations are usually intended for technical experts, such as developers, researchers, or regulators, who need to verify, debug, or audit the system.
• Decision-level explanations?aim to provide a local understanding of how an AI system makes a specific decision or prediction for a given input, such as the factors, rules, or evidence that influenced the outcome. These explanations are usually intended for end-users, such as customers, patients, or employees, who need to trust, accept, or challenge the system.

Depending on the type and complexity of the AI system, different XAI methods may be more suitable or effective. For example:

• Intrinsic methods?rely on designing or modifying the AI system itself to make it more inherently interpretable or transparent. These methods often involve using simpler or more structured models, such as decision trees, linear models, or rule-based systems, that can provide clear and intuitive explanations. However, these methods may also sacrifice some performance or accuracy compared to more complex or flexible models.
• Post-hoc methods?rely on applying external techniques or tools to analyze or approximate the AI system after it is trained or deployed. These methods often involve using visualizations, feature importance scores, local approximations, or counterfactual examples to provide insights or evidence for the system’s behaviour. However, these methods may also introduce some errors or inconsistencies compared to the original system.

Why is XAI important?

XAI is important for several reasons:

• Ethical reasons: XAI can help ensure that AI systems are aligned with human values and norms, and that they do not violate human rights or cause harm. For example, XAI can help detect and mitigate bias, discrimination, or unfairness in AI systems that may affect certain groups of people negatively. XAI can also help respect and protect the privacy and dignity of individuals whose data or decisions are affected by AI systems.
• Legal reasons: XAI can help comply with existing or emerging laws and regulations that require AI systems to be transparent, accountable, or explainable. For example, XAI can help meet the requirements of the General Data Protection Regulation (GDPR) in the European Union (EU), which grants individuals the right to access, rectify, or object to automated decisions made about them by AI systems. XAI can also help provide evidence or justification for legal disputes or challenges involving AI systems.
• Practical reasons: XAI can help improve the quality and reliability of AI systems by enabling technical experts to verify, debug, or optimize them. For example, XAI can help identify errors, bugs, or anomalies in AI systems that may affect their performance or accuracy. XAI can also help fine-tune or enhance the features, parameters, or data of AI systems to achieve better results or efficiency.
• Social reasons: XAI can help foster trust and acceptance of AI systems by enabling end-users to understand, interact, or collaborate with them. For example, XAI can help increase confidence, satisfaction, or loyalty of customers, patients, or employees who use or benefit from AI systems in various domains or applications. XAI can also help facilitate communication, education, or awareness of stakeholders, policymakers, or the public about the benefits and risks of AI systems.

When is XAI needed?

XAI is not always necessary or desirable for every AI system or situation. There may be cases where XAI is not feasible or useful, such as when the AI system is too complex or dynamic to be explained, or when the explanation is too technical or lengthy to be understood. There may also be cases where XAI is not appropriate or ethical, such as when the explanation reveals sensitive or proprietary information, or when the explanation manipulates or deceives the human recipient.

Therefore, XAI should be applied with care and context, taking into account various factors, such as:

• The purpose of the AI system: What is the goal or function of the AI system? Is it for entertainment, education, advice, recommendation, diagnosis, prediction, decision-making, or something else? How does it affect human lives or well-being?
• The impact of the AI system: What are the potential benefits or harms of the AI system? How significant or severe are they? Who are the beneficiaries or victims of them? How likely or frequent are they?
• The complexity of the AI system: How complicated or flexible is the AI system? How many features, parameters, layers, or components does it have? How does it learn or adapt over time?
• The audience of the explanation: Who are the recipients or consumers of the explanation? What are their backgrounds, roles, interests, expectations, or preferences? How much do they know about AI? How much do they need to know about the AI system?

Based on these factors, XAI may be more needed when:

• The purpose of the AI system is critical or sensitive, such as medical diagnosis, legal judgment, financial investment, military operation, etc.
• The impact of the AI system is high-stakes or irreversible, such as life-or-death situations, human rights violations, environmental disasters, etc.
• The complexity of the AI system is high-dimensional or non-linear, such as deep neural networks, reinforcement learning agents, generative models, etc.
• The audience of the explanation is diverse or demanding, such as customers, patients, employees, regulators, judges, journalists, etc.

AI Model Accuracy Vs Explainability

AI model accuracy and explainability are two important aspects of AI systems that are often seen as conflicting or trade-off. AI model accuracy refers to how well an AI system can perform its intended task, such as predicting, classifying, or recommending outcomes. The correlation between AI model accuracy and explainability is not straightforward or universal, but it depends on various factors, such as the type, complexity, and domain of the AI system, the level and method of explanation, and the expectations and needs of the stakeholders. In general, there are four possible scenarios:

• High accuracy and high explainability: This is the ideal scenario where an AI system can achieve both high performance and high transparency. This may be possible for some simple or structured AI models, such as decision trees, linear models, or rule-based systems, that can provide clear and intuitive explanations. However, this scenario may be rare or challenging for more complex or flexible AI models, such as deep neural networks, reinforcement learning agents, or generative models, that may require more advanced or approximate explanation methods.
• High accuracy and low explainability: This is the common scenario where an AI system can achieve high performance but low transparency. This may be the case for many complex or flexible AI models, such as deep neural networks, reinforcement learning agents, or generative models, that are often considered as black-boxes or difficult to interpret. This scenario may be acceptable for some low-stakes or non-critical applications, such as entertainment, recommendation, or personalization, where the accuracy of the AI system is more important than its explainability.
• Low accuracy and high explainability: This is the undesirable scenario where an AI system can achieve high transparency but low performance. This may be the case for some simple or structured AI models, such as decision trees, linear models, or rule-based systems, that may provide clear and intuitive explanations but may also sacrifice some performance or accuracy compared to more complex or flexible AI models. This scenario may be unacceptable for most applications, especially for high-stakes or critical ones, such as medical diagnosis, legal judgment, financial investment, or military operation, where the accuracy of the AI system is more important than its explainability.
• Low accuracy and low explainability: This is the worst scenario where an AI system can achieve neither high performance nor high transparency. This may be the case for some poorly designed or implemented AI models that may suffer from errors, bugs, anomalies, bias, unfairness, or other issues that may affect their performance or transparency. This scenario may be unacceptable for any application and should be avoided at all costs.

Conclusion

XAI is a vital and challenging field of research that aims to make AI systems more transparent, interpretable, and accountable.

XAI can help ensure that AI systems are ethical, legal, practical, and social, and that they serve human interests and values. However, XAI is not a one-size-fits-all solution, and it should be applied with care and context, depending on various factors, such as the purpose, impact, complexity, and audience of the AI system and its explanation.

I believe that XAI is not only a technical problem, but also a human problem, that requires collaboration and communication among different disciplines, stakeholders,and perspectives.

I hope that this blog post has provided you with some insights and inspiration on XAI, and I invite you to join me in exploring and advancing this exciting and important field.