How Explainable AI is demystifying machine learning models

Explainable AI (XAI) demystifies machine learning models, making their decision-making process transparent and understandable. Think of it as the difference between a black box and a clear glass jar - XAI allows us to see the ingredients involved and the recipe used to make predictions. Why does this matter and who should care? Let’s dive in!

Why XAI matters

Not long ago deep learning-based analysis of COVID-19 X-ray images made headlines: researchers unveiled a crucial flaw in AI diagnostics where the machine learning model was? making predictions based on image artifacts rather than actual disease markers. High stake domains other than healthcare are not short of similar controversial examples. What has gone wrong in these scenarios?

Unlike human reasoning, the reasoning of machine learning models lacks context. What the model is focused on is learning a pattern that gives rise to a certain prediction, may it be an artifact of imaging machines in the COVID scenario. In other words the model might be making right predictions for the wrong reasons.?

XAI is a set of methods, processes and frameworks that aim at demystifying machine learning models, making their decision-making process transparent and understandable. The transparency then allows verifying whether the model is right for the right reasons and subsequently earning human trust.

Who should care

The pursuit of trustworthy AI is of value to:

• Individuals who are subjected to algorithmic decision making. The EU’s General Data Protection Regulation (GDPR) gives individuals the right to ask for an explanation of a decision made about them and potentially object to the decision.?
• Domain expert stakeholders who want to leverage the power of machine learning. Explanation of what a machine learning model makes its decision based on enables domain experts to verify the correctness of decision making and intervene if the need be.
• AI practitioners who want to debug their models and ensure their generalisability. Explainability helps detecting bias, errors and unexpected behaviour in models and consequently mitigating them.
• Policy makers who need to form and enforce regulations governing the safe and ethical use of machine learning in sensitive areas such as finance, healthcare, and criminal justice.?

How XAI works

The tool box of XAI ranges from explainability methods that are model agnostic (post-hoc methods) to architectures that are inherently explainable. The explainability itself can take various forms ranging from explanation in the form of features important for a prediction to training data points (i.e. examples) important for a prediction.?

Figure 1 below shows a saliency map, a type of visual explanation which highlights the regions of an image that contribute most to a prediction. In this example, the bold red highlighted region on the right hand side image, identified by saliency techniques, reveals that areas outside patient’s lung are the main determining region in predicting a COVID-19 positive case. This unexpected insight into the model’s decision making points to the undesired and not biologically relevant spurious “shortcuts” that the model exploited to make its predictions.

Figure 1. from “AI for radiographic COVID-19 detection selects shortcuts over signal” https://www.ncbi.nlm.nih.gov/pmc/articles/PMC7523163/#R16

In Figure 2 the popular feature importance method, SHAP, is used in a sentiment analysis scenario to show what words in a sentence are most important in predicting the sentiment of this sample as “sadness”. The bolder red correlates to more contribution, rightly, making “hopeless” the most important word in the sentence conveying the “sadness” sentiment.?

Figure 2. From “Emotion classification” https://shap.readthedocs.io/en/latest/example_notebooks/text_examples/sentiment_analysis/Emotion%20classification%20multiclass%20example.html

Figure 3 shows how example-based explanation methods can shed light on why a bird is? incorrectly classified as an airplane given their similarity. The likely cause of this misprediction is the lack of similar examples to the input image that are indeed birds and not planes.? The information provided by this method of explainability can thus be used to enrich the input data with examples similar to the input image that are labeled correctly, increasing the likelihood of model to classify future examples correctly.?

Figure 3. from “Example-based explanations improve ML via explainability” https://cloud.google.com/blog/products/ai-machine-learning/example-based-explanations-to-build-better-aiml-models

Current limitation of XAI

Whilst the current explainability methods have come a long way in terms of highlighting the potential errors and biases a model has, there are several obstacles facing their wide uptake in practice.?

An open problem is the lack of well-defined frameworks, metrics and principles that allow evaluating explainability methods and their effectiveness. There is also the issue of whom the explanation serves. One size does not fit all! The effectiveness of explanation is directly linked with their users. The type of explanation that is effective for an AI practitioner to debug their models is unlikely to be effective for a policy maker who wants to ensure the ethical and safe use of an AI-enabled technology. Whilst researchers suggest extensive user studies are required to determine effectiveness of explanations for various stakeholders, this type of studies are far and few in between. A combination of user studies and technical work on well-formulated metrics and guidelines is required to provide a framework for evaluating XAI rigorously.?

Another obstacle is the lack of actionability. Feature importance methods in particular suffer from this. Whilst these methods identify? which features are important for a model, they do not provide any information on how these features should be changed to achieve a desired outcome. For example, knowing that "income" is a significant feature in predicting loan approval does not specify how to modify income or other factors to increase the likelihood of approval. In image data this is even more complicated as raw inputs (e.g., pixel intensities) are not necessarily semantically meaningful to reason about. New lines of research, such as concept-based explainability (C-XAI) try to remedy this by composing explanations at a higher level of abstraction. Such explanations are accessible to domain experts and policy makers as well as AI practitioners, allowing them to easily intervene on concepts they understand and relate to. Figure 4, shows an application of C-XAI in detecting arthritis severity based on clinical concepts such as “bone spurs” rather than raw input pixels.

Figure 4. from “Concept bottleneck models (CBMs)”?

https://proceedings.mlr.press/v119/koh20a.html

It is time

Deep learning is ripe enough to revolutionise many high stake domains. Lack of trust, however, is a major challenge facing deploying AI in safety and security critical domains, where technological break throughs are needed the most. XAI aims to unlock the full potential of AI by shedding light on how models arrive at their decisions, paving the way for the uptake of? fair, ethical, and reliable AI. Along side research advancement required to make explanations robust and accessible to stakeholders, strong push from regulatory bodies is needed to enforce explainability as a requirement in critical domains, encouraging more advanced explainability efforts making their way from research labs to practice.