What is XAI?

Testing Artificial Intelligence (AI) systems in general is hard because the models tend to be complex, solve a specific problem and rely heavily on the quality of the data. 

The underlying programming behind computing systems has traditionally been coded by humans through transparent and visible instruction sets. While the complexity of the software could create myriad interaction effects that yield behaviors outside the scope of a programmer’s intent, ultimately it is a human that fully expresses and understands the choices their software is designed to make and can modify it to address changing conditions.

Therefore, for testing software, the general practice, is to develop a suite of tests where the internals of the software programs are unknown, to make sure the output from the software system is as expected given the data fed in, called, black-box testing. That is, given a model trained to map data inputs to outputs, if the mechanisms underlying predictions cannot be looked at or understood, that model is black-box. In addition, software developers also do white-box testing of the program structure, design, parameters etc. to make sure it matches the requirements.

In machine learning, which underlies most AI solutions, the design process is handed off to algorithms that return a sealed black box which, from the outside, meets the design requirements, but whose internal workings are largely unknown. AI white-box categorization of models and algorithms refers to their interpretability also called explainable AI (XAI). There are two types of XAI. The first type are explanations that help to understand the data better. The second type of explanations are the ones that help to understand the model better. 

The need for XAI is driven by the following challenges with AI:

1)    Bias – the systems often unwittingly encode bias, since the data may not be inclusive enough. There are calls for more transparency on how models are trained including some ideas on how to improve this such as seven very simple steps to design more ethical AI. An example is the difficulty of image systems in recognizing non-white faces.

2)   Brittleness - since the contours of their encoded worldview are not visible, a system that performs with human-like fluency can abruptly and unexpectedly degrade into gibberish with a single changed word that it had incorrectly learned as a key variable. The systems can run afoul of even the most carefully curated data, as machines learn the predictive power of spurious artifacts of their training data their human creators look past. The most common example is identification of wolves based on the snowy background.

3)   The drivers of certain results need to be understood by the expert using them such as a medical diagnosis or loan approval decision. This is not necessary for all AI applications but definitely for ones making critical decisions, and could be needed after the fact such as in understanding the cause of an accident in a driver-less car.

 The current most developed form of XAI is a black box process that incorporates knowledge of the context to perform analysis of AI results using data that is specific to the problem and determine which attributes are driving the results. In other analytics applications we would call this sensitivity analysis. The advantage is that you do not need to open the algorithm to understand how it works but the user can better understand the reason for the decision. 

There are a few available open source options for this type of XAI:

1)  LIME - Local Interpretable Model-agnostic Explanations - because we want to be model-agnostic, what we can do to learn the behavior of the underlying model is to perturb the input and see how the predictions change. This turns out to be a benefit in terms of interpretability, because we can perturb the input by changing components that make sense to humans (e.g., words or parts of an image), even if the model is using much more complicated components as features (e.g., word embeddings).

2)  Google has been an early leader in emphasizing interpretability and how practitioners can build more understandable, representative and resilient AI solutions. Last year the company unveiled its What-If Tool, which offers a range of interactive visualizations and guided explorations of a TensorFlow model, allowing developers to explore how their model interpreted its training data and how subtle changes to a given input would change its classification, yielding insights into the model’s robustness

3)  Microsoft’s InterpretML - InterpretML implements a number of intelligible models—including Explainable Boosting Machine (an improvement over generalized additive models ), and several methods for generating explanations of the behavior of black-box models or their individual predictions. By having an easy way to access many intelligibility methods, developers will be able to compare and contrast the explanations produced by different methods, and to select methods that best suit their needs. Such comparisons can also help data scientists understand how much to trust the explanations by checking for consistency between methods.

However, all of these are black box XAI generated Ad-hoc or post-hoc are more like justifications and may not be capture the truth of the decision process. Explainable AI (XAI )is NOT an AI that can explain itself, it is a design decision by developers. It is AI that is transparent enough so that the explanations that are needed are part of the design process.

There are some arguments as to why this is not achievable. Humans use complex and incomprehensible decision processes as well. These may not be explainable themselves. Or perhaps blackbox models cannot be avoided because the problems we want to solve are too complex and non-linear. If simpler models were used, there will be a trade-off in performance (for explainability) which we don’t want.

However, a major deployer of AI, Darpa, sees this as the next phase of AI development - The Explainable AI (XAI) program aims to create a suite of machine learning techniques that:

• Produce more explainable models, while maintaining a high level of learning performance (prediction accuracy); and
• Enable human users to understand, appropriately trust, and effectively manage the emerging generation of artificially intelligent partners.

The vision is that new machine-learning systems will have the ability to explain their rationale, characterize their strengths and weaknesses, and convey an understanding of how they will behave in the future. The strategy for achieving that goal is to develop new or modified machine-learning techniques that will produce more explainable models. These models will be combined with state-of-the-art human-computer interface techniques capable of translating models into understandable and useful explanation dialogues for the end user.

DARPA’s goal is to enable the next (third) wave of AI systems, where machines understand the context and environment in which they operate, and over time build underlying explanatory models that allow them to characterize real world phenomena. It is not clear how this will be achieved, but there is plenty of investment being done in this cutting-edge space.