What are the best ways to evaluate explainable AI?

Evaluating explainable AI is like checking if a recipe makes sense. You want to be sure it's clear and doesn't have any strange ingredients. The best ways to do this involve two main things: First, make sure the AI can tell you why it makes certain decisions, just like a chef explaining their cooking steps. Second, test it with different situations, like tasting the same recipe with different ingredients. Ask questions like, "Why did you recommend this?" and see if the AI can explain. Also, see if it's consistent and doesn't act weird in different situations. It's all about making AI as understandable as your favorite recipe – clear, reliable, and tasty!

This is too intrusive. An AI system should not attempt to know my background, goals, preferences and level of expertise in order to provide me an explanation/answer to my question. It should only use my input to generate an answer. If I don't understand something, I will ask a follow-up question. My goals, preferences and level of expertise are not static. My background may be in one area and I may be asking a question about a different area. This paragraph is a recipe for building manipulative AI systems.

Different users may have varying levels of familiarity with AI systems. Evaluating XAI's usability involves assessing how easily users can interact with the system, interpret its explanations, and make informed decisions. User satisfaction is an important metric. Did the XAI system meet users' expectations, and did they find the explanations provided by the system helpful and informative? Users' trust and confidence in the AI system are critical. Some users, particularly domain experts, may require a high level of trust in the system's decision-making process. Understanding how much trust users place in the XAI system is valuable.

The accuracy of the AI system in achieving the task's objectives is fundamental. Classification tasks involve measuring how often the system correctly classifies instances. Reliability is essential, especially in tasks where decisions have significant consequences. It assesses the AI system's consistency in making accurate decisions over time. Efficiency is crucial for tasks that require real-time or near-real-time responses. Evaluating efficiency involves measuring the speed at which the AI system can complete its tasks without compromising accuracy. Fairness is critical, particularly in tasks involving decision-making for individuals or groups.

This paragraph doesn't explain task objectives and metrics though it mentions "different aspects" earlier. It should indicate what part of the input contributed to what fraction of the output score. It should also provide a what-if scenario of if your input had been this then the output would have been that for these reasons. If I say for my inputs I expected that output then it should say why I am thinking incorrectly (which could tell me why it is thinking incorrectly).

It is crucial to understand that the more complex a model is, it might be more accurate in its predictions, but at the same time, it could be harder to interpret. This is a constant challenge in the field of artificial intelligence: balancing accuracy with interpretability. Therefore, the most complex model isn't always the best for every situation. In contexts where decisions based on AI predictions can have critical consequences, such as in medicine or finance, it's essential that models are not only accurate but also understandable to humans. This allows for an additional level of review and understanding before making decisions based on their results.

There are multiple ways to look at the model dimension. First, visual methods must produce easy to understand results for a general audience. From another point of view, when you interact with another person you don't examine their brain to understand or believe what they are saying. Irrespective of the model, if it produces an answer to my question that an independent, impartial observer would rate as a valid answer, then I think the explainability of the answer is implicit.

Generative AI offers an approach for probing bias within datasets and decision-making processes. By scrutinizing the model's prioritization of variables during decision-making and evaluating the resultant impact on generated data, one can effectively explored in more depth these aspects.

Surrogate models, i.e. interpretable models that are trained with the predictions of our “black box” model to gain a global approximation of that model, can be useful to gain insights into the inner workings of the black-box model. However, global surrogate interpretations do not come with any mathematical guarantee of validity and thus we often turn to Local Surrogate models which more accurately is able to find local, monotonic regions in a complex model’s decision boundary. We will be discussing the two most popular methods in this category that are model agnostic explainers – LIME and SHAP.

In addition to the points mentioned, it's crucial to emphasize the importance of data provenance. An AI system is only as good as the data it has been trained on. If the quality of the dataset is low or contains biases, this will be reflected in the model's predictions and explanations. Therefore, a thorough evaluation of XAI should not only consider how data is represented and related but also how it was collected, processed, and labeled. Furthermore, it's vital to recognize that transparency in the data phase can play a crucial role in building trust. If users and stakeholders understand and trust the data feeding a system, they are more likely to trust the model's predictions and explanations.

Most of this paragraph seems to be about things one can do while training the model. After deployment, the model should always tell me what part of my input contributed to what extent to the output. In addition to showing relevant regions, for example, it should provide a ranking of the regions to show their relative contributions to the output. If it shows the distribution of the input it is trained on and show where in that distribution my input falls, it would be even better.

Transfer in reinforcement learning is a novel research area that focuses on the development of methods to transfer knowledge from a set of source tasks to a target task. Whenever the tasks are similar, the transferred knowledge can be used by a learning algorithm to solve the target task and significantly improve its performance (e.g., by reducing the number of samples needed to achieve a nearly optimal performance).

We should discuss 2 popular fairness metrics and examine their ability to detect unfair bias in predictions 1.Disparate Impact: Often also termed as “Statistical Parity” aims to remove disparate impact, which occurs when a selection process has widely different outcomes for different groups, even as it appears to be neutral. 2. Equal Opportunity: This metric compared True positive rates between protected groups. E.g. in the context of bank loans, Equal opportunity metric requires the bank to lend both groups at the same rate among individuals who can repay their loan.