Demystifying AI: The Power of Explainability in the Age of Language Models

Explainability in AI

Businesses are increasingly relying on artificial intelligence (AI) systems to make decisions that can significantly affect individual rights, human safety, and critical business operations. But how do these models derive their conclusions? What data do they use? And can we trust the results?

Explainability in AI refers to the ability of an artificial intelligence system to provide understandable and clear explanations for its decisions or actions. It aims to simplify complex AI algorithms, making their inner workings transparent to humans. Explainable AI is crucial for:

• Gaining trust.
• Ensuring accountability.
• Comprehending why AI systems make specific choices or predictions.

Let us understand this through a simplified loan application example:

When applicants are denied a loan, they will receive a straightforward justification, "Everyone who is over 40 years old, saves less than $433 per month, and applies for credit with a payback period of over 38 years will be denied a loan. The same goes for younger applicants who save less than $657 per month."

?In the realm of artificial intelligence (AI), the concept of Explainability is pivotal. AI models are often structured in two distinct formats: white box and black box.?

• White box models provide more visibility and understandable results.
• Black box models are extremely hard to explain.

Explainable AI techniques:

The setup of Explainability techniques consists of the main following methods:?

A. Prediction Accuracy: We determine prediction accuracy through simulations and comparing AI outputs to training data. Local Interpretable Model-Agnostic Explanations (LIME) is the popular method for explaining classifier predictions by the machine learning algorithm.

For example, the following image of a Lime model interpretation for the Boston housing dataset suggests that the relatively higher price of a house in the given data can be attributed to various socio-economic factors, such as neighbourhood status, house size, property tax rates, education quality, age of the property, and the presence of non-retail businesses.

B. Traceability: To enhance transparency in AI, we limit decision-making processes and narrow the scope of machine learning rules and features. DeepLIFT (Deep Learning Important Features) is an example of a traceability technique that links the activation of each neuron to a reference neuron, revealing dependencies and creating traceable connections.

C. Model Explainability: Shape (Shapley Additive Explanations) is another powerful technique used to explain AI models. It provides a structured way to understand the contribution of individual features or variables in a model's decision-making process. For image recognition models, Shape can provide insights into why a specific image was classified as, for example, a cat or a dog. It can reveal the importance of various image features, like fur texture, colour, or the presence of certain objects, in the model's decision.

Here is an example of how SHAP values can be used to interpret the output of an XGBoost model (XGBoost -eXtreme Gradient Boosting is a machine learning library that implements an optimized and distributed induction-based approach for classification and prediction) for consumer's propensity to buy instant and micro-insurance policies from a technology-driven insurance company using Artificial Intelligence and smartphone data, to understand consumer preferences and provide insurance coverage tailored to their specific needs.

To understand this better, researchers trained an XGBoost model, and they interpreted the results using SHAP values. The figure below contains the SHAP summary plot from TreeSHAP, which shows the contribution of each variable by representing its Shapley value averaged across all customers.

SHAP values help us understand how each variable influences the model's predictions. They provide insights into the relative importance of each feature and how it impacts user behaviour. This information can be used to make data-driven decisions and optimize strategies for encouraging policy purchases.

1. Days Since Last Interaction: SHAP analysis indicates that recent interaction significantly boosts purchase likelihood.
2. Number of Bought Items: SHAP values demonstrate that users who have bought more items are more likely to make additional purchases.
3. Seasonality and Weekdays: SHAP values reveal how seasonality and weekdays influence user behaviour, such as their likelihood to purchase policies during specific seasons or on particular weekdays.
4. Device Type: SHAP values indicate that users on certain device types may have a higher or lower likelihood of making a purchase.

D. Decision Understanding: This is the human factor. This is accomplished by educating the team working with the AI so they can understand how and why the AI makes decisions.

Benefits of Explainability in AI:

• Operationalize AI with trust and confidence.
• Speed time to AI results.
• Mitigate risk and cost of model governance.

Few use Cases of Explainable AI:

• Healthcare: Accelerate diagnostics, image analysis, resource optimization, and medical diagnosis. Improve transparency and traceability in decision-making for patient care. Streamline the pharmaceutical approval process with explainable AI.
• Financial services: Improve customer experiences with a transparent loan and credit approval process. Speed credit risk, wealth management, and financial crime risk assessments. Accelerate resolution of potential complaints and issues. Increase confidence in pricing, product recommendations, and investment services.
• Legal: Optimize processes for prediction and risk assessment. Accelerate resolutions using explainable AI on DNA analysis, prison population analysis, and crime forecasting. Detect potential biases in training data and algorithms.

Explainability in LLMs

Large language models (LLMs) have demonstrated impressive capabilities in natural language processing. Since LLMs are notoriously complex “black-box” systems, their inner working mechanisms are opaque, and the high complexity makes model interpretation much more challenging. This lack of model transparency can lead to the generation of harmful content or hallucinations in some cases.

Improving the Explainability of LLMs is crucial for two key reasons:

• End users are able to understand the capabilities, limitations, and potential flaws of LLMs.?
• For Researchers and Developers, explaining model behaviours provides insight to identify unintended biases, risks, and areas for performance improvements.

We categorize LLM Explainability techniques into three major paradigms:

Paradigm 1: Traditional Fine-Tuning Paradigm: The traditional fine-tuning paradigm in Large Language Models (LLMs) involves refining a pre-trained model on specific tasks or domains, enhancing its performance by adapting it to specialized tasks or data.?

A. Local Explanations: These techniques help us understand why an LLM made a specific prediction.

• Attention Visualization reveals which parts of the input is the LLM focused on.
• Input Saliency shows how sensitive the prediction is to changes in input.
• Counterfactual Explanations generate alternative inputs for different outcomes.

B. Global Explanations: These methods aim to explain the overall behaviour of an LLM.

• Feature Analysis identifies which aspects of input are most influential.?
• Model Distillation simplifies the LLM for easier explanation.
• Decision Trees create tree structures to analyze how the LLM makes?decisions.

Paradigm 2: Model Introspection: The Model Introspection Paradigm in Large Language Models (LLMs) focuses on enhancing model interpretability and transparency by enabling better insight into how it generates responses and decisions.

A. Self-attention: LLMs can use this mechanism to explain their reasoning process by revealing which parts of their own internal states they're paying attention to.

B. Probing: Auxiliary models are trained to predict what's happening inside the LLM. These auxiliary models help explain the LLM's predictions.

C. Explainable Modules: We modify LLM modules, like attention layers, to make them produce more understandable results.

Paradigm 3: Human-in-the-loop Explainability: This approach focuses on creating ways for humans to interact with LLMs to understand their predictions and behaviours.

A. Interactive Explainers: Tools are designed to let users engage with LLMs to explore predictions and explanations.?

B. Co-interpretation: Humans and LLMs work together to explain predictions. Humans can ask clarifying questions or provide context, helping the LLM generate more informative explanations.

Real-life examples of Explainability in AI and LLMs

1. Pharmaceutical Document Classifier Precision.

The Slimmer AI Science team built a pharmaceutical document classifier. They used Local Interpretable Model-Agnostic Explanations (LIME) for interpretability, discovering that a seemingly random number (169) was vital due to copyright tags. They improved preprocessing to enhance model reliability.

2. Medical Image Trustworthiness

The team developed an AI app for skin infection detection around ventricular assist device drivelines. Using Grad-CAM, they found the model sometimes focused on the driveline, not the wound. They filtered the driveline from images to boost model accuracy.

3. Debt Prediction Model Privacy

Collaborating with an institution to predict severe debt, the team trained Logistic Regression( Logistic Regression is a linear analysis approach that uses a generalized linear equation to describe the directed dependencies among a set of variables) and XGBoost Models on one database. XGBoost outperformed, and combining a second database didn't improve results. Feature analysis revealed redundancy, allowing the client to prioritize privacy and the first database.

Conclusion

In the evolving landscape of artificial intelligence, the pursuit of Explainability remains paramount. It serves as a guide for ensuring the reliability and accountability of AI systems. Through techniques like LIME and paradigms like fine-tuning, the complex inner workings of AI models become more transparent.

Embracing and advancing Explainability stands as a technological necessity, as demonstrated by the real-world applications that also underscore the practical benefits of Explainability.

Sources:

• https://www.algolia.com/blog/ai/what-is-explainable-ai-and-why-is-transparency-so-important-for-machine-learning-solutions/
• https://www.techtarget.com/whatis/definition/explainable-AI-XAI
• https://www.arxiv-vanity.com/papers/2309.01029/
• https://www.ibm.com/topics/explainable-a
• https://towardsdatascience.com/explainable-ai-in-practice-6d82b77bf1a7
• https://www.researchgate.net/figure/The-framework-of-XAI_fig2_350848308
• https://www.researchgate.net/publication/347633178_Why_to_Buy_Insurance_An_Explainable_Artificial_Intelligence_Approach
• https://www.geeksforgeeks.org/introduction-to-explainable-aixai-using-lime/