Explainability: from philosophical principles to practice.

In the previous blog, I investigated the various meanings of explainability within the EU AI Act tracing back some of its regulatory precedents within GDPR. In this piece, I am going deeper into my central argument. Explainability acts as a fundamental principle for achieving other normative objectives rather than being a goal per se. I argue that within the Act, the primary outcomes related to explainability are user empowerment and regulatory compliance.

Additionally, I will offer a concise overview of AI explainability, focussing on the ‘last mile problem’: the process of conveying explanations from the AI system to the end user. I look at the feasibility of Natural Language Explanations (NLE) and dialogue systems as practical means for delivering context-specific and user-friendly explanations. I adopt the explanatory pragmatism framework (Nyrup & Robinson, 2022) to establish practical design principles for delivering a ‘good explanation’ and a dialogue system that addresses the user empowerment and regulatory compliance objectives described in my analysis of the EU AI Act.

1. Explainability: user empowerment and regulatory compliance

Explainability and user empowerment

The concept of explainability as a user-empowering condition is articulated in Article 13(1) and Recital 47 of the Act. These provisions mandate that high-risk AI systems function in a manner comprehensible to their users, providing them with sufficient information to interpret the system’s operations and utilise it appropriately.

Article 14(4)(c) further delineates human oversight responsibilities, requiring any individual in charge of supervising an AI system to accurately and adequately interpret its outputs. Moreover, Article 14(4)(d) stipulates that the individual should possess the authority to ‘disregard, override, or reverse the output.’ This user-empowering form of explainability is also applicable to other entities, such as law enforcement agencies, as highlighted in Recital 38. Ensuring a level of explanation about AI decision-making is crucial to providing an adequate amount of information when an AI system’s operation conflicts with fundamental rights or standards, for which existing EU law affords mechanisms for legal recourse.

Explainability and regulatory compliance

The notion of explainability as a condition for compliance is articulated in Article 13(2), which prescribes information transparency requirements ‘with a view to achieving compliance with the relevant obligations of the user and the provider.’ This compliance-oriented objective is also evident in Article 11, which outlines technical documentation requirements, and Article 40, which establishes compliance as a necessary safety condition aligned with technical standards (Sovrano et al., 2021).

Annex IV(2)(b) specifies that ‘the design specifications of the system, namely the general logic of the AI system and of the algorithms’ should be accessible to ensure conformity with the AIA prior to the AI system’s public deployment. The system must be explainable to facilitate evaluation against the technical requirements delineated in the regulation and subsequent post-market launch monitoring activities.

Considering the user-empowering and compliance-related objectives within the broader normative framework of the Act, as set out in Recital 5, which seeks to promote ‘the use and adoption of artificial intelligence in the internal market that simultaneously adheres to a high standard of safeguarding public interests, such as health and safety and the protection of fundamental rights,’ explainability serves to mitigate and regulate potential harm resulting from the system’s operation.

The user-empowering and compliance-related dimensions of explainability intersect in Article 29(4) (Sovrano et al. 2021), which mandates that the user should be able to ‘monitor the operation of the high-risk AI system on the basis of the instructions for use.’

In the event that the monitoring process uncovers any risks, the user should have the ability to suspend the system and inform the AI system provider. Arguably, this entails real-time or near-real-time monitoring capabilities, enabling the user to assess any risk that makes the system non- compliant with the Act or surpasses internal risk tolerance thresholds.

Given the Act represents horizontal cross-sectoral legislation, adherence to the regulatory requirements can facilitate positive outcomes for sector-specific regulatory compliance.

2. Zooming out from the Act to the XAI

Explainable AI (XAI) is a growing field of research that seeks to enhance the transparency and comprehensibility of AI models for humans, thereby fostering trust and adoption of AI (Hoffman et al., 2018; Saeed & Omlin, 2023). In this realm, various perspectives have been offered in the literature.

Explanations can be broadly classified into two main types: local and global.

• Local explanations focus on explaining individual model predictions by providing additional information or justification regarding how the prediction was made (Guidotti et al., 2018)
• Global explanations aim to explain the overall functioning of the model, irrespective of specific predictions (Bastani et al., 2017).

Further, there are distinctions as to how explanations are generated.

• Self-explaining models are inherently interpretable, providing explanations alongside the prediction process, and include less complex models, such as decision trees and rule-based models.
• ?Post hoc explanations require additional processing after the prediction is made and are typical of more complex models, such as deep learning models, with LIME (Ribeiro et al., 2016) and SHAP being the most widely adopted post hoc explanation frameworks (Lundberg et al., 2017).

In addition to explanation generation, the explainability field also encompasses how to render the explanation in the most appropriate way to the end user.

• The first is saliency-based representations, which are used to represent the weighting of the scores of the constitutive elements in the model. This is the first and most popular category and is widely used in various domains, such as computer vision (Simonyan et al., 2014) and speech (Aldeneh & Provost, 2017).
• Second, raw declarative representations visualise the logic rules and trees, providing a raw representation. This type of framework assumes that end users have sufficient technical knowledge to understand the specific representations.
• The third category includes Natural Language Explanations (NLE). This explanatory framework provides a human- comprehensible verbalisation of the working of the model in natural language. This type of explanation can be generated in different ways, ranging from deep generative models to simple template-based frameworks.

The focus of my research is on this final stage of AI explainability, specifically the challenge of delivering explanations generated by XAI to end-users in a manner that is both comprehensible and meaningful.

My research investigates the ‘last mile problem’ of AI explainability and NLE. This is an under-researched area compared to explanation generation, and I am interested in evaluating the role of NLE due to the growing prominence of large language models (LLMs) as the default interface for users interacting with AI systems. I hypothesise that using LLMs as the interface can facilitate dialogue systems that enable users to inquire about system functionality and result generation.

3. Explanatory pragmatism

Numerous theories of explanation have been developed in philosophy, which are often influenced by fields such as psychology and linguistics. Prominent theories in contemporary philosophy, including causal realism (Salmon, 1984), constructive empiricism (Van Fraassen, 1980), ordinary language philosophy (Achinstein, 1983), cognitive science (Holland et al., 1989), naturalism and scientific realism (Sellars, 1962), offer distinct definitions of ‘explanation’, sometimes in complementary ways. All theories except for causal realism are pragmatic, as they aim to make explanations specific and tailored for the individual recipients.

Most definitions incorporate the process of question answering as an element of the act of explaining. This pragmatism can also be seen in the objective to adapt explanations to suit individual users, ensuring that the same explainable information is presented and reformulated in a unique manner for each user.

Specifically, I narrow my definition of explanation under the explanatory pragmatism framework (Nyrup & Robinson, 2022), which is based on the following aspects:

• Communicative view of explanation

Explanations are communicative acts, where an explainer shares certain information with an audience to help them achieve relevant comprehension. This definition contains two key concepts. First, explanations should be regarded as speech acts (Austin, 1962) and thus evaluated based on their effectiveness in fulfilling their communicative function. Second, the primary communicative function of explanations is to facilitate audience understanding of the information transferred by the explainer (Franco, 2019).

• Inferentialist view of understanding

Understanding is a context-dependent concept (Kelp, 2015; Wilkenfeld, 2017). Merely acknowledging someone’s ability to draw inferences is not enough to claim that the person has ‘understood’ something. Building on this idea proposed by Nyrup and Robinson (2022), I propose that the conversation plays a crucial role in determining the class of inferences that are relevant for achieving that purpose.

Drawing on these two dimensions of the explanatory pragmatism framework, I use its definition of explainability:

‘Explainability: in the conversational context C, a given phenomenon (model, system, prediction, ...), P, is explainable by an explainer, S, to an audience, A, to the extent S is able to convey information to A that enables A to draw inferences about P that are needed to achieve the purposes that are salient in C. ‘ (Nyrup & Robinson, 2022)

It is important to note that the level of explainability of a phenomenon is always relative to a specific audience and contextual purpose, and without specifying these factors, the application of explainability lacks a clear meaning.

Defining principles of a good explanation

I attempt to translate the philosophical structure presented above into terms that product managers and business analysts can readily understand and apply when designing explainability frameworks.

Principle of Explanation:

1. Factually correct

Explanations must be accurate within the technical framework of the AI model, relating to either a specific prediction (local) or the system’s higher-level workings (global).

2. Useful

Beyond accuracy, explanations need to provide actionable insights that are meaningful to the recipient within their specific context.

3. Context specific

Utility is dependent on the context, informed by the setting’s norms, goals, and user constraints.

4. User specific

The explanation’s utility and context are experienced uniquely by users based on their technical knowledge and relationship to the AI model (e.g., creator, regulator).

5. Pluralism

Explanations should accommodate a range of normative perspectives, offering inferences that help various agents achieve their objectives without imposing a single viewpoint.

Challenges to implementing explainability

Nyrup and Robinson (2022) identify several challenges to explainability, which are common to explanatory pragmatism as well as to explainability frameworks in general.

Information Restrictions

• Confidential or proprietary information may not be shareable. -Audience may face access limitations based on permissions. -Technical literacy issues such as understanding model optimization.

Model Complexity & Size

• Technical difficulty in explaining large, complex models.
• Trade-off between explainability and model complexity.
• Hierarchical abstraction in models is challenging.
• Audience comprehension of complex decision rules with many parameters.
• Audience's Knowledge Integration: 1) Semantic mapping: difficulty in integrating new information with existing knowledge; 2) Domain knowledge: challenges in applying explained information to the audience's operational context.

While certain issues, such as restrictions on information due to confidentiality and intellectual property concerns, may be insurmountable because of IP and legal issues, challenges relating to the complexity of the model and the semantic understanding of the explanation may be addressed through an explanation model that is tailored to the language of the audience.

4. NLE and dialogue systems

After introducing the conceptual framework for a good explanation based on explanatory pragmatism, I propose a pragmatic solution for delivering explanations from AI systems to users. Specifically, I explore the use of NLE and dialogue systems as a delivery mechanism.

NLE has the potential to enhance the user experience and foster trust in AI systems by using familiar language and presenting information in a natural way (Paek & Horvitz, 2000). By making information more accessible and user-friendly, NLE can improve user understanding and trust in the system.

I concentrate on a less frequently employed approach to NLE that involves generating explanations through a dialogue between the user and the AI model. As academic resources and definitions of this approach are scarce, I use a generalised understanding of a ‘dialogue system’, encompassing conversational agents and chatbots (Lakkaraju et al., 2022).

The precedent for using dialogue systems for NLE can be traced back to expert systems, a category of symbolic AI that emerged around the mid-1960s. These expert systems were based on the principle of transferring specific human expertise into a computer. This transferred knowledge enabled the computer to offer advice as needed, similar to a human advisor, and if necessary, to clarify the reasoning behind its suggestions. However, the application of expert systems was restricted due to various overarching AI challenges, including issues related to knowledge representation, generalisation and learning (Liao, 2005).

Rather than offering a rigid, one-directional output that the user must merely accept, a dialogue system allows users to interact with the model using their own language. A dialogue system can improve contextual comprehension and promote user trust in the system.

Below I relate how my five design principles constituting a “sound explanation” can be incorporated into a dialogue system with respect to user empowerment and regulatory objectives under the normative trajectory of the EU AI Act.

1. An explanation should be factually correct

This is a fundamental principle that is agnostic of the explanation delivery framework between the system and the user. The dialogue system needs to be designed so that the models that generate the explanation and transfer it to the user through natural language are based on accurate and reliable information.

User empowerment: The information provided in the explanation can be empowering to the user only if it is correct and relevant to the product or service concerned. Incorrect information will be misleading to the user and may lead to detrimental outcomes.

Regulatory compliance: Information about the workings of the system must be correct in order to meet external audit and record-keeping requirements.

The following principles are strictly interdependent, and thus I analyse them in a single section:

2. An explanation should be useful; 3. An explanation should be context specific; 4. An explanation should be user specific.

The concept of utility (principle 2) is strictly linked to the context (principle 3) and user specificity (principle 4). The utility of something is measured as a function of the outcomes delivered to a particular user in a specific context. For example, a technical explanation using scientific language and formulae will be of little utility to a lay person. Similarly, an explanation using plain English with a simplified version of the information will not benefit a technical auditor or specialist, but it will be highly relevant to a lay person.

User empowerment: Providing explanations that are relevant to the user and the context of use will be useful to the recipient of the explanation, allowing the user to act on the information provided and make decisions in an empowering way.

Regulatory compliance: The concept of utility in relation to regulatory compliance can be described as a meta-outcome. If the information is presented in a way that is not meaningful, for example, as a disorganised collection of code and training data, it will not be deemed suitable for the purpose it is meant to serve. This principle regarding information clarity and utility is already widely adopted in financial services regulation for retail customers (ESMA, 2014).

5. An explanation should provide pluralism

Providing multiple perspectives on a specific issue, not tied to a single normative outcome, can help empower users to make decisions that are informed by a range of viewpoints. This is a principle that supports user autonomy in decision-making.

User empowerment: By starting with the purposes that are important in the given context, the explanation can be tailored to the user’s needs and preferences, empowering the user to make better decisions. Allowing for a range of normative views can also help users understand different perspectives and make more informed decisions.

Regulatory compliance: The diversity of explanations is not tied to a specific regulatory outcome. However, it aligns with the mandate that the information provided to the user should be clear and not misleading, allowing users to make decisions that best suit their individual circumstances rather than prioritising the interests of the business entities supporting the AI system.

5. Design principles for dialogue systems

I propose a set of design principles and components for a dialogue system focussed on delivering user explanations in a business product context.

1. The user should be able to prompt the dialogue system using natural language, eliminating the requirement to write any code to interact with the system.
2. The system should be able to understand ongoing user requests and associate them with appropriate explanations to be generated and presented to the user.
3. The system should understand the context of the questions and adapt the explanations accordingly.
4. The system should build on previous inputs and clarify/rephrase anything generated by previous prompt.
5. The system should state when it does not have an answer due to a lack of information or because it does not understand the question instead of trying to persuade the user.
6. The system should provide a confidence level of its explanation.
7. The system should be near real time to simulate a human verbal dialogue.

By enabling users to interact with the system using natural language (1), the dialogue system reduces the need for technical expertise and is accessible to a wider range of users, reducing audience comprehension issues. The ability of the system to understand ongoing user requests and associate them with appropriate explanations (2) aids in the delivery of relevant and coherent information, addressing the challenge of audience comprehension, particularly with complex models. By understanding the context of questions and adapting explanations accordingly (3), the dialogue system helps improve the domain knowledge, allowing users to make meaningful inferences in their specific fields of interest. This implies that the user is being educated with new knowledge, or at least guided towards it, which I suggest adds an innovative dimension to XAI. Instead of treating the audience as a static receiver that the XAI system needs to accommodate, the system could actively improve their comprehension and use of explanations.

6. Core components of a dialogue system:

1. Graphical User Interface (GUI): The GUI is the visual interface designed to facilitate user interaction with the dialogue system. It should be user-friendly and intuitive, making it easy for users to enter their input via text and receive the system’s output in a clear and understandable format. The GUI may incorporate elements such as text boxes, buttons or visual cues to guide users through the conversation.
2. Natural Language Understanding (NLU): This component is responsible for processing and interpreting user input in natural language. NLU involves parsing the input text, identifying its structure and extracting relevant information, such as keywords, intents and entities. By doing so, the dialogue system can comprehend the user’s query or request and determine the appropriate response or action.
3. Explanation Mapping: This component is responsible for connecting the user’s input to the appropriate explanation. It may involve querying a knowledge base, database or another AI model to retrieve the necessary information to generate the explanation.
4. Explanation Generation: The explanation generation component is responsible for creating human-readable explanations based on the retrieved information from the mapping component. Using NLE, it produces explanations that are both context specific and comprehensible to the user based on the language of the initial query. The generated explanation is then returned to the user through the GUI.

In real-world applications, these capabilities can be implemented as separate components or as an end-to-end model. For instance, components 2 and 4 could be integrated within a single model, while component 3 might be part of the same model or a separate component. ChatGPT’s plug-ins exemplify this approach, where a text model interacts with another system or model through model prompting (OpenAI, 2023).

In the next blog I will integrate all aspects covered so far and outline how the design principles of a dialogue system based on the explanatory pragmatism framework would apply within a case study of robo- advising product context.