AI explainability: an assessment of natural language explanations via dialogue systems to promote user empowerment and compliance under the EU AI Act
This year, as part of my Master's dissertation in AI Ethics & Society at 英国剑桥大学 , I delved into the role of explainability within the forthcoming EU AI Act.
Recently, I’ve adapted it to cater to a broader audience. My aim: to assist business stakeholders in understanding and implementing the explainability clauses of the Act, and to contribute to the wider AI ethics community. If you're keen to discuss this further, I'd love to connect.
So, what drove my interest in explainability? My reflections loosely followed this chain of ideas which ultimately resulted into a more precise research question:
Navigating these considerations felt like a journey down a rabbit hole, teetering between academic semantics across philosophy, law, the discipline of Explainable AI (“XAI”) and intuitive notions that “yes, we do need shared frameworks to grasp a technology whilst making it the bedrock for the future of human society as we know it”.
TLDR:
In my research project I reframe explainability from mere regulatory compliance with the EU AI Act to an organising principle that can drive user empowerment and compliance with broader EU regulations. I attempt to tackle the ‘last mile’ of AI explainability: conveying explanations from AI systems to users. Utilising explanatory pragmatism as the philosophical framework, I formulate pragmatic design principles for conveying ‘good explanations’ through dialogue systems using natural language explanations. I use AI-powered robo-advising as a case study to assess the design principles, showcasing their potential benefits and limitations. I explore parallel EU regulatory efforts enhancing robo-advisory transparency. I acknowledge challenges in the implementation of explainability standards and user trust, urging future researchers to empirically test the proposed principles.
This is the first part of a blog series where I set out explainability within the EU regulatory discourse and build a foundation for the upcoming posts.
1. Introduction and motivation
My motivation and target audience
The European Commission initially proposed the Artificial Intelligence Act (henceforth the ‘AI Act’, ‘the Act’, or ‘AIA’) in April 2021 (European Commission, 2021). Its emergence has significantly impacted the field of AI regulation, addressing the growing need for guidelines and frameworks to govern AI-powered technologies. Recently, the European Parliament, Council, and Member states completed their fifth negotiation round. It's anticipated that the Act will be presented for a final plenary vote early next year. This legislation is a complex cross-industry corpus of standards designed to address an ever-evolving technology.
One of the core AI ethics risk areas that the new AI Act focusses on is AI explainability, which also constitutes one of the foundational AI risks. I show that the concept of explainability in the Act is problematic due to a lack of clear definitions and prescriptive implementation standards. In my research project, I aim to address interpretation challenges related to the meaning of explainability requirements for business stakeholders attempting to comply with the new regulation. However, the central argument of my work is that explainability is not a goal in itself related to minimal regulatory compliance with the EU AI Act.
Rather, explainability serves as an organising principle that facilitates the attainment of other normative objectives, which are advantageous for both businesses and society, in line with the value of implementing ethical AI (Vilone & Longo, 2021).
My research project targets the private sector audience, specifically decision-makers who must choose between minimal regulatory compliance as prescribed by the Act and the broader aspirations of creating ethical AI. This decision entails various commercial implications and resource allocations along the spectrum of these choices.
By adopting a proactive approach to explainability, businesses can not only promote regulatory compliance but also foster an organisational culture that emphasises ethical AI practices. This, in turn, can lead to user empowerment, increased trust from customers and stakeholders and better AI alignment with European societal values. Further, by embracing the spirit of the regulation, businesses can anticipate future regulatory developments, including sector-specific regulations acknowledging AI risks (e.g. in financial services) and position themselves as leaders in ethical AI adoption.
Looking beyond high-risk AI systems
The cornerstone of the AI Act is a risk-based taxonomy of AI systems concerning their impact on individuals’ health, safety, or fundamental rights. AI systems that are deemed to pose an unacceptable risk, such as state-driven social scoring and real-time biometric identification systems in public spaces, are prohibited outright. In contrast, AI systems with limited or minimal risk, such as spam filters or video games, can enter the market with minimal new requirements. The majority of the new regulatory requirements in the Act address so-called ‘high-risk systems’, requiring AI system developers and organisations using such AI products to comply with conformance testing, documentation, data quality and governance frameworks that ensure accountability and human oversight.
My unique contribution
In addition to my analysis of the role of explainability in the Act and the case for going beyond minimal regulatory compliance, my unique contribution to the debate is illuminating an under-researched area of AI explainability: the process of transferring the explanation from the system to the end user, with a particular focus on the role of natural language explanations via dialogue systems.
Drawing on the philosophical framework for ‘good explanations’ provided by explanatory pragmatism (Nyrup & Robinson, 2022), I develop a set of design principles to guide:
I translate the academic literature into a language and a pragmatic set of principles targeting my specific audience — business stakeholders, product managers and engineers — aiming to embed explainability requirements within AI product design.
I have also constructed a series of hypothetical user vignettes, aiming to show how this dialogue system would work in practice in an AI-powered robo-advising case study.
My objective is to present exemplary instances of a dialogue system generating explainability results and to suggest opportunities for future research to experimentally validate such a system.
2. Explainability in the context of the EU AI Act
2.1 Introduction
The EU AI Act is a central component of the EU’s digital single-market strategy, aiming to facilitate the efficient functioning of the internal market by establishing common regulations for the development, deployment and adoption of AI-powered products and services (European Commission, 2021). The proposed legislation adopts a risk-based approach, classifying AI systems as i) unacceptable risk, ii) high risk and iii) low or minimal risk. Title II of the regulation identifies AI applications with unacceptable risk levels that contravene EU values and are consequently prohibited, such as those that manipulate individuals through subliminal techniques or exploit vulnerable populations.
Title III constitutes the majority of the regulation, outlining prescriptive rules for high-risk AI systems, which are defined as systems that pose significant threats to the health, safety or fundamental rights of natural persons. Annex III contains a list of high-risk applications, subject to revision by the Commission due to rapid technological advancements. Requirements for high-risk AI systems encompass data governance, record-keeping and documentation, user transparency, human oversight and technical robustness and security. The rules do not stipulate specific technical requirements, allowing AI system providers flexibility in implementing the regulations according to evolving engineering and scientific knowledge.
My research focusses on explainability, an aspect of the Act that involves not only a lack of prescriptive technical rules but also general definitional challenges.
In fact, it is one of the biggest challenges in AI ethics and achieving human-aligned AI (Miller, 2019; Liao, Gruen, & Miller, 2020). Despite not providing explicit AI explainability requirements, the Act includes provisions, specifically Articles 13 and 14, that imply the need for a degree of AI explainability for high-risk systems to which the rules apply. In agreement with a mainstream idea in explainable AI literature (Saeed & Omlin, 2023), I note that AI explainability is not a goal in itself but serves as an organising principle, which in turn enables operational conditions for other normative goals.
I argue that explainability serves as a facilitator in realising the normative objectives delineated in Articles 13 and 14, specifically user empowerment and regulatory compliance. To accomplish the normative goals established by the Act, it is essential to develop a practical model of AI explainability.
Before delving into the Act, I briefly examine the main precedent in recent EU legal history concerning the topic of explainability of automated systems. The ‘right to explanation’ is a key if controversial topic of the GDPR (Casey et al., 2019; GDPR, 2016). Article 22 of the GDPR states that data subjects, individuals whose personal data is collected or processed by an organisation within the EU, ‘shall have the right not to be subject to a decision based solely on automated processing, including profiling, which produces legal effects concerning him or her or similarly significantly affects him or her’. Recital 71 of the GDPR states the following: ‘[such] processing should be subject to suitable safeguards, which should include specific information to the data subject and the right to obtain human intervention, to express his or her point of view, to obtain an explanation of the decision reached after such assessment and to challenge the decision’.
I recognise that the right to explanation with respect to automated processing in the GDPR includes a broad definition of automation (i.e. any method that is based solely on automated processing without human intervention and does not relate solely to AI). However, GDPR represents a previous attempt to define requirements aimed at explaining the proceedings of automated systems, including AI, which help set the normative context for the EU AI Act itself.
This right to explanation enables data subjects to request information from AI service providers with regard to the automated decision output. As mentioned in Recital 71, data subjects have the right to ‘obtain an explanation of the decision reached after such assessment’.
Fulfilling the ‘right to explanation’ requirement is complex for several reasons, including i) inherent technical challenges related to mechanistic interpretability; ii) the individual’s ability to understand the explanation provided; iii) the IP and confidentiality of the algorithm; and iv) security vulnerability. It is important to note that these challenges pertain not only to the issue of explanation within the context of the GDPR but are common problems related to AI explainability as a field (Hacker & Passoth, 2022).
Additionally, data controllers are mandated to provide ‘meaningful information about the logic involved’ in automated decision-making processes, as specified in Articles 13(2)(f), 14(2)(g) and 15(1)(h). This requirement is deemed ‘right-enabling,’ as affected users would not be able to form opinions or contest automated decisions without a meaningful understanding of the decision-making logic, as outlined in Article 22. In addition to the information requirements related to automated processing set forth in Articles 13 and 14, Recital 60 states that companies should provide any supplementary information necessary to ensure fair and transparent processing of personal data, considering the specific circumstances of personal data processing.
Examining the placement of explanation and information-sharing provisions within the GDPR, I argue that such requirements serve both user-enabling and user-protective functions.
领英推荐
By providing the user with appropriate information, including an explanation of the operation of automated decision-making and its impact on the individual, the GDPR aims to equip users with actionable knowledge about the specific automated system. This empowers users to take appropriate action to ensure that the workings of the system are compliant with the regulation. User empowerment and compliance are fundamental principles that will be further discussed in the context of the EU AI Act and the role of explanations therein.
Following five years of GDPR implementation, relying solely on the GDPR to help define what explainability means for contemporary AI systems presents challenges. In fact, there is an ongoing debate on the ‘right of explanation’ delineated in the regulation, and there have been no subsequent updates in the regulation to establish more explicit standards.
The GDPR does not specify whether explanations should be ex ante or ex post, nor does it define the scope and format of the explanation (e.g. ranging from complex mathematical explanations to natural language) (Sovrano et al., 2021). Nevertheless, it is crucial to recognise that the GDPR has established a trajectory in European jurisprudence concerning the ‘right to explanation’ and the role of explainability discourse in shaping the development and distribution of AI systems.
2.2 EU AI Act: meanings of explainability
?Considering the absence of clear definitional boundaries for the term ‘explainability’ within the EU AI Act regulatory text, I try to define it by considering the context established by the GDPR as well as the broader EU normative discourse.
The Act builds on the themes of user explanation and information-sharing requirements introduced by the GDPR. Like its regulatory precedent, the Act does not provide specific definitions or requirements for developing explainability frameworks. Although the Act does not contain an explicit mandate for ‘AI explainability’ requirements, Recital 38 alludes to the concept of ‘explainable AI,’ cautioning against potential negative impacts on individuals’ fundamental rights and power imbalances if AI systems lack sufficient transparency, explainability and documentation.
Notably, the AI Act incorporates two essential articles that imply a degree of explanation for AI systems in use: Article 13, ‘Transparency and provision of information to users,’ and Article 14, ‘Human oversight’.
?Article 13 delineates three sub-articles with key requirements:
The notion of transparency is not value-neutral and has been subject to critical scrutiny (Ananny & Crawford, 2018). The term lacks a singular definition, and in the literature on explainability, ‘transparency’ and ‘explainability’ are often employed together but not synonymously (Grady, 2022).
For example, OECD Principle 1.3 on ‘Transparency and explainability’ puts forward ‘transparency and responsible disclosure around AI systems to ensure that people understand when they are engaging with them and can challenge outcomes’ (OECD, 2022).
?I apply Hayes’s (2020) notion of transparency to interpret its meaning in the AI Act.
Transparency is a condition that is conducive to knowledge acquisition about X, a phenomenon, an object or in our case an AI algorithm. Such a condition is defined by several properties:
While availability and accessibility are necessary for transparency, they are largely insufficient for empowering agents with information conducive to knowledge creation and response.
Transparency is morally valuable, as it promotes knowledge creation and dissemination. The degree of transparency can exist on a spectrum, ranging from fully transparent to completely opaque, based on the presence and depth of the transparent characteristics mentioned above.
I argue that, in general terms, transparency serves as an enabling principle for explainability.
Without availability and access to information, generating and sharing explanations about AI systems would not be possible. However, transparency alone is not sufficient for understanding and knowledge acquisition.
Knowledge is a pre-requisite for user empowerment: if something is known, it can be acted upon.
Article 13 of the Act delineates transparency as a pre-emptive measure. Thus, providers of high-risk AI systems are required to offer designated information to the end user prior to the use of the product or service.
Consequently, Article 13 implicitly stipulates that transparency is a pre-requisite for facilitating the knowledge about high-risk AI systems.
The provisions delineated in Article 13 endeavour to empower users to interpret and comprehend the outputs of the AI model. As such, as long as the providers of high-risk AI systems fulfil the information requisites specified in Article 13, they can support the user’s ability to understand the AI model outputs (Sovrano et al., 2022).
In the next blog, I will outline what I identify as the primary normative objectives of the explainability discourse within the Act and broader EU regulatory landscape: user empowerment and regulatory compliance.
Sources:
Ananny, M., & Crawford, K. (2018). Seeing without knowing: Limitations of the transparency ideal and its application to algorithmic accountability. New Media & Society, 20(3), 973–989. https://journals.sagepub.com/doi/10.1177/1461444816676645
Casey, B., Farhangi, A., & Vogl, R. (2019). Rethinking explainable machines: The GDPR’s right to explanation debate and the rise of algorithmic audits in enterprise. Berkeley Technology Law Journal, 34(1), 143. https://doi.org/10.15779/Z38M32N986
European Commission. (2021). Proposal for a regulation of the European Parliament and the Council laying down harmonised rules on artificial intelligence and amending certain Union legislative acts. https://eur-lex.europa.eu/legal-content/EN/TXT/?uri=celex%3A52021PC0206
Hayes, P. (2020). An ethical intuitionist account of transparency of algorithms and its gradation. Business Research, 13(3), 849-874 https://ideas.repec.org/a/spr/busres/v13y2020i3d10.1007_s40685-020-00138- 6.html#refs
Hacker, P., & Passoth, J. H. (2022). Varieties of AI explanations under the law: From the GDPR to the AIA, and beyond. In A. Holzinger, R. Goebel, R. Fong, T. Moon, K. R. Mu?ller, & W. Samek (Eds.), xxAI – Beyond explainable AI. xxAI 2020. Lecture Notes in Computer Science (Vol. 13200). Springer. https://doi.org/10.1007/978-3-031-04083- 2_17
Liao, Q. V., Gruen, D., & Miller, S. (2020). Questioning the AI: Informing design practices for explainable AI user experiences. In Proceedings of the 2020 CHI Conference on Human Factors in Computing Systems (pp. 1–15).
Miller, T. (2019). Explanation in artificial intelligence: Insights from the social sciences. https://arxiv.org/pdf/1706.07269.pdf
Nyrup, R., & Robinson, D. (2022). Explanatory pragmatism: A context-sensitive framework for explainable medical AI. Ethics and Information Technology, 24 (13). https://doi.org/10.1007/s10676-022-09632-3
OECD. (2022). Recommendation of the Council on Artificial Intelligence. https://legalinstruments.oecd.org/en/instruments/oecd-legal-0449#dates
Saeed W., & Omlin C. (2023). Explainable AI (XAI): A systematic meta-survey of current challenges and future opportunities. Knowledge-Based Systems, 263, 110273 https://www.sciencedirect.com/science/article/pii/S0950705123000230#section
Sovrano, F., Vitali, F., & Palmirani, M. (2021). Making things explainable vs explaining: Requirements and challenges under the GDPR. https://arxiv.org/abs/2110.00758
Vilone, G., & Longo, L. (2021), Notions of explainability and evaluation approaches for explainable artificial intelligence. Information Fusion, 76, 89–106, https://doi.org/10.1016/j.inffus.2021.05.009
AI Governance Lead, Swift | MSc Artificial Intelligence
7 个月Very interesting, I would love to read the whole paper and you may also like this one https://elizabethseger.com/art/
Global Tech Lead - Quantum Techs | Responsible AI | Diversity | Career coaching
1 年Excellent article, I like in particular the connection with #GDPR to help understand where the EU is coming from --and what it aims to achieve-- when requiring #explainability. The focus on empowering citizens ("users") so that they have an ability to assess whether an #AI system may be flawed and, thus, bring a case is in my view a distinctive feature of the EU approach to regulating this technology through the #AIAct. Will look forward to the next article in the series, Anna Nicolis!