Ethical motivations and a model for constructing Trusted AI.

The widespread application of AI will mean that lots of people will be asked to accept the decisions that AI’s are making. Many people have observed that the evolution of technologies such as genetically modified organisms and nuclear power has been significantly affected by public perception, and it’s of wide interest to ask if this could happen with AI because AI offers a new engine of economic growth and opportunities to address many of the pressing problems that threaten to derail our society. For example, the effects of our aging population could be significantly mitigated if AI could be developed to support elder care (although it will be a huge challenge to do this)

AI technology impacts on at least three specific concerns of public interest :

1.      Safety : the Association for Computing Machinery has published guidelines for accountability [USACM 2017] that focus on the harm that opaque automated decision systems can do when they introduce hidden biases into decision making. AI is seen as having the potential to do unexpected harm, its potential to do harm is hidden and inconsistent with our expectations.

2.      Rights : The General Directive on Data Processing conceptualises explanation as necessary for promoting the rights of citizens. “Advances in technology and the capabilities of big data analytics, artificial intelligence and machine learning have made it easier to create profiles and make automated decisions with the potential to significantly impact individuals’ rights and freedoms.” [GDPR Working Party 2017]. Opaque AI is seen as constraining choice and, potentially deceiving individuals into making choices that aren’t in their best interests. An unethically constructed AI could limit expectations, ambitions and life chances and reduce people’s freedom.

3.      Accountability : AI may be difficult to control in the sense that humans may not be effectively included in decisions in which they have a stake and to which they could make a positive contribution [Cassidy et-al 2018] and in the sense that the point of control for AI may become diffuse in the sense that there may be a difficulty in attributing decisions to specific components developed and owned by discreet entities [Danaher 2017]. If an AI makes poor decisions due to a workflow involving three cloud providers, 20 source code providers and two telecoms networks who is to blame when it fails? Should the cloud provider have anticipated the possibility of a network outage? Or are the “as is warrants” in software licenses sufficient to shift responsibility to the integrator?

For AI to be exploited successfully it must be trusted. Trust is a positive outcome required for users to want to use and interact with AI systems. In the context of the three issues described above for AI to be widely used people need to feel that they won’t be harmed, will retain their freedom and can understand what and who is in control at any given moment; in short they need to trust the systems that they are working with.

Therefore it seems sensible to say that convenient and simple methods of creating trust in these systems that can be accessed by ordinary people are needed. Sometimes policy makers and the media discuss the need for “Explainable AI”. Explanation can be seen as an act made by an agent or entity with the intent to establish trust in its decision making. Agents are motivated to do this because those who can be trusted are accountable and therefore can be given responsibility for decisions, and being allowed to make independent decisions has high utility.

But simplistic ideas of an explaining AI exclude the wider system in which it is embedded. People using AI’s may not wish to reveal their intents by requesting or receiving explanations; they may want to audit and observe rather than inquire and solicit information. Individuals may want to come to a view on the drivers for a decision independently of the AI. And, as noted above, diffusion of control can mean that the notion of a single explaining agent doesn’t make sense.

Trust can be seen as a psychological phenomena [Lewicki 2006], and this is the position that we adopt in this paper – information about motivation is incomplete, leaks from the system and is not shared amongst the participants evenly. Trust and distrust in AI by people will not be wholly rational.

Trust is used as a construct in the reasoning of AI systems to determine the value of information received [for example : Venanzi 2013], this tradition is important to the discussion presented here in so far as it will allow the simulation and design of systems of AI’s and people. In the next section some case studies are presented to ground and illustrate the challenges for creating trusted AI. The examples that we give emphasise the view that trust in AI will be established within wide contexts and with diverse concerns. The case studies are then used to motivate and construct a model of trust for AI systems which is then formalised and examined in a later article.

Establishing Trust : Case studies

Asthma Care.

A widely cited example[1] of the need to provide explanation describes models of healthcare that classify patients for in / out patient care. A variable (asthma) is wrongly found by machines to reduce the risk of mortality, because the correlation between death in hospital and admission with asthma is negative. This is thought to be because asthmatics are rapidly identified and carefully treated; they benefit most from hospitalisation. Essentially, if you are Asthmatic and you develop breathing difficulties then you don't ring someone and ask what's wrong, you get yourself to hospital double quick.

The inability of a neural network to be inspected/to explain its reasoning marks it out as not of interest as a tool in this domain [Caruana et-al 2015]. Importantly no attempt to find a technical means for extracting explanations from the neural network were attempted because it appears that other factors related to the application were identified as making the use of any artificial decision maker impossible. In particular quality of the data bases used to train the classifiers (inpatient data only, better data compromised by ethical challenges) [Cooper et-al 1997 ] and need for a classification system that didn’t make any false negative classification (that is to say the classifier had to be conservative when rejecting a patient for admission) impeded progress[Ambrosino 1995]. The ethical challenge of generating training data is a fundamental blocker, running randomised trials of admissions to observe the mortality rates of patients who previously would have been admitted but then weren’t will never be acceptable.

The preoccupations of the physicians as reported indicates that technical means for forensically establishing trust, or determining that an AI cannot be trusted are not limited to examining the behaviour of decision making AI, but must include an understanding of the context in which it is deployed.  

Embedding bias in legal decision making.

Consider a situation in which I came to believe that I would be discriminated against because of my skin colour or gender. Further, that I believed that I would suffer greater penalties – or at least no redress, if I highlighted that I believed that I was discriminated against, and that I believed that the agent discriminating against me should be removed or altered. In this scenario the person who is being wronged believes that no benefit will accrue to them if they acknowledge the wrong, but may believe that the knowledge of future bias will help them avoid harm – by not consulting the discriminating agent in the future or by trying to deceive it.

The harm caused to individuals by bias in AI is a well founded concern. However, those who perceive that they are being discriminated against are rarely in a position where raising concerns will be in their best interests. They are vulnerable to further harm if they demand explanations, especially if the explanations are held to show no bias by others who share the bias and power of the AI.

To avoid extra harm, people in receipt of decisions that are biased should be able to generate explanations themselves so that they can form beliefs about the likely future actions of decision makers without exposing themselves. Progress is made where the behaviour of the AI agent can be observed in the context of a rich pool of information about the AI process used. The exposure of rich information about the AI process has two effects; firstly it makes it possible for observers to duplicate or approximate the AI systems processes enabling intuitive beliefs to be more reliably and usefully created. Secondly the rich content exposes the origin of bias and makes it possible for hidden bias to be identified without decisions being made. Finally by exposing a rich context adversarial explanation becomes hard – as we will discuss below.

Note, the explanation obtained here is not the one provided by the decision making agent – at least not in societies where racism and sexism is socially unacceptable. The explanation that is created and considered to have utility is manufactured by agents outside of the AI – observers and victims, facilitating this step enables the avoidance of harm.

Social Constraints: Field Technicians & Automated Diagnosis

An automated network diagnosis system can be expected to make some errors. The source of these errors can be faults that have anomalous characteristics due to unusual masking or compensating behaviour in the complex equipment in the network, or simply noise in the inputs; perhaps created by poor data quality. Field technicians using the system are able to physically inspect the network components as well as review telemetry. This allows them to form opinions about the diagnosis, sometimes these are wrong as well, for example proving new piece of network termination equipment and asking for a control parameter reset can temporarily clear a fault, but as the equipment beds in the symptoms will re-emerge.

Unfortunately the errors made are easily perceived as the AI’s, the AI’s mistakes are often transparently obvious to someone physically inspecting the network. The technician’s errors can appear to vanish. Additionally the human’s talk to each other and compare their experiences exaggerating the perception that the AI is inaccurate.

Evidence of the AI’s capability can be created by comparing behaviour to the evaluations of expert panels – demonstrating that the AI outperforms the best engineer, additionally the AI can provide simple explanations of its reasoning.

In this scenario trust in the AI would equate to the technicians using its diagnosis unless they could physically observe a condition unavailable to the system’s sensors. But human trust in the system is undermined by social proof.

Social structures for establishing trust : Pilots and Self Driving Cars

The trust established by human pilots using automated systems is an example of a trusted AI system, self driving cars are not yet trusted, yet it is widely believed that flying aircraft is harder than driving cars.

The human pilot may not be as competent as the automation that she manages, but is made responsible for its management. When a human pilot or driver is asked to explain decisions they will use conceptions of the world as intuitive theories that they believe will be understood by other humans [for example Gopnik & Schulz 2004], but the pilot’s explanation of riding out or getting over the turbulence or a drivers belief that the grip would return if they steered into the skid are distant from realistic physical accounts of the causal sequences that they are being held responsible for.

The self-driving car may be more competent than the driver that is supervising it, but is not trusted for use. There is no trust system established for self driving cars, but there is for automated flight systems. The pilot believes that she is in possession of expert knowledge and insight – possibly won via many hours of study and investigation – that causes her to trust the automated systems of the aircraft. Passengers don’t trust the aircrafts systems, they trust the institution of pilots, having no knowledge of the individual pilot or the systems used themselves.

In the case of self driving cars there is no pilot – there is no proxy to assume the mantel of trust and instead a new mechanism of oversight is needed to create trust in all self driving cars in the minds of all citizens including, sadly, pedestrians.

Trust can be gained individually, through evidence or explanation, or as in the case of aircraft systems through deliberately engineered social institutions.

Discussion

Trust in human systems is created over time by accumulating evidence of behaviour that allows expectations to be established [Gambatta 1988]. Humans will accumulate trust in artificial decision makers by being able to gather evidence over a number of episodes. The quality and amount of evidence will be proportional to the number of episodes required, the trust created will be conditional and qualified in nature [Teacy et-al 2006].

Fundamental constraints on creating trusted AI exist as evidenced by the Asthma example, ethical considerations as well as limitations of the art may bound the ability of practioners to create systems that can be responsibly expected to perform adequately in production. The lesson is that evaluation stretches beyond available data and beyond the data scientist and technical team. These are requirements for transactional mechanisms of trust creation.

As well as generating trust within the transaction between human and AI, our examples show the need to consider the wider system in which the AI is embedded. The mechanism that has brought the AI into production, and its visibility for inspection will be used passively by humans to create trust or distrust. This points to the need for systematic mechanisms of trust creation.

The social context and cultural setting of AI will dictate the level of trust that can be created and must drive the amount that is invested by AI developers in creating trust in their creations.

The examples above illustrate the critical role of social systems in creating trust and distrust. Field technicians distrust because of social proof; passengers trust because of social proof. Social proof is established in a group and cultural context and for AI to be trusted a wide effort is required to establish the conditions for warranted social proof of AI’s trustworthiness to emerge. This points to the need for social mechanisms for trust creation for AI.

In summary there are three mechanisms that create trust between AI and humans. Transactional and episodic; systematic and persistent; social and cultural. Within each of these mechanisms a range of considerations and challenges can currently be identified and further issues and opportunities will emerge as AI develops and interacts with people and society.

Ambrosino, R., Buchanan, B.G., Cooper, G.F. and Fine, M.J., 1995. The use of misclassification costs to learn rule-based decision support models for cost-effective hospital admission strategies. In Proceedings of the Annual Symposium on Computer Application in Medical Care (p. 304). American Medical Informatics Association.

Caruana, R., Lou, Y., Gehrke, J., Koch, P., Sturm, M. and Elhadad, N., 2015, August. Intelligible models for healthcare: Predicting pneumonia risk and hospital 30-day readmission. In Proceedings of the 21th ACM SIGKDD International Conference on Knowledge Discovery and Data Mining (pp. 1721-1730). ACM.

Cooper, G.F., Aliferis, C.F., Ambrosino, R., Aronis, J., Buchanan, B.G., Caruana, R., Fine, M.J., Glymour, C., Gordon, G., Hanusa, B.H. and Janosky, J.E., 1997. An evaluation of machine-learning methods for predicting pneumonia mortality. Artificial intelligence in medicine, 9(2), pp.107-138.

Colombo, M., 2017. Experimental philosophy of explanation rising: The case for a plurality of concepts of explanation. Cognitive science, 41(2), pp.503-517.

Gambetta, D., 2000. Can we trust trust. Trust: Making and breaking cooperative relations, 13, pp.213-237.

GDPR Working Party 2017. Guidelines on Automated individual decision-making and Profiling for the purposes of Regulation 2016/679. ARTICLE 29 DATA PROTECTION WORKING PARTY 17/EN WP251rev.01

Gopnik, A. and Schulz, L., 2004. Mechanisms of theory formation in young children. Trends in cognitive sciences, 8(8), pp.371-377.

Lewicki, R.J., Tomlinson, E.C. and Gillespie, N., 2006. Models of interpersonal trust development: Theoretical approaches, empirical evidence, and future directions. Journal of management, 32(6), pp.991-1022.

Teacy, W.L., Patel, J., Jennings, N.R. and Luck, M., 2006. Travos: Trust and reputation in the context of inaccurate information sources. Autonomous Agents and Multi-Agent Systems, 12(2), pp.183-198.

[1] https://www.nytimes.com/2017/11/21/magazine/can-ai-be-taught-to-explain-itself.html