Partnering People with Deep Learning Systems: Human Cognitive Effects of Explanations

Advances in “deep learning” algorithms have led to intelligent systems that provide automated classifications of unstructured data. Until recently these systems could not provide the reasons behind a classification. This lack of “explainability” has led to resistance in applying these systems in some contexts. An intensive research and development effort to make such systems more transparent and interpretable has proposed and developed multiple types of explanation to address this challenge. Relatively little research has been conducted into how humans process these explanations. Theories and measures from areas of research in social cognition were selected to evaluate attribution of mental processes from intentional systems theory, measures of working memory demands from cognitive load theory, and self-efficacy from social cognition theory. Crowdsourced natural disaster damage assessment of aerial images was employed using a written assessment guideline as the task. The “Wizard of Oz” method was used to generate the damage assessment output of a simulated agent. The output and explanations contained errors consistent with transferring a deep learning system to a new disaster event. A between-subjects experiment was conducted where three types of natural language explanations were manipulated between conditions. Counterfactual explanations increased intrinsic cognitive load and made participants more aware of the challenges of the task. Explanations that described boundary conditions and failure modes (“hedging explanations”) decreased agreement with erroneous agent ratings without a detectable effect on cognitive load. However, these effects were not large enough to counteract decreases in self-efficacy and increases in erroneous agreement as a result of providing a causal explanation. The extraneous cognitive load generated by explanations had the strongest influence on self-efficacy in the task. Presenting all of the explanation types at the same time maximized cognitive load and agreement with erroneous simulated output. Perceived interdependence with the simulated agent was also associated with increases in self-efficacy; however, trust in the agent was not associated with differences in self-efficacy. These findings identify effects related to research areas which have developed methods to design tasks that may increase the effectiveness of explanations

An interdisciplinary concept for human-centered explainable artificial intelligence - Investigating the impact of explainable AI on end-users

Since the 1950s, Artificial Intelligence (AI) applications have captivated people. However, this fascination has always been accompanied by disillusionment about the limitations of this technology. Today, machine learning methods such as Deep Neural Networks (DNN) are successfully used in various tasks. However, these methods also have limitations: Their complexity makes their decisions no longer comprehensible to humans - they are black-boxes. The research branch of Explainable AI (XAI) has addressed this problem by investigating how to make AI decisions comprehensible. This desire is not new. In the 1970s, developers of intrinsic explainable AI approaches, so-called white-boxes (e.g., rule-based systems), were dealing with AI explanations. Nowadays, with the increased use of AI systems in all areas of life, the design of comprehensible systems has become increasingly important. Developing such systems is part of Human-Centred AI (HCAI) research, which integrates human needs and abilities in the design of AI interfaces. For this, an understanding is needed of how humans perceive XAI and how AI explanations influence the interaction between humans and AI. One of the open questions concerns the investigation of XAI for end-users, i.e., people who have no expertise in AI but interact with such systems or are impacted by the system's decisions. This dissertation investigates the impact of different levels of interactive XAI of white- and black-box AI systems on end-users perceptions. Based on an interdisciplinary concept presented in this work, it is examined how the content, type, and interface of explanations of DNN (black box) and rule-based systems (white box) are perceived by end-users. How XAI influences end-users mental models, trust, self-efficacy, cognitive workload, and emotional state regarding the AI system is the centre of the investigation. At the beginning of the dissertation, general concepts regarding AI, explanations, and psychological constructs of mental models, trust, self-efficacy, cognitive load, and emotions are introduced. Subsequently, related work regarding the design and investigation of XAI for users is presented. This serves as a basis for the concept of a Human-Centered Explainable AI (HC-XAI) presented in this dissertation, which combines an XAI design approach with user evaluations. The author pursues an interdisciplinary approach that integrates knowledge from the research areas of (X)AI, Human-Computer Interaction, and Psychology. Based on this interdisciplinary concept, a five-step approach is derived and applied to illustrative surveys and experiments in the empirical part of this dissertation. To illustrate the first two steps, a persona approach for HC-XAI is presented, and based on that, a template for designing personas is provided. To illustrate the usage of the template, three surveys are presented that ask end-users about their attitudes and expectations towards AI and XAI. The personas generated from the survey data indicate that end-users often lack knowledge of XAI and that their perception of it depends on demographic and personality-related characteristics. Steps three to five deal with the design of XAI for concrete applications. For this, different levels of interactive XAI are presented and investigated in experiments with end-users. For this purpose, two rule-based systems (i.e., white-box) and four systems based on DNN (i.e., black-box) are used. These are applied for three purposes: Cooperation & collaboration, education, and medical decision support. Six user studies were conducted for this purpose, which differed in the interactivity of the XAI system used. The results show that end-users trust and mental models of AI depend strongly on the context of use and the design of the explanation itself. For example, explanations that a virtual agent mediates are shown to promote trust. The content and type of explanations are also perceived differently by users. The studies also show that end-users in different application contexts of XAI feel the desire for interactive explanations. The dissertation concludes with a summary of the scientific contribution, points out limitations of the presented work, and gives an outlook on possible future research topics to integrate explanations into everyday AI systems and thus enable the comprehensible handling of AI for all people.Seit den 1950er Jahren haben Anwendungen der Künstlichen Intelligenz (KI) die Menschen in ihren Bann gezogen. Diese Faszination wurde jedoch stets von Ernüchterung über die Grenzen dieser Technologie begleitet. Heute werden Methoden des maschinellen Lernens wie Deep Neural Networks (DNN) erfolgreich für verschiedene Aufgaben eingesetzt. Doch auch diese Methoden haben ihre Grenzen: Durch ihre Komplexität sind ihre Entscheidungen für den Menschen nicht mehr nachvollziehbar - sie sind Black-Boxes. Der Forschungszweig der Erklärbaren KI (engl. XAI) hat sich diesem Problem angenommen und untersucht, wie man KI-Entscheidungen nachvollziehbar machen kann. Dieser Wunsch ist nicht neu. In den 1970er Jahren beschäftigten sich die Entwickler von intrinsisch erklärbaren KI-Ansätzen, so genannten White-Boxes (z. B. regelbasierte Systeme), mit KI-Erklärungen. Heutzutage, mit dem zunehmenden Einsatz von KI-Systemen in allen Lebensbereichen, wird die Gestaltung nachvollziehbarer Systeme immer wichtiger. Die Entwicklung solcher Systeme ist Teil der Menschzentrierten KI (engl. HCAI) Forschung, die menschliche Bedürfnisse und Fähigkeiten in die Gestaltung von KI-Schnittstellen integriert. Dafür ist ein Verständnis darüber erforderlich, wie Menschen XAI wahrnehmen und wie KI-Erklärungen die Interaktion zwischen Mensch und KI beeinflussen. Eine der offenen Fragen betrifft die Untersuchung von XAI für Endnutzer, d.h. Menschen, die keine Expertise in KI haben, aber mit solchen Systemen interagieren oder von deren Entscheidungen betroffen sind. In dieser Dissertation wird untersucht, wie sich verschiedene Stufen interaktiver XAI von White- und Black-Box-KI-Systemen auf die Wahrnehmung der Endnutzer auswirken. Basierend auf einem interdisziplinären Konzept, das in dieser Arbeit vorgestellt wird, wird untersucht, wie der Inhalt, die Art und die Schnittstelle von Erklärungen von DNN (Black-Box) und regelbasierten Systemen (White-Box) von Endnutzern wahrgenommen werden. Wie XAI die mentalen Modelle, das Vertrauen, die Selbstwirksamkeit, die kognitive Belastung und den emotionalen Zustand der Endnutzer in Bezug auf das KI-System beeinflusst, steht im Mittelpunkt der Untersuchung. Zu Beginn der Arbeit werden allgemeine Konzepte zu KI, Erklärungen und psychologische Konstrukte von mentalen Modellen, Vertrauen, Selbstwirksamkeit, kognitiver Belastung und Emotionen vorgestellt. Anschließend werden verwandte Arbeiten bezüglich dem Design und der Untersuchung von XAI für Nutzer präsentiert. Diese dienen als Grundlage für das in dieser Dissertation vorgestellte Konzept einer Menschzentrierten Erklärbaren KI (engl. HC-XAI), das einen XAI-Designansatz mit Nutzerevaluationen kombiniert. Die Autorin verfolgt einen interdisziplinären Ansatz, der Wissen aus den Forschungsbereichen (X)AI, Mensch-Computer-Interaktion und Psychologie integriert. Auf der Grundlage dieses interdisziplinären Konzepts wird ein fünfstufiger Ansatz abgeleitet und im empirischen Teil dieser Arbeit auf exemplarische Umfragen und Experimente und angewendet. Zur Veranschaulichung der ersten beiden Schritte wird ein Persona-Ansatz für HC-XAI vorgestellt und darauf aufbauend eine Vorlage für den Entwurf von Personas bereitgestellt. Um die Verwendung der Vorlage zu veranschaulichen, werden drei Umfragen präsentiert, in denen Endnutzer zu ihren Einstellungen und Erwartungen gegenüber KI und XAI befragt werden. Die aus den Umfragedaten generierten Personas zeigen, dass es den Endnutzern oft an Wissen über XAI mangelt und dass ihre Wahrnehmung dessen von demografischen und persönlichkeitsbezogenen Merkmalen abhängt. Die Schritte drei bis fünf befassen sich mit der Gestaltung von XAI für konkrete Anwendungen. Hierzu werden verschiedene Stufen interaktiver XAI vorgestellt und in Experimenten mit Endanwendern untersucht. Zu diesem Zweck werden zwei regelbasierte Systeme (White-Box) und vier auf DNN basierende Systeme (Black-Box) verwendet. Diese werden für drei Zwecke eingesetzt: Kooperation & Kollaboration, Bildung und medizinische Entscheidungsunterstützung. Hierzu wurden sechs Nutzerstudien durchgeführt, die sich in der Interaktivität des verwendeten XAI-Systems unterschieden. Die Ergebnisse zeigen, dass das Vertrauen und die mentalen Modelle der Endnutzer in KI stark vom Nutzungskontext und der Gestaltung der Erklärung selbst abhängen. Es hat sich beispielsweise gezeigt, dass Erklärungen, die von einem virtuellen Agenten vermittelt werden, das Vertrauen fördern. Auch der Inhalt und die Art der Erklärungen werden von den Nutzern unterschiedlich wahrgenommen. Die Studien zeigen zudem, dass Endnutzer in unterschiedlichen Anwendungskontexten von XAI den Wunsch nach interaktiven Erklärungen verspüren. Die Dissertation schließt mit einer Zusammenfassung des wissenschaftlichen Beitrags, weist auf Grenzen der vorgestellten Arbeit hin und gibt einen Ausblick auf mögliche zukünftige Forschungsthemen, um Erklärungen in alltägliche KI-Systeme zu integrieren und damit den verständlichen Umgang mit KI für alle Menschen zu ermöglichen

Triggering and measuring neural response indicative of inhibition in humans during conversations with virtual humans

The aim of this PhD was to determine if a confrontational virtual human can evoke a response in the prefrontal cortex, indicative of inhibiting an antisocial response. It follows previous studies by Aleksandra Landowska (2018) and Schilbach (2016) demonstrating that a prefrontal cortex response indicative of inhibition can be evoked by a virtual environment. The test scenario was a conversation about Brexit, the United Kingdom leaving the European Union. This was used in three experiments which varied in level of immersivity of the interface and iteratively tweaked methods. A virtual reality head-mounted display (HMD) was adopted in the first experiment, a 50-inch display monitor was adopted in the second experiment, while the third experiment was carried out in an immersive suite. The independent variable in the experiments was the friendliness of the virtual human confederates. fNIRS was used to measure changes in haemoglobin in the medial and dorsolateral prefrontal cortex. Video recordings were taken to capture possible behavioural evidence that may be associated with inhibition. The friendliness of the virtual human was measured using the likeability section of the Godspeed Questionnaire series. This may be the first study to use functional near infrared spectroscopy (fNIRS) to measure response to virtual humans; previous studies have used functional magnetic resonance imaging (fMRI), which provides a less natural experience and is not conducive to non-verbal communication. The results from the first experiment suggest an effect emanating from prior experience with VR and gaming. Consequently, participants were grouped into two, with G1 representing the group with prior VR and gaming experience and G2 representing the group with no VR and gaming experience. Increased activation was found in the dorsolateral prefrontal cortex (DLPFC) during conversation with the confrontational (unfriendly) virtual human confederate for G2, in line with similar studies of emotional regulation. G1, on the other hand, showed increased activation in the medial prefrontal cortex (MPFC) during the conversation with the friendly virtual human confederate. The second experiment which was aimed at validating the outcome of the first experiment also showed an effect emanating from prior experience with VR and gaming. The results suggest increased activation in the MPFC for G1 and increased activation in the MPFC and DLPFC for G2 during the conversation with the friendly virtual human confederate in both groups. The third experiment showed increased activation in the DLPFC during the conversation with the unfriendly virtual human confederate across participants. Furthermore, head-mounted displays complicated data capture with the fNIRS; a problem alleviated by screen or projection-based approaches. Although all the experiments in this research targeted healthy subjects, the outcome may be of interest to health professionals and technology providers interested in mental deficits relating to antisocial behaviours. It also finds potential application in mental health illness such as PTSD and autism where inhibitory responses are impaire

Human-Robot Teams – Paving the Way for the Teams of the Future

Thanks to advances in artificial intelligence and robotics, robots, and especially social robots that can naturally interact with humans, are now found in more and more areas of our lives. At the same time, teams have been the norm in organizations for decades. To bring these two circumstances together, this dissertation addresses the use of social robots together with humans in teams, so-called human-robot teams (HRTs). This work aims to advance knowledge about HRTs and important underlying mechanisms in the establishment of such teams, thereby providing insights in two aspects. First, a structured and universal definition of HRTs is derived from the various perspectives of extant research, and based on a comprehensive literature overview, important characteristics and influencing factors of HRTs as well as research gaps in HRT research are identified. Second, insights into the underlying mechanisms of the establishment of human-robot teams are provided for settings with social robots in two different team roles: team assistant and lower-level (team) manager. For this purpose, this dissertation contains three research studies that cover the currently largely unexplored area of social robots' use in organizational teams at both the employee and lower-level manager levels. The first study (conceptual study) provides a foundation for this dissertation and beyond by developing a structured and universal definition of HRTs. It also structures extant research on HRTs and proposes an agenda for future research on HRTs based on research gaps identified in a comprehensive literature review that includes 194 studies on HRTs. The second and third studies (empirical studies 1 and 2) use empirical online studies to address two of the research gaps identified in the conceptual study. They examine the underlying mechanisms in decisions for social robots in two different team roles: team assistant (empirical study 1) and team manager (empirical study 2). By looking at expectations and experiences of taskwork-/performance-related and teamwork-related/relational features of social robots using polynomial regressions and response surface analyses, these studies rely on expectation disconfirmation theory to provide a detailed investigation of the underlying mechanisms of organizational decisions for social robots. Empirical study 1 thereby shows that for teamwork, positive disconfirmation and high levels of experiences lead to higher acceptance of humanoid and android robotic team assistants, and similar results emerge for a humanoid robot’s taskwork skills. In contrast, for taskwork skills of android team assistants, high levels of positive disconfirmation lead to lower robot acceptance. For robotic lower-level managers, empirical study 2 shows that there are discrepancies in the evaluation of performance-related usefulness and relational attitude. While for usefulness a slight overfulfilment of expectations leads to a positive impact on the readiness to work with, before evaluations decrease with greater overfulfillment, for attitude increasing positive experiences are associated with (decreasing) positive evaluations of readiness. In summary, this dissertation contributes to scientific research on HRTs by advancing the understanding of HRTs, providing a structured and universal definition of HRTs, and suggesting avenues for future research. The systematic investigation of underlying mechanisms for the selection of different types of social robots for different team roles provides a holistic view of this new form of organizational teams. In addition to the research contributions, this thesis also provides practical guidance for the successful establishment of HRTs in organizations