Affectively Aligned Cognitive Assistance Using Bayesian Affect Control Theory

Abstract. This paper describes a novel emotionally intelligent cognitive assistant to engage and help older adults with Alzheimer’s disease (AD) to complete activities of daily living (ADL) more independently. Our new system combines two research streams. First, the development of cognitive assistants with artificially intelligent controllers using partially observable Markov decision processes (POMDPs). Second, a model of the dynamics of emotion and identity called Affect Control Theory that arises from the sociological literature on culturally shared sentiments. We present background material on both of these research streams, and then demonstrate a prototype assistive technology that combines the two. We discuss the affective reasoning, the probabilistic and decision-theoretic reasoning, the computer-vision based activity monitoring, the embodied prompting, and we show results in proof-of-concept tests.

Design and Study of Emotions in Virtual Humans for Assistive Technologies

This thesis presents the design and study of emotionally aligned prompts given by virtual humans for persons with cognitive disabilities such as Alzheimer’s disease and related dementias (ADRD). Our goal is to understand how emotions in virtual humans are interpreted by people. Persons with ADRD often need assistance from a care partner to complete activities of daily living such as washing hands, making food, or getting dressed. Artificially intelligent systems have been developed that can assist in such situations by giving automated prompts or cues. Our long term aim is to enhance such systems by delivering automated prompts that are emotionally aligned with individuals in order to help with prompt adherence and with long-term adoption. As a step in this direction, we designed and conducted user study with three different virtual humans, who expressively communicate prompts for a simple handwashing task. The user study was conducted in two phases. The phase I study had all age group people as participants and involved a female virtual human character with facial expressions and body gestures. The phase II study had elderly people as participants and involved both male and female virtual human characters with a focus on their facial expressions. Prompts were evaluated with respect to three basic and important dimensions of emotional experience: evaluation, potency, and activity. The results of the phase I study showed that, people generally agree on the evaluation dimension, whereas in phase II, we had more consensus on evaluation and potency dimensions and were close to consensus on activity. This thesis gives an overview of the hand washing system, and then details the design of the virtual human character and prompts and the results and analysis of the user study for both phases

Technologies to support community-dwelling persons with dementia: a position paper on issues regarding development, usability, effectiveness and cost-effectiveness, deployment, and ethics

Background: With the expected increase in the numbers of persons with dementia, providing timely, adequate, and affordable care and support is challenging. Assistive and health technologies may be a valuable contribution in dementia care, but new challenges may emerge. Objective: The aim of our study was to review the state of the art of technologies for persons with dementia regarding issues on development, usability, effectiveness and cost-effectiveness, deployment, and ethics in 3 fields of application of technologies: (1) support with managing everyday life, (2) support with participating in pleasurable and meaningful activities, and (3) support with dementia health and social care provision. The study also aimed to identify gaps in the evidence and challenges for future research. Methods: Reviews of literature and expert opinions were used in our study. Literature searches were conducted on usability, effectiveness and cost-effectiveness, and ethics using PubMed, Embase, CINAHL, and PsycINFO databases with no time limit. Selection criteria in our selected technology fields were reviews in English for community-dwelling persons with dementia. Regarding deployment issues, searches were done in Health Technology Assessment databases Results: According to our results, persons with dementia want to be included in the development of technologies; there is little research on the usability of assistive technologies; various benefits are reported but are mainly based on low-quality studies; barriers to deployment of technologies in dementia care were identified, and ethical issues were raised by researchers but often not studied. Many challenges remain such as including the target group more often in development, performing more high-quality studies on usability and effectiveness and cost-effectiveness, creating and having access to high-quality datasets on existing technologies to enable adequate deployment of technologies in dementia care, and ensuring that ethical issues are considered an important topic for researchers to include in their evaluation of assistive technologies. Conclusions: Based on these findings, various actions are recommended for development, usability, effectiveness and cost-effectiveness, deployment, and ethics of assistive and health technologies across Europe. These include avoiding replication of technology development that is unhelpful or ineffective and focusing on how technologies succeed in addressing individual needs of persons with dementia. Furthermore, it is suggested to include these recommendations in national and international calls for funding and assistive technology research programs. Finally, practitioners, policy makers, care insurers, and care providers should work together with technology enterprises and researchers to prepare strategies for the implementation of assistive technologies in different care settings. This may help future generations of persons with dementia to utilize available and affordable technologies and, ultimately, to benefit from them

An Assistive Handwashing System with Emotional Intelligence

Whether emotional intelligence (or, affective reasoning) is included can influence the effectiveness of a cognitive assistive system. This thesis presents a novel emotionally intelligent hand-washing assistant that aims to help older adults with Alzheimer's disease complete hand-washing tasks more independently. The thesis reviews previous works in the development of cognitive assistants and in the study of emotional intelligence, and then designs a hand-washing system prototype that combines the two research streams. The difficulties in designing such a system, including probabilistic and decision-theoretic reasoning of the user's functional and emotional states, computer-vision based activity monitoring and affective recognition, and embodied prompting are discussed. Designing the hand-washing system as an integration of independent components, the thesis also discusses coordination between the components. The thesis implements the system in the end, and shows by preliminary tests in laboratory settings that the system implemented (1) runs in real-time from the perspective of the user group, (2) is able to provide a level of functional assistance, (3) produces system prompts that have encoded to some extent the emotional state of its user. The preliminary tests also indicated that a user with emotions with high potency levels (and high activity levels) is more likely to receive system prompts with low potency levels (and high activity levels). This thesis is one of the exploratory works in the area of integrating emotional intelligence with cognitive intelligent assistive systems. It provides a solution to fitting emotional intelligence in a functional system, as well as points out directions for future improvements. The framework designed in this thesis is portable and extensible, and can be generalized to be used in other applications

Emotion-Aware and Human-Like Autonomous Agents

In human-computer interaction (HCI), one of the technological goals is to build human-like artificial agents that can think, decide and behave like humans during the interaction. A prime example is a dialogue system, where the agent should converse fluently and coherently with a user and connect with them emotionally. Humanness and emotion-awareness of interactive artificial agents have been shown to improve user experience and help attain application-specific goals more quickly. However, achieving human-likeness in HCI systems is contingent on addressing several philosophical and scientific challenges. In this thesis, I address two such challenges: replicating the human ability to 1) correctly perceive and adopt emotions, and 2) communicate effectively through language. Several research studies in neuroscience, economics, psychology and sociology show that both language and emotional reasoning are essential to the human cognitive deliberation process. These studies establish that any human-like AI should necessarily be equipped with adequate emotional and linguistic cognizance. To this end, I explore the following research directions. - I study how agents can reason emotionally in various human-interactive settings for decision-making. I use Bayesian Affect Control Theory, a probabilistic model of human-human affective interactions, to build a decision-theoretic reasoning algorithm about affect. This approach is validated on several applications: two-person social dilemma games, an assistive healthcare device, and robot navigation. - I develop several techniques to understand and generate emotions/affect in language. The proposed methods include affect-based feature augmentation of neural conversational models, training regularization using affective objectives, and affectively diverse sequential inference. - I devise an active learning technique that elicits user feedback during a conversation. This enables the agent to learn in real time, and to produce natural and coherent language during the interaction. - I explore incremental domain adaptation in language classification and generation models. The proposed method seeks to replicate the human ability to continually learn from new environments without forgetting old experiences