Usability and acceptability assessment of an empathic virtual agent to prevent major depression

In Human-Computer Interaction, the adaptation of the content and the way of how this content is communicated to the users in interactive sessions is a critical issue to promote the acceptability and usability of any computational system. We present a user-adapted interactive platform to identify and provide an early intervention for symptoms of depression and suicide. In particular, we describe the work performed to assess users' system acceptability and usability. An empathic Virtual Agent is the main interface with the user, and it has been designed to generate the appropriate dialogues and emotions during the interactions according to the detected user's specific needs. This personalization is based on a dynamic user model nurtured with clinical, demographical and behavioural information. The evaluation was performed with 60 participants from the university community. The obtained results were promising, allowing the execution of a further clinical trial. The system's usability score was 75.7%, and the score of the user-adapted content and the emotional responses of the Virtual Agent was 70.9%.The work presented in this manuscript has been partially funded by the Conselleria de Sanidad of Generalitat Valenciana, in the research project entitled 'Sistema computacional de ayuda a la prevencion de episodios de depresion y suicidio - PREVENDEP'. We thank the company Faceshift (www.faceshift.com) for providing their software to perform facial motion capture in order to develop the talking head that represent our empathic virtual agent.Bresó Guardado, A.; Martinez-Miranda, J.; Botella Arbona, C.; Baños Rivera, RM.; García Gómez, JM. (2016). Usability and acceptability assessment of an empathic virtual agent to prevent major depression. Expert Systems. 33(4):297-312. doi:10.1111/exsy.12151S29731233

The audio/visual mismatch and the uncanny valley: an investigation using a mismatch in the human realism of facial and vocal aspects of stimuli

Indiana University-Purdue University Indianapolis (IUPUI)Empirical research on the uncanny valley has primarily been concerned with visual elements. The current study is intended to show how manipulating auditory variables of the stimuli affect participant’s ratings. The focus of research is to investigate whether an uncanny valley effect occurs when humans are exposed to stimuli that have an incongruity between auditory and visual aspects. Participants were exposed to sets of stimuli which are both congruent and incongruent in their levels of audio/visual humanness. Explicit measures were used to explore if a mismatch in the human realism of facial and vocal aspects produces an uncanny valley effect and attempt to explain a possible cause of this effect. Results indicate that an uncanny valley effect occurs when humans are exposed to stimuli that have an incongruity between auditory and visual aspects

Producing Acoustic-Prosodic Entrainment in a Robotic Learning Companion to Build Learner Rapport

abstract: With advances in automatic speech recognition, spoken dialogue systems are assuming increasingly social roles. There is a growing need for these systems to be socially responsive, capable of building rapport with users. In human-human interactions, rapport is critical to patient-doctor communication, conflict resolution, educational interactions, and social engagement. Rapport between people promotes successful collaboration, motivation, and task success. Dialogue systems which can build rapport with their user may produce similar effects, personalizing interactions to create better outcomes. This dissertation focuses on how dialogue systems can build rapport utilizing acoustic-prosodic entrainment. Acoustic-prosodic entrainment occurs when individuals adapt their acoustic-prosodic features of speech, such as tone of voice or loudness, to one another over the course of a conversation. Correlated with liking and task success, a dialogue system which entrains may enhance rapport. Entrainment, however, is very challenging to model. People entrain on different features in many ways and how to design entrainment to build rapport is unclear. The first goal of this dissertation is to explore how acoustic-prosodic entrainment can be modeled to build rapport. Towards this goal, this work presents a series of studies comparing, evaluating, and iterating on the design of entrainment, motivated and informed by human-human dialogue. These models of entrainment are implemented in the dialogue system of a robotic learning companion. Learning companions are educational agents that engage students socially to increase motivation and facilitate learning. As a learning companion’s ability to be socially responsive increases, so do vital learning outcomes. A second goal of this dissertation is to explore the effects of entrainment on concrete outcomes such as learning in interactions with robotic learning companions. This dissertation results in contributions both technical and theoretical. Technical contributions include a robust and modular dialogue system capable of producing prosodic entrainment and other socially-responsive behavior. One of the first systems of its kind, the results demonstrate that an entraining, social learning companion can positively build rapport and increase learning. This dissertation provides support for exploring phenomena like entrainment to enhance factors such as rapport and learning and provides a platform with which to explore these phenomena in future work.Dissertation/ThesisDoctoral Dissertation Computer Science 201

A Study of Accomodation of Prosodic and Temporal Features in Spoken Dialogues in View of Speech Technology Applications

Inter-speaker accommodation is a well-known property of human speech and human interaction in general. Broadly it refers to the behavioural patterns of two (or more) interactants and the effect of the (verbal and non-verbal) behaviour of each to that of the other(s). Implementation of thisbehavior in spoken dialogue systems is desirable as an improvement on the naturalness of humanmachine interaction. However, traditional qualitative descriptions of accommodation phenomena do not provide sufficient information for such an implementation. Therefore, a quantitativedescription of inter-speaker accommodation is required. This thesis proposes a methodology of monitoring accommodation during a human or humancomputer dialogue, which utilizes a moving average filter over sequential frames for each speaker. These frames are time-aligned across the speakers, hence the name Time Aligned Moving Average (TAMA). Analysis of spontaneous human dialogue recordings by means of the TAMA methodology reveals ubiquitous accommodation of prosodic features (pitch, intensity and speech rate) across interlocutors, and allows for statistical (time series) modeling of the behaviour, in a way which is meaningful for implementation in spoken dialogue system (SDS) environments.In addition, a novel dialogue representation is proposed that provides an additional point of view to that of TAMA in monitoring accommodation of temporal features (inter-speaker pause length and overlap frequency). This representation is a percentage turn distribution of individual speakercontributions in a dialogue frame which circumvents strict attribution of speaker-turns, by considering both interlocutors as synchronously active. Both TAMA and turn distribution metrics indicate that correlation of average pause length and overlap frequency between speakers can be attributed to accommodation (a debated issue), and point to possible improvements in SDS “turntaking” behaviour. Although the findings of the prosodic and temporal analyses can directly inform SDS implementations, further work is required in order to describe inter-speaker accommodation sufficiently, as well as to develop an adequate testing platform for evaluating the magnitude ofperceived improvement in human-machine interaction. Therefore, this thesis constitutes a first step towards a convincingly useful implementation of accommodation in spoken dialogue systems