How Was Your Day? evaluating a conversational companion

The “How Was Your Day” (HWYD) Companion is an embodied conversational agent that can discuss work-related issues, entering free-form dialogues that lack any clearly defined tasks and goals. The development of this type of Companion technology requires new models of evaluation. Here, we describe a paradigm and methodology for evaluating the main aspects of such functionality in conjunction with overall system behaviour, with respect to three parameters: functional ability (i.e., does it do the ‘right’ thing), content (i.e., does it respond appropriately to the semantic context), and emotional behaviour (i.e., given the emotional input from the user, does it respond in an emotionally appropriate way). We demonstrate the functionality of our evaluation paradigm as a method for both grading current system performance, and targeting areas for particular performance review. We show correlation between, for example, ASR performance and overall system performance (as is expected in systems of this type) but beyond this, we show where individual utterances or responses, which are indicated as positive or negative, show an immediate response from the user, and demonstrate how our combination evaluation approach highlights issues (both positive and negative) in the Companion system’s interaction behaviou

How to Convey Resilience: Towards A Taxonomy for Conversational Agent Breakdown Recovery Strategies

Conversational agents (CAs) have permeated our everyday lives in the past decade. Yet, the CAs we encounter today are far from perfect as they are still prone to breakdowns. Studies have shown that breakdowns have an immense impact on the user-CA relationship, user satisfaction, and retention. Therefore, it is important to investigate how to react and recover from breakdowns appropriately so that failures do not impair the CA experience lastingly. Examples for recovery strategies are the assumption of the most likely user intent (CA self-repair) or to ask for clarification (user-repair). In this paper, we iteratively develop a taxonomy to classify breakdown recovery strategies based on studies from scholarly literature and experiements with productive CA instances, and identify the current best practices described using our taxonomy. We aim to synthesize, structure and further the knowledge on breakdown handling and to provide a common language to describe recovery strategies

Bodily sensation maps: Exploring a new direction for detecting emotions from user self-reported data

The ability of detecting emotions is essential in different fields such as user experience (UX), affective computing, and psychology. This paper explores the possibility of detecting emotions through user-generated bodily sensation maps (BSMs). The theoretical basis that inspires this work is the proposal by Nummenmaa et al. (2014) of BSMs for 14 emotions. To make it easy for users to create a BSM of how they feel, and convenient for researchers to acquire and classify users’ BSMs, we created a mobile app, called EmoPaint. The app includes an interface for BSM creation, and an automatic classifier that matches the created BSM with the BSMs for the 14 emotions. We conducted a user study aimed at evaluating both components of EmoPaint. First, it shows that the app is easy to use, and is able to classify BSMs consistently with the considered theoretical approach. Second, it shows that using EmoPaint increases accuracy of users’ emotion classification when compared with an adaptation of the well-known method of using the Affect Grid with the Circumplex Model, focused on the same set of 14 emotions of Nummenmaa et al. Overall, these results indicate that the novel approach of using BSMs in the context of automatic emotion detection is promising, and encourage further developments and studies of BSM-based methods