Automatically Classifying User Engagement for Dynamic Multi-party Human–Robot Interaction

© 2017, The Author(s). A robot agent designed to engage in real-world human–robot joint action must be able to understand the social states of the human users it interacts with in order to behave appropriately. In particular, in a dynamic public space, a crucial task for the robot is to determine the needs and intentions of all of the people in the scene, so that it only interacts with people who intend to interact with it. We address the task of estimating the engagement state of customers for a robot bartender based on the data from audiovisual sensors. We begin with an offline experiment using hidden Markov models, confirming that the sensor data contains the information necessary to estimate user state. We then present two strategies for online state estimation: a rule-based classifier based on observed human behaviour in real bars, and a set of supervised classifiers trained on a labelled corpus. These strategies are compared in offline cross-validation, in an online user study, and through validation against a separate test corpus. These studies show that while the trained classifiers are best in a cross-validation setting, the rule-based classifier performs best with novel data; however, all classifiers also change their estimate too frequently for practical use. To address this issue, we present a final classifier based on Conditional Random Fields: this model has comparable performance on the test data, with increased stability. In summary, though, the rule-based classifier shows competitive performance with the trained classifiers, suggesting that for this task, such a simple model could actually be a preferred option, providing useful online performance while avoiding the implementation and data-scarcity issues involved in using machine learning for this task

A Deep Learning Approach for Multi-View Engagement Estimation of Children in a Child-Robot Joint Attention Task

International audienceIn this work we tackle the problem of child engagement estimation while children freely interact with a robot in a friendly, room-like environment. We propose a deep-based multi-view solution that takes advantage of recent developments in human pose detection. We extract the child's pose from different RGB-D cameras placed regularly in the room, fuse the results and feed them to a deep neural network trained for classifying engagement levels. The deep network contains a recurrent layer, in order to exploit the rich temporal information contained in the pose data. The resulting method outperforms a number of baseline classifiers, and provides a promising tool for better automatic understanding of a child's attitude, interest and attention while cooperating with a robot. The goal is to integrate this model in next generation social robots as an attention monitoring tool during various Child Robot Interaction (CRI) tasks both for Typically Developed (TD) children and children affected by autism (ASD)

Service Robots Rising:How Humanoid Robots Influence Service Experiences and Elicit Compensatory Consumer Responses

Interactions between consumers and humanoid service robots (HSRs; i.e., robots with a human-like morphology such as a face, arms, and legs) will soon be part of routine marketplace experiences. It is unclear, however, whether these humanoid robots (compared with human employees) will trigger positive or negative consequences for consumers and companies. Seven experimental studies reveal that consumers display compensatory responses when they interact with an HSR rather than a human employee (e.g., they favor purchasing status goods, seek social affiliation, and order and eat more food). The authors investigate the underlying process driving these effects, and they find that HSRs elicit greater consumer discomfort (i.e., eeriness and a threat to human identity), which in turn results in the enhancement of compensatory consumption. Moreover, this research identifies boundary conditions of the effects such that the compensatory responses that HSRs elicit are (1) mitigated when consumer-perceived social belongingness is high, (2) attenuated when food is perceived as more healthful, and (3) buffered when the robot is machinized (rather than anthropomorphized)

Automated technologies: how do they enable value co-creation, value co-destruction & customer brand engagement?

As novel automated technologies continue to play an increasingly prominent role in value- based service settings, there is an increased likelihood that the way in which value is co-created and co-destructed will concomitantly change (Paschen et al., 2021; Van Esch et al., 2019). Such technology-induced changes, along with their impacts on customers’ experiences of value co-creation and value co-destruction, are the focus of this research. To this end, this research unveils a more accurate understanding of how novel automated technologies enable value co-creation, value co-destruction and customer brand engagement (CBE). On this basis, the thesis addresses four research objectives: (1) to explore how customers perceive the impact of brands’ automated technology on their experiences of value co-creation and value co-destruction; (2) to examine the variables influencing CBE when customers interact with brands’ automated technology; (3) to examine the CBE outcomes/consequences that occur when customers interact with brands’ automated technology; and (4) to examine customers’ reasons for using brands’ automated technology during service encounters. A mixed-method (qualitative and quantitative) approach is used for this research, consisting of semi-structured interviews and an online survey. Previous value co-creation research has primarily been qualitative or conceptual. For the first stage of data collection, 12 in-depth interviews were carried out. The sample included consumers who had interactions with the chatbot of at least one of the following brands: Asos, Amazon, Skyscanner and Vodafone. These interviews were conducted to explore how customers perceive the impact of brands’ automated technology (chatbots) on their experiences of value co-creation and value co- destruction. The findings indicate that customers’ experiences of value co-creation or value co- destruction are largely dependent on the characteristics of the chatbots they interact with. The chatbot characteristics identified include social presence, information quality, interactivity, personalisation, comprehension and empathy. For the second stage of data collection, an online survey was administered. The sample consisted of 736 consumers divided across Amazon, Vodafone, O2 and H&M. The respondents had prior interactions with these specific brands’/service providers’ chatbots. An online survey was conducted to examine the variables influencing CBE when customers interact with brands’ automated technology, the CBE outcomes/consequences that occur following automated service interactions and customers’ reasons for using these brands’ automated technology. The findings indicate that nine variables influence CBE in chatbot-enabled service settings: social presence, information quality, interactivity, personalisation and empathy, comprehension, utilitarian value, value co-creation and value co-destruction. Moreover, CBE was found to have a significant effect on customers’ continuance intention with the chatbot and brand intention. This research contributes to the value co-creation and CBE literature. Firstly, this research extends the value co-creation literature by exploring experiences of value co-creation and value co-destruction between customers and non-human actors (chatbots) within value-based service networks. Previous value co-creation research falls short in addressing the role nonhuman actors play in the value co-creation and value co-destruction process. Secondly, this research extends the value co-creation literature by revealing six key characteristics of chatbots and the role they play in the value co-creation and/or value co-destruction process. Previous value co-creation does not highlight the key characteristics of technology that facilitate customers' experiences of value co-creation or value co-destruction. Thirdly, this research extends the CBE literature by examining the 12 variables that influence CBE in automated (chatbot-enabled) service settings. Prior CBE research is yet to examine the variables that influence CBE in service settings that are chatbot driven. Fourth, this research extends the CBE literature by examining the impact of value co-creation and value co-destruction on CBE in settings where chatbots facilitate customer-brand interactions. Previous CBE research has not examined the impact value co-creation and value co-destruction have on CBE in chatbot driven service settings. Fifth, this research extends the CBE literature by examining customers’ intention to continue using the chatbot as a consequence/outcome of CBE fostered in chatbot-enabled service settings. Previous CBE research is yet to examine the customers’ continuance intention with the chatbot as an outcome of CBE in chatbot driven service settings.As novel automated technologies continue to play an increasingly prominent role in value- based service settings, there is an increased likelihood that the way in which value is co-created and co-destructed will concomitantly change (Paschen et al., 2021; Van Esch et al., 2019). Such technology-induced changes, along with their impacts on customers’ experiences of value co-creation and value co-destruction, are the focus of this research. To this end, this research unveils a more accurate understanding of how novel automated technologies enable value co-creation, value co-destruction and customer brand engagement (CBE). On this basis, the thesis addresses four research objectives: (1) to explore how customers perceive the impact of brands’ automated technology on their experiences of value co-creation and value co-destruction; (2) to examine the variables influencing CBE when customers interact with brands’ automated technology; (3) to examine the CBE outcomes/consequences that occur when customers interact with brands’ automated technology; and (4) to examine customers’ reasons for using brands’ automated technology during service encounters. A mixed-method (qualitative and quantitative) approach is used for this research, consisting of semi-structured interviews and an online survey. Previous value co-creation research has primarily been qualitative or conceptual. For the first stage of data collection, 12 in-depth interviews were carried out. The sample included consumers who had interactions with the chatbot of at least one of the following brands: Asos, Amazon, Skyscanner and Vodafone. These interviews were conducted to explore how customers perceive the impact of brands’ automated technology (chatbots) on their experiences of value co-creation and value co- destruction. The findings indicate that customers’ experiences of value co-creation or value co- destruction are largely dependent on the characteristics of the chatbots they interact with. The chatbot characteristics identified include social presence, information quality, interactivity, personalisation, comprehension and empathy. For the second stage of data collection, an online survey was administered. The sample consisted of 736 consumers divided across Amazon, Vodafone, O2 and H&M. The respondents had prior interactions with these specific brands’/service providers’ chatbots. An online survey was conducted to examine the variables influencing CBE when customers interact with brands’ automated technology, the CBE outcomes/consequences that occur following automated service interactions and customers’ reasons for using these brands’ automated technology. The findings indicate that nine variables influence CBE in chatbot-enabled service settings: social presence, information quality, interactivity, personalisation and empathy, comprehension, utilitarian value, value co-creation and value co-destruction. Moreover, CBE was found to have a significant effect on customers’ continuance intention with the chatbot and brand intention. This research contributes to the value co-creation and CBE literature. Firstly, this research extends the value co-creation literature by exploring experiences of value co-creation and value co-destruction between customers and non-human actors (chatbots) within value-based service networks. Previous value co-creation research falls short in addressing the role nonhuman actors play in the value co-creation and value co-destruction process. Secondly, this research extends the value co-creation literature by revealing six key characteristics of chatbots and the role they play in the value co-creation and/or value co-destruction process. Previous value co-creation does not highlight the key characteristics of technology that facilitate customers' experiences of value co-creation or value co-destruction. Thirdly, this research extends the CBE literature by examining the 12 variables that influence CBE in automated (chatbot-enabled) service settings. Prior CBE research is yet to examine the variables that influence CBE in service settings that are chatbot driven. Fourth, this research extends the CBE literature by examining the impact of value co-creation and value co-destruction on CBE in settings where chatbots facilitate customer-brand interactions. Previous CBE research has not examined the impact value co-creation and value co-destruction have on CBE in chatbot driven service settings. Fifth, this research extends the CBE literature by examining customers’ intention to continue using the chatbot as a consequence/outcome of CBE fostered in chatbot-enabled service settings. Previous CBE research is yet to examine the customers’ continuance intention with the chatbot as an outcome of CBE in chatbot driven service settings