Automatic Replication of Teleoperator Head Movements and Facial Expressions on a Humanoid Robot

Robotic telepresence aims to create a physical presence for a remotely located human (teleoperator) by reproducing their verbal and nonverbal behaviours (e.g. speech, gestures, facial expressions) on a robotic platform. In this work, we propose a novel teleoperation system that combines the replication of facial expressions of emotions (neutral, disgust, happiness, and surprise) and head movements on the fly on the humanoid robot Nao. Robots' expression of emotions is constrained by their physical and behavioural capabilities. As the Nao robot has a static face, we use the LEDs located around its eyes to reproduce the teleoperator expressions of emotions. Using a web camera, we computationally detect the facial action units and measure the head pose of the operator. The emotion to be replicated is inferred from the detected action units by a neural network. Simultaneously, the measured head motion is smoothed and bounded to the robot's physical limits by applying a constrained-state Kalman filter. In order to evaluate the proposed system, we conducted a user study by asking 28 participants to use the replication system by displaying facial expressions and head movements while being recorded by a web camera. Subsequently, 18 external observers viewed the recorded clips via an online survey and assessed the quality of the robot's replication of the participants' behaviours. Our results show that the proposed teleoperation system can successfully communicate emotions and head movements, resulting in a high agreement among the external observers (ICC_E = 0.91, ICC_HP = 0.72).This work was funded by the EPSRC under its IDEAS Factory Sandpits call on Digital Personhood (Grant Ref· EP/L00416X/1)

Automatic analysis of facilitated taste-liking

This paper focuses on: (i) Automatic recognition of taste-liking from facial videos by comparatively training and evaluating models with engineered features and state-of-the-art deep learning architectures, and (ii) analysing the classification results along the aspects of facilitator type, and the gender, ethnicity, and personality of the participants. To this aim, a new beverage tasting dataset acquired under different conditions (human vs. robot facilitator and priming vs. non-priming facilitation) is utilised. The experimental results show that: (i) The deep spatiotemporal architectures provide better classification results than the engineered feature models; (ii) the classification results for all three classes of liking, neutral and disliking reach F1 scores in the range of 71%-91%; (iii) the personality-aware network that fuses participants’ personality information with that of facial reaction features provides improved classification performance; and (iv) classification results vary across participant gender, but not across facilitator type and participant ethnicity.EPSR

Eigenface algorithm-based facial expression recognition in conversations - an experimental study

Recognising facial expressions is important in many fields such as computer-human interface. Though different approaches have been widely used in facial expression recognition systems, there are still many problems in practice to achieve the best implementation outcomes. Most systems are tested via the lab-based facial expressions, which may be unnatural. Particularly many systems have problems when they are used for recognising the facial expressions being used during conversation. This paper mainly conducts an experi-mental study on Eigenface algorithm-based facial expression recognition. It primarily aims to investigate the performance of both lab-based facial expressions and facial expressions used during conversation. The experiment also aims to probe the problems arising from the recognition of facial expression in conversations. The study is carried out using both the author’s facial expression as the basis for the lab-based expressions and the facial expression from one elderly person during conversation. The experiment showed a good result in lab-based facial expressions, but there are some issues observed when using the case of facial expressions obtained in conversation. By analysing the experimental results, future research focus has been highlighted as the investigation of how to recognise special emotions such as a wry smile and how to deal with the interferences in the lower part of face when speaking

A survey of the state-of-the-art techniques for cognitive impairment detection in the elderly

With a growing number of elderly people in the UK, more and more of them suffer from various kinds of cognitive impairment. Cognitive impairment can be divided into different stages such as mild cognitive impairment (MCI) and severe cognitive impairment like dementia. Its early detection can be of great importance. However, it is challenging to detect cognitive impairment in the early stage with high accuracy and low cost, when most of the symptoms may not be fully expressed. This survey paper mainly reviews the state of the art techniques for the early detection of cognitive impairment and compares their advantages and weaknesses. In order to build an effective and low-cost automatic system for detecting and monitoring the cognitive impairment for a wide range of elderly people, the applications of computer vision techniques for the early detection of cognitive impairment by monitoring facial expressions, body movements and eye movements are highlighted in this paper. In additional to technique review, the main research challenges for the early detection of cognitive impairment with high accuracy and low cost are analysed in depth. Through carefully comparing and contrasting the currently popular techniques for their advantages and weaknesses, some important research directions are particularly pointed out and highlighted from the viewpoints of the authors alone

Comparing methods for assessment of facial dynamics in patients with major neurocognitive disorders

International audienceAssessing facial dynamics in patients with major neurocogni-tive disorders and specifically with Alzheimers disease (AD) has shown to be highly challenging. Classically such assessment is performed by clinical staff, evaluating verbal and non-verbal language of AD-patients, since they have lost a substantial amount of their cognitive capacity, and hence communication ability. In addition, patients need to communicate important messages, such as discomfort or pain. Automated methods would support the current healthcare system by allowing for telemedicine, i.e., lesser costly and logistically inconvenient examination. In this work we compare methods for assessing facial dynamics such as talking, singing, neutral and smiling in AD-patients, captured during music mnemotherapy sessions. Specifically, we compare 3D Con-vNets, Very Deep Neural Network based Two-Stream ConvNets, as well as Improved Dense Trajectories. We have adapted these methods from prominent action recognition methods and our promising results suggest that the methods generalize well to the context of facial dynamics. The Two-Stream ConvNets in combination with ResNet-152 obtains the best performance on our dataset, capturing well even minor facial dynamics and has thus sparked high interest in the medical community

Computational Analysis of Affect, Personality and Engagement for Human-Robot Interactions

This chapter focuses on recent advances in social robots that are capable of sensing their users, and support their users through social interactions, with the ultimate goal of fostering their cognitive and socio-emotional wellbeing. Designing social robots with socio-emotional skills is a challenging research topic still in its infancy. These skills are important for robots to be able to provide physical and social support to human users, and to engage in and sustain long-term interactions with them in a variety of application domains that require human–robot interaction, including healthcare, education, entertainment, manufacturing, and many others. The availability of commercial robotic platforms and developments in collaborative academic research provide us with a positive outlook; however, the capabilities of current social robots are quite limited. The main challenge is understanding the underlying mechanisms of humans in responding to and interacting with real life situations, and how to model these mechanisms for the embodiment of naturalistic, human-inspired behavior in robots. Addressing this challenge successfully requires an understanding of the essential components of social interaction, including nonverbal behavioral cues such as interpersonal distance, body position, body posture, arm and hand gestures, head and facial gestures, gaze, silences, vocal outbursts, and their dynamics. To create truly intelligent social robots, these nonverbal cues need to be interpreted to form an understanding of the higher level phenomena including first-impression formation, social roles, interpersonal relationships, focus of attention, synchrony, affective states, emotions, personality, and engagement, and in turn defining optimal protocols and behaviors to express these phenomena through robotic platforms in an appropriate and timely manner. This chapter sets out to explore the automatic analysis of social phenomena that are commonly studied in the fields of affective computing and social signal processing, together with an overview of recent vision-based approaches used by social robots. The chapter then describes two case studies to demonstrate how emotions and personality, two key phenomena for enabling effective and engaging interactions with robots, can be automatically predicted from visual cues during human–robot interactions. The chapter concludes by summarizing the open problems in the field and discussing potential future directions

Investigating Bias and Fairness in Facial Expression Recognition.

Recognition of expressions of emotions and a ect from facial images is a well-studied research problem in the elds of a ective computing and computer vision with a large number of datasets available containing facial images and corresponding expression labels. However, virtually none of these datasets have been acquired with consideration of fair distribution across the human population. Therefore, in this work, we undertake a systematic investigation of bias and fairness in facial expression recognition by comparing three di erent approaches, namely a baseline, an attribute-aware and a disentangled approach, on two wellknown datasets, RAF-DB and CelebA. Our results indicate that: (i) data augmentation improves the accuracy of the baseline model, but this alone is unable to mitigate the bias e ect; (ii) both the attribute-aware and the disentangled approaches equipped with data augmentation perform better than the baseline approach in terms of accuracy and fairness; (iii) the disentangled approach is the best for mitigating demographic bias; and (iv) the bias mitigation strategies are more suitable in the existence of uneven attribute distribution or imbalanced number of subgroup data.European Union H202