Emotion recognition in the human face and voice

This thesis was submitted for the award of Doctor of Philosophy and was awarded by Brunel University LondonAt a perceptual level, faces and voices consist of very different sensory inputs and therefore, information processing from one modality can be independent of information processing from another modality (Adolphs & Tranel, 1999). However, there may also be a shared neural emotion network that processes stimuli independent of modality (Peelen, Atkinson, & Vuilleumier, 2010) or emotions may be processed on a more abstract cognitive level, based on meaning rather than on perceptual signals. This thesis therefore aimed to examine emotion recognition across two separate modalities in a within-subject design, including a cognitive Chapter 1 with 45 British adults, a developmental Chapter 2 with 54 British children as well as a cross-cultural Chapter 3 with 98 German and British children, and 78 German and British adults. Intensity ratings as well as choice reaction times and correlations of confusion analyses of emotions across modalities were analysed throughout. Further, an ERP Chapter investigated the time-course of emotion recognition across two modalities. Highly correlated rating profiles of emotions in faces and voices were found which suggests a similarity in emotion recognition across modalities. Emotion recognition in primary-school children improved with age for both modalities although young children relied mainly on faces. British as well as German participants showed comparable patterns for rating basic emotions, but subtle differences were also noted and Germans perceived emotions as less intense than British. Overall, behavioural results reported in the present thesis are consistent with the idea of a general, more abstract level of emotion processing which may act independently of modality. This could be based, for example, on a shared emotion brain network or some more general, higher-level cognitive processes which are activated across a range of modalities. Although emotion recognition abilities are already evident during childhood, this thesis argued for a contribution of ‘nurture’ to emotion mechanisms as recognition was influenced by external factors such as development and culture.Economics and Social Research Counci

What influences the neural correlates of social cognition? Studies from a microscopic and macroscopic perspective

Social cognition, as of the fundament of social interaction, is central to our daily social life. Although the past two decades have witnessed a huge increase in academic interest in social cognition, knowledge of the neural correlates of social cognition is still limited. With a growing number of studies investigating social cognition with a neuroscientific approach, a well-framed structure based on systematic perspectives to understand social cognition is urgently needed. The present dissertation attempted to investigate social cognition from two domains based on the idea what influences social cognition, the so-called microscopic perspective on the individual and the macroscopic perspective on the culture. From the microscopic perspective, the effects of schizophrenia risk factors (including schizotypy and rs1344706 SNP) on neural correlates of social cognition were investigated in a healthy German sample. The results show associations between schizotypy, as well as the risk allele of the rs1344706 SNP and posterior superior temporal sulcus (pSTS) activation in response to neutral facial stimuli, suggesting right pSTS dysfunction in response to neural social stimuli might present an endophenotype for schizophrenia. Furthermore, these findings give evidence on the microscopic perspective proposed above that neural correlates of social cognition can be influenced by risk factors for mental illnesses in healthy participants. Regarding the macroscopic perspective, the cultural effects on neural responses to different facets of social categorization were investigated with participants from different ethnicities. During the ethnicitybased categorization, the Chinese group showed higher ventral medial prefrontal cortex (MFC) activation for categorizing in-ethnicity versus out-ethnicity faces than the German group, even inethnicity bias was not observed in the Chinese group on the behavioral level. Since ventral MFC is welldocumented to be associated with representing the preference of stimuli even in an unconscious or automatic fashion, the increased ventral MFC activation in the Chinese group may indicate that they present higher in-ethnicity preference than the German group. Further, increased dorsal MFC activation in response to in-team versus out-team faces was found in both ethnic groups during team-basedcategorization, inferring that the dorsal MFC might be a generalized neural code for encoding in-team members across ethnicities. In addition, by comparing the contrasts of in-team versus in-ethnicity and out-team versus out-ethnicity, the results suggest that ethnicity-based and team-based categorization probably presenting different dimensions of social categorization (such as perceptual- and knowledgebased categorization). To summarize, the present dissertation aimed to advance the understanding of social cognition from microscopic and macroscopic perspectives. Such an approach might transfer to the clinical and psychotherapeutic field for developing more generalized interventions and treatments across ethnicities to prevent people from mental disorders or to optimize interventions for people with mental illnesses

Emotion-aware voice interfaces based on speech signal processing

Voice interfaces (VIs) will become increasingly widespread in current daily lives as AI techniques progress. VIs can be incorporated into smart devices like smartphones, as well as integrated into autos, home automation systems, computer operating systems, and home appliances, among other things. Current speech interfaces, however, are unaware of users’ emotional states and hence cannot support real communication. To overcome these limitations, it is necessary to implement emotional awareness in future VIs. This thesis focuses on how speech signal processing (SSP) and speech emotion recognition (SER) can enable VIs to gain emotional awareness. Following an explanation of what emotion is and how neural networks are implemented, this thesis presents the results of several user studies and surveys. Emotions are complicated, and they are typically characterized using category and dimensional models. They can be expressed verbally or nonverbally. Although existing voice interfaces are unaware of users’ emotional states and cannot support natural conversations, it is possible to perceive users’ emotions by speech based on SSP in future VIs. One section of this thesis, based on SSP, investigates mental restorative effects on humans and their measures from speech signals. SSP is less intrusive and more accessible than traditional measures such as attention scales or response tests, and it can provide a reliable assessment for attention and mental restoration. SSP can be implemented into future VIs and utilized in future HCI user research. The thesis then moves on to present a novel attention neural network based on sparse correlation features. The detection accuracy of emotions in the continuous speech was demonstrated in a user study utilizing recordings from a real classroom. In this section, a promising result will be shown. In SER research, it is unknown if existing emotion detection methods detect acted emotions or the genuine emotion of the speaker. Another section of this thesis is concerned with humans’ ability to act on their emotions. In a user study, participants were instructed to imitate five fundamental emotions. The results revealed that they struggled with this task; nevertheless, certain emotions were easier to replicate than others. A further study concern is how VIs should respond to users’ emotions if SER techniques are implemented in VIs and can recognize users’ emotions. The thesis includes research on ways for dealing with the emotions of users. In a user study, users were instructed to make sad, angry, and terrified VI avatars happy and were asked if they would like to be treated the same way if the situation were reversed. According to the results, the majority of participants tended to respond to these unpleasant emotions with neutral emotion, but there is a difference among genders in emotion selection. For a human-centered design approach, it is important to understand what the users’ preferences for future VIs are. In three distinct cultures, a questionnaire-based survey on users’ attitudes and preferences for emotion-aware VIs was conducted. It was discovered that there are almost no gender differences. Cluster analysis found that there are three fundamental user types that exist in all cultures: Enthusiasts, Pragmatists, and Sceptics. As a result, future VI development should consider diverse sorts of consumers. In conclusion, future VIs systems should be designed for various sorts of users as well as be able to detect the users’ disguised or actual emotions using SER and SSP technologies. Furthermore, many other applications, such as restorative effects assessments, can be included in the VIs system

Tracking the temporal dynamics of cultural perceptual diversity in visual information processing

Human perception and cognition processing are not universal. Culture and experience markedly modulate visual information sampling in humans. Cross-cultural studies comparing between Western Caucasians (WCs) and East Asians (EAs) have shown cultural differences in behaviour and neural activities in regarding to perception and cognition. Particularly, a number of studies suggest a local perceptual bias for Westerners (WCs) and a global bias for Easterners (EAs): WCs perceive most efficiently the salient information in the focal object; as a contrast EAs are biased toward the information in the background. Such visual processing bias has been observed in a wide range of tasks and stimuli. However, the underlying neural mechanisms of such perceptual tunings, especially the temporal dynamic of different information coding, have yet to be clarified. Here, in the first two experiments I focus on the perceptual function of the diverse eye movement strategies between WCs and EAs. Human observers engage in different eye movement strategies to gather facial information: WCs preferentially fixate on the eyes and mouth, whereas EAs allocate their gaze relatively more on the center of the face. By employing a fixational eye movement paradigm in Study 1 and electroencephalographic (EEG) recording in study 2, the results confirm the cultural differences in spatial-frequency information tuning and suggest the different perceptual functions of preferred eye movement pattern as a function of culture. The third study makes use of EEG adaptation and hierarchical visual stimulus to access the cultural tuning in global/local processing. Culture diversity driven by selective attention is revealed in the early sensory stage. The results here together showed the temporal dynamic of cultural perceptual diversity. Cultural distinctions in the early time course are driven by selective attention to global information in EAs, whereas late effects are modulated by detail processing of local information in WC observers

Differential effects of script system acquisition and social immersion experience on face perception

Informelle Berichte zeigten größere Amplituden der durch Gesichter ausgelösten N170 Komponente im Ereigniskorrelierten Potenzial (EKP) bei asiatischen als bei kaukasischen Probanden. Als mögliche Ursache vermutete ich unterschiedliche Erfahrungen mit logographischen bzw. alphabetischen Schriftsystemen (Schriftsystem-Hypothese) oder die verstärkte Exposition mit unbekannten Gesichtern während der Immersion in eine neue soziale oder ethnische Umgebung (soziale Immersions-Hypothese). Zur Überprüfung dieser Hypothesen führte ich zwei kulturvergleichende Studien mit Erwachsenen bzw. Kindern durch. In Studie 1 untersuchte ich einheimische Chinesen und nicht-chinesische Auswärtige in Hongkong und deutsche Einheimische und chinesische Auswärtige in Berlin. Die Auswärtigen an beiden Orten zeigten größere N170 Amplituden auf Gesichter als die Einheimischen. Außerdem zeigten Deutsche, die erst kurze Zeit in Berlin lebten ähnliche Amplituden wie langjährige Einheimische. Insgesamt unterstützt Studie 1 die soziale Immersions-Hypothese, dass die Immersion in eine neue ethnische Umgebung zu einer Vergrößerung der N170 führt. Studie 2 untersuchte die Schriftsystem-Hypothese bei chinesischen und deutschen Erstlesern am Ende der ersten oder zu Beginn des zweiten Schuljahres an ihrem jeweiligen Heimatort. Die Ergebnisse unterstützen die Schriftsystem-Hypothese, dass chinesische Kinder eine größere N170 auf Gesichter zeigen als deutsche. Insgesamt konnte die vorliegende Dissertation zwei neue Einfluss-Faktoren auf das Gesichterverarbeitungs-System nachweisen, das erworbene Schriftsystem (logographische versus alphabetisch) und die Erfahrung sozialer Immersion in eine neue ethnische Umgebung. Diese Effekte zu ganz unterschiedlichen Zeitpunkten der Entwicklung (Kindheit vs. junges Erwachsenen-Alter) zeigen, dass das Gesichterverarbeitungs-System über lange Zeit seine Plastizität behält.Informal reports have shown larger face-elicited N170 component of event-related potential (ERP) in Asians than Caucasians participants. I proposed that different experience with logographic versus alphabetic scripts (script system hypothesis) or by exposure to abundant novel faces during the immersion into a new social and/or ethnic environment (social immersion hypothesis) as a possible cause. To test these hypotheses, I conducted two cross-cultural ERP studies with adults and Children. In Study 1, I examined Chinese locals and non-Chinese foreigners in Hongkong, and German locals and Chinese foreigners in Berlin. It turned out that the foreigners in both locations showed larger N170 amplitudes to faces than the locals. In addition, Germans who had only lived in Berlin for a short time showed similar face N170 amplitudes as long-term Berlin residents. In sum, Study 1 supports the social immersion hypothesis that immersing into a new ethnic environment drives an increase of face N170. Study 2 investigated the scripts system hypothesis in Chinese and German early readers assessed at the end of the first-grade or at the beginning of the second grade in their respective home towns. The findings support the script system hypothesis that Chinese children showed larger face N170 amplitudes than German children. Overall, the present thesis demonstrated two new influencing factors on the face processing system, the acquired script system (logographic vs. alphabetic) and the social immersion experience in a new ethnic environment. More specifically, learning to read a visually complex logographic Chinese script or immersing into an other-ethnic social environment facilitates early perceptual processing of faces. These effects acquired at different stages of development (early childhood versus young adulthood) show that the face processing system retains its plasticity over a long period of time

Attention Restraint, Working Memory Capacity, and Mind Wandering: Do Emotional Valence or Intentionality Matter?

Attention restraint appears to mediate the relationship between working memory capacity (WMC) and mind wandering (Kane et al., 2016). Prior work has identifed two dimensions of mind wandering—emotional valence and intentionality. However, less is known about how WMC and attention restraint correlate with these dimensions. Te current study examined the relationship between WMC, attention restraint, and mind wandering by emotional valence and intentionality. A confrmatory factor analysis demonstrated that WMC and attention restraint were strongly correlated, but only attention restraint was related to overall mind wandering, consistent with prior fndings. However, when examining the emotional valence of mind wandering, attention restraint and WMC were related to negatively and positively valenced, but not neutral, mind wandering. Attention restraint was also related to intentional but not unintentional mind wandering. Tese results suggest that WMC and attention restraint predict some, but not all, types of mind wandering