The role of facial movements in emotion recognition

Most past research on emotion recognition has used photographs of posed expressions intended to depict the apex of the emotional display. Although these studies have provided important insights into how emotions are perceived in the face, they necessarily leave out any role of dynamic information. In this Review, we synthesize evidence from vision science, affective science and neuroscience to ask when, how and why dynamic information contributes to emotion recognition, beyond the information conveyed in static images. Dynamic displays offer distinctive temporal information such as the direction, quality and speed of movement, which recruit higher-level cognitive processes and support social and emotional inferences that enhance judgements of facial affect. The positive influence of dynamic information on emotion recognition is most evident in suboptimal conditions when observers are impaired and/or facial expressions are degraded or subtle. Dynamic displays further recruit early attentional and motivational resources in the perceiver, facilitating the prompt detection and prediction of others’ emotional states, with benefits for social interaction. Finally, because emotions can be expressed in various modalities, we examine the multimodal integration of dynamic and static cues across different channels, and conclude with suggestions for future research

Exploring the representation of caricatures, facial motion, and view-invariance in face space.

Faces present a vast array of information, from invariable features such as identity, to variable features such as expression, speech and pose. Humans have an incredible capability of recognising faces (familiar faces at least) and interpreting facial actions, even across changes in view. While there has been an explosion of research into developing artificial neural networks for many aspects of face processing, some of which seem to predict neural responses quite well, the current work focuses on face processing through simpler linear projection spaces. These linear projection spaces are formal instantiations of ‘face space’, built using principal component analysis (PCA). The concept of ‘face space’ (Valentine, 1991) has been a highly influential account of how faces might be represented in the brain. In particular, recent research supports the presence of a face space in the macaque brain in the form of a linear projection space, referred to as ‘axis coding’ in which individual faces can be coded as linear sum of orthogonal features. Here, these linear projection spaces are used for two streams of investigation. Firstly, we assessed the neurovascular response to hyper-caricatured faces in an fMRI study. Based on the assumption that faces further from average should project more strongly onto components in the linear space, we hypothesised that they should elicit a stronger response. Contrary to our expectations, we found little evidence for this in the fusiform face area (FFA) and face-selective cortex more generally, although the response pattern did become more consistent for caricatured faces in the FFA. We then explored the response to these caricatured faces in cortex typically associated with object processing. Interestingly, both the average response magnitude and response pattern consistency increased to these stimuli as caricaturing increased. At the current time it is unclear if this response allows some functional benefit for processing caricatured faces, or whether it simply reflects similarities in the low- and mid-level properties to certain objects. If the response is functional, then hyper-caricaturing could pave a route to improving face processing in individuals with prosopagnosia if technologies can be developed to automatically caricature faces in real-time. The second line of work addressed these linear projection spaces in the context of achieving view-invariance, specifically in the domain of facial motion and expression. How humans create view-invariant representations is still of interest, despite much research, however little work has focused on creating view-invariant representations outside of identity recognition. Likewise, there has been much research into face space and view-invariance separately, yet there is little evidence for how different views may be represented within a face space framework, and how motion might also be incorporated. Automatic face analysis systems mostly deal with pose by either aligning to a canonical frontal view or by using separate view-specific models. There is inconclusive evidence that the brain possesses an internal 3D model for ‘frontalising’ faces, therefore here we investigate how changes in view might be processed in a unified multi-view face space based on using a few prototypical 2D views. We investigate the functionality and biological plausibility of five identity-specific faces spaces, created using PCA, that allow for different views to be reconstructed from single-view video inputs of actors speaking. The most promising of these models first builds a separate orthogonal space for each viewpoint. The relationships between the components in neighbouring views are learned, and then reconstructions across views are made using a cascade of projection, transformation, and reconstruction. These reconstructions are then collated and used to build a multi-view space, which can reconstruct motion well across all learned views. This provides initial insight into how a biologically plausible, view-invariant system for facial motion processing might be represented in the brain. Moreover, it also has the capacity to improve view-transformations in automatic lip-reading software

Demystifying emotion-processing: autism, alexithymia, and the underlying psychological mechanisms

Despite extensive research, the mechanisms underpinning successful emotion recognition remain unclear. Constructionist, template-matching, and signal detection theories illuminate several emotion-related psychological processes that may be involved – namely the conceptualisation, experience, visual representation, and production of emotion – however, this requires empirical verification. Therefore, across the six empirical chapters described here, I developed and applied several novel experimental paradigms to assess the way in which individuals conceptualise, experience, visualise, produce and recognise emotion, and created new mathematically plausible, mechanistic models that shed light on the processes involved in emotion recognition. In doing so, I identified several candidate mechanisms that may underpin the emotion recognition difficulties seen in a range of clinical conditions, including autism spectrum disorder, and I (1) determined whether there are differences between autistic and non- autistic individuals in these emotion-related psychological processes, and (2) ascertained whether differences therein underpin emotion recognition challenges for autistic people. Ten years ago, it was theorised that the emotion-related difficulties of autistic individuals do not stem from autism per se, but rather alexithymia – a subclinical condition highly prevalent in the autistic population characterised by difficulties identifying and describing emotions. Since its inception, this theory has gained empirical support, with multiple studies documenting that alexithymia, and not autism, is associated with emotion- processing differences. However, to date, this evidence has largely been confined to the domain of emotion recognition. As such, it is unclear whether there are differences between autistic and non-autistic individuals in the conceptualisation, experience, visual representation, and production of emotion, after controlling for alexithymia. Here, I resolved this ambiguity, discerning the explanatory scope of the “alexithymia hypothesis”: there were no differences between autistic and non-autistic individuals in the understanding or differentiation of emotion concepts (Chapter 6), the precision or differentiation of emotional experiences (Chapter 6), and the speed (Chapter 3) or differentiation of visual emotion representations (Chapter 5), after controlling for alexithymia. Nevertheless, there were differences between groups with respect to the precision of visual representations (Chapter 5), the production of emotional facial expressions (Chapter 7), and recognition of specific emotions (Chapter 2), even after accounting for this confound. Despite suggestions that autistic individuals adopt alternative strategies to recognise the emotions of others, very few studies have examined mechanistic differences in emotion recognition between autistic and non-autistic people. Therefore, here I aimed to compare the processes involved in emotion recognition for these groups. Across multiple empirical chapters, I identified that there are similarities and differences in the processes implicated in emotion recognition for autistic and non-autistic people (Chapters 4, 5, 6, and 7), with autistic individuals relying on fewer emotion-related psychological processes. By elucidating several candidate mechanisms underpinning superior emotion recognition, my doctoral work paves the way for future supportive interventions to help both autistic and non-autistic individuals to accurately interpret other people’s emotions, thus ultimately fostering more successful and fluid social interactions

Exploring the representation of caricatures, facial motion, and view-invariance in face space.

Faces present a vast array of information, from invariable features such as identity, to variable features such as expression, speech and pose. Humans have an incredible capability of recognising faces (familiar faces at least) and interpreting facial actions, even across changes in view. While there has been an explosion of research into developing artificial neural networks for many aspects of face processing, some of which seem to predict neural responses quite well, the current work focuses on face processing through simpler linear projection spaces. These linear projection spaces are formal instantiations of ‘face space’, built using principal component analysis (PCA). The concept of ‘face space’ (Valentine, 1991) has been a highly influential account of how faces might be represented in the brain. In particular, recent research supports the presence of a face space in the macaque brain in the form of a linear projection space, referred to as ‘axis coding’ in which individual faces can be coded as linear sum of orthogonal features. Here, these linear projection spaces are used for two streams of investigation. Firstly, we assessed the neurovascular response to hyper-caricatured faces in an fMRI study. Based on the assumption that faces further from average should project more strongly onto components in the linear space, we hypothesised that they should elicit a stronger response. Contrary to our expectations, we found little evidence for this in the fusiform face area (FFA) and face-selective cortex more generally, although the response pattern did become more consistent for caricatured faces in the FFA. We then explored the response to these caricatured faces in cortex typically associated with object processing. Interestingly, both the average response magnitude and response pattern consistency increased to these stimuli as caricaturing increased. At the current time it is unclear if this response allows some functional benefit for processing caricatured faces, or whether it simply reflects similarities in the low- and mid-level properties to certain objects. If the response is functional, then hyper-caricaturing could pave a route to improving face processing in individuals with prosopagnosia if technologies can be developed to automatically caricature faces in real-time. The second line of work addressed these linear projection spaces in the context of achieving view-invariance, specifically in the domain of facial motion and expression. How humans create view-invariant representations is still of interest, despite much research, however little work has focused on creating view-invariant representations outside of identity recognition. Likewise, there has been much research into face space and view-invariance separately, yet there is little evidence for how different views may be represented within a face space framework, and how motion might also be incorporated. Automatic face analysis systems mostly deal with pose by either aligning to a canonical frontal view or by using separate view-specific models. There is inconclusive evidence that the brain possesses an internal 3D model for ‘frontalising’ faces, therefore here we investigate how changes in view might be processed in a unified multi-view face space based on using a few prototypical 2D views. We investigate the functionality and biological plausibility of five identity-specific faces spaces, created using PCA, that allow for different views to be reconstructed from single-view video inputs of actors speaking. The most promising of these models first builds a separate orthogonal space for each viewpoint. The relationships between the components in neighbouring views are learned, and then reconstructions across views are made using a cascade of projection, transformation, and reconstruction. These reconstructions are then collated and used to build a multi-view space, which can reconstruct motion well across all learned views. This provides initial insight into how a biologically plausible, view-invariant system for facial motion processing might be represented in the brain. Moreover, it also has the capacity to improve view-transformations in automatic lip-reading software