Decoding High Level Influences on Facial Expression Recognition

The present thesis explores the neural mechanisms underlying the recognition of emotion; the effect of high-level influences such as prior knowledge, task goals and the possible contribution of embodied simulation in facial expression recognition. The initial experiments (Chapters 2 & 3) investigate high-level processing that occurs when facial features are occluded in the recognition of facial expressions (visual route of recognition). This research examines the information about occluded facial features in early visual (V1-V3), face and emotion sensitive areas with fMRI, as well as the temporal dynamics of posterior brain regions in processing occluded facial features with EEG. MVPA reveals similar patterns of decoding across nonoverlapping samples of face information, suggesting the involvement of contextual influences beyond low-level processing (Chapter 2), as well as reliable decoding of facial expression (happy, fear and disgust) in conditions missing feature information (Chapters 2 & 3). This decoding, found 50-700ms, has three decoding phases, which potentially reveal the presence of feedforward and feedback processes (Chapter 3). These chapters also investigate the influence of task constraints, finding decoding differences between implicit and explicit processing conditions. Overall, this research helps understand how the brain deals with occluded stimuli; in keeping with accounts implying the rich role for top-down influences, such as predictive coding. The following experiment (Chapter 4) investigates embodiment in emotion recognition with fMRI; exploring shared representation in the perception and production of facial expression. MVPA reveals reliable expression decoding in the premotor brain regions across perception and production, demonstrating representational overlap across the sensory perception and motor production of expression. This tentatively supports strongly embodied simulation-based (nonvisual) theories. Collectively, the present research contributes to our knowledge of high-level influences in facial expression recognition, supporting the involvement of visual and non-visual routes to recognition, as well as providing further directions for future research

Psicofisiologia do testemunho ocular

Doutoramento em PsicologiaAs testemunhas oculares são muitas vezes o único meio que temos para aceder à autoria de um crime. Contudo, apesar dos 100 anos de evidência de erros no testemunho ocular, a consciência das suas limitações como meio de prova só ganhou força no advento do ADN. De facto os estudos de exoneração mostraram que 70 % das ilibações estavam associadas a erros de testemunho ocular. Estes erros têm um impacto social elevado principalmente os falsos positivos, por colocar inocentes na prisão. De acordo com a literatura, deverão ser utilizadas novas abordagens para tentar reduzir o numero de erros de identificação. Destas abordagens, destacam-se a análise dos padrões de movimentos oculares e os potenciais evocados. Nos nossos estudos utilizamos essas novas abordagens com o objetivo de examinar os padrões de acerto ou de identificação do criminoso, usando um paradigma de deteção de sinal. No que diz respeito aos movimentos oculares, não foram encontrados padrões robustos de acerto. No entanto, obtiveram-se evidências oculométricas de que a fusão de dois procedimentos (Alinhamento Simultâneo depois de um Alinhamento Sequencial com Regra de Paragem) aumenta a probabilidade de acerto. Em relação aos potenciais evocados, a P100 registou maior amplitude quando identificamos um inocente. Este efeito é concomitante com uma hiperactivação no córtex prefrontal ventromedial (CPFVM) identificada na análise de estimação de fontes. Esta hiperativação poderá estar relacionada com uma exacerbação emocional da informação proveniente da amígdala. A literatura relaciona a hiperativação no CPFVM com as falsas memorias, e estes resultados sugerem que a P100 poderá ser um promissor indicador de falsos positivos. Os resultados da N170 não nos permitem associar este componente ao acerto na identificação. Relativamente à P300, os resultados mostram uma maior amplitude deste componente quando identificamos corretamente um alvo, mas não diferiu significativamente de quando identificamos um inocente. Porém, a estimação de fontes mostrou que nessa janela temporal (300-600 ms) se verifica uma hipoativação dos Campos Oculares Frontais (COF) quando um distrator é identificado. Baixas ativações dos COF estão relacionadas com redução da eficiência de processamento e com a incapacidade para detetar alvos. Nas medidas periféricas, a eletromiografia facial mostrou que a maior ativação do corrugador e a menor ativação do zigomático são um bom indicador de quando estamos perante um criminoso. No que diz respeito ao ritmo cardíaco, a desaceleração esperada para os alvos devido à sua saliência emocional apenas foi obtida quando a visualização de um alvo foi acompanhada por um erro na identificação (i.e., um falso negativo). Neste trabalho de investigação parece que o sistema nervoso periférico está a responder corretamente, identificando o alvo, por ser emocionalmente mais saliente, enquanto que a modulação executiva efectuada pelo CPFVM conduz ao falso positivo. Os resultados obtidos são promissores e relevantes, principalmente quando o resultado de um erro poderá ser uma condenação indevida e, consequentemente, uma vida injustamente destruída.Eyewitnesses are often the only way we can access the author of a crime. However, despite 100 years of evidence of errors in eyewitness testimony, awareness of its limitations only gained strength with the advent of DNA. In fact, 70% of exonerations have been associated with eyewitness errors. These errors have a high social impact, mainly false positives. According to the literature, new approaches to try to reduce the number of identification errors should be used. Of these, the study of oculometric patterns and event-related Potentials (ERP) stand out. In our studies, these new approaches were used with the objective of examining patterns of accuracy, using a signal detection paradigm. Regarding eye movements, no entirely clear patterns were found. However, there was oculometric evidence that the merging of two procedures (Simultaneous Lineup after a Sequential Lineup with Stopping Rule) increases performance accuracy. Regarding ERPs, the P100 registered a larger amplitude when an innocent was identified. This effect is concomitant with a hyperactivation in the ventromedial prefrontal cortex (VMPFC) identified by source estimation analysis. This hyperactivation might be related to an emotional exacerbation of the information coming from the amygdala. The literature relates the hyperactivation in the VMPFC with false memories, and these results suggest that the P100 component might be a promising marker of false positive errors. The results of the N170 do not allow to associate this component with accuracy. Regarding the P300, the results showed a greater amplitude of this component when a target was correctly identified but did not differ significantly from when an innocent was identified. However, source analysis in this time window (300-600 ms) showed a hypoactivation of Frontal Eye Fields (FEF) when a distractor was identified. FEF inactivations are related to the reduction of processing efficiency and to the inability to detect a target. Concerning the peripheral measures, facial electromyography showed that the greater activation of the corrugator and the lower activation of the zygomaticus are a good marker of when we are facing a perpetrator. Regarding heart rate, the expected deceleration for the targets due to their emotional salience was only obtained when the visualization of a target was accompanied by an error in the identification (i.e., a miss). In this research it seems that the peripheral nervous system is responding correctly, identifying the target, because it is emotionally more salient, while the executive modulation carried out by the VMPFC causes the false positive error. The results presently obtained are promising and relevant, especially when the result of an error might be an undue condemnation of an innocent and consequently a destroyed life

A externalização e o processamento de expressões faciais de emoção

Mestrado em Psicologia ForenseA externalização consiste num espectro de personalidade que conjuga comportamentos antissociais, traços de personalidade associados ao comportamento desinibido e agressivo e o consumo de substâncias lícitas e ilícitas. A etiologia desta perturbação ainda não é consensual. Para além de outros problemas, esta população apresenta dificuldades na regulação emocional nomeadamente através da interpretação e demonstração de emoções. Como tal, esta investigação teve como principal objetivo investigar se existem diferenças ao nível do processamento de expressões faciais de emoção entre indivíduos com diferentes níveis de externalização através de medidas comportamentais (tempos de reação e percentagem de acertos) e eletroencefalográficas (análise de potenciais evocados). Relativamente às últimas, estudou-se os componentes de onda P100 e N170 bem como 11 janelas temporais de 50 ms cada entre os 230 ms e os 780 ms após o estímulo. Neste estudo participaram 54 estudantes da Universidade de Aveiro que constituíram os grupos de elevado, médio e baixo nível de externalização com base na pontuação obtida no Inventário de Externalização. Os participantes realizaram uma tarefa experimental de categorização emocional que tinha como objetivo avaliar a capacidade de discriminar entre expressões faciais emocionais e neutras, tendo sido incluídas as emoções básicas alegria, medo, raiva, tristeza, nojo e surpresa. Os resultados comportamentais parecem apontar para dificuldades no reconhecimento de expressões faciais de tristeza comparativamente com as restantes emoções e expressões faciais neutras. Relativamente aos resultados obtidos para a P100 e N170, estes apontam para a existência de diferenças ao nível do processamento emocional a curtas latências. A longas latências também se verificaram diferenças entre os grupos experimentais para as emoções de medo e raiva. Para a emoção medo verificaram-se diferenças entre o processamento de caras neutras e com conteúdo emocional. Transversalmente, observou-se que o hemisfério que registou maiores amplitudes foi o hemisfério direito. Havendo ainda uma lacuna na literatura relativamente ao estudo da externalização e suas consequências ao nível do processamento facial de emoções, é importante a realização de outras investigações que explorem estes aspetos. Nomeadamente seria de interesse explorar de que forma ocorre o processamento facial de emoções em estímulos faciais cuja intensidade emocional é variável. No entanto, o presente estudo contribui já para uma melhor compreensão dos aspetos comportamentais e correlatos psicofisiológicos associados ao processamento de expressões emocionais em indivíduos com elevada externalização.The externalization consists in a spectrum of personality that encompasses anti-social behaviors, personality traits associated with disinhibited and aggressive behavior and the abuse of licit and illicit substances. The etiology of this disorder is still not consensual. In addition to other problems, this population shows difficulties in emotional regulation, namely through the interpretation and demonstration of emotions. Therefore, this research aimed to investigate whether there are differences in the processing of facial expressions of emotion between individuals with different levels of externalization, using behavioral measures (reaction times and percentage of correct responses) and electroencephalographic measures (analysis of event-related potentials). Regarding the latter, the waves P100 and N170 were studied, as well as 11 temporal windows of 50ms each, between 230ms and 780ms after stimulus onset. Participants were 54 students from the University of Aveiro, divided in groups of high, medium and low externalization, based on their score on the Externalizing Inventory. Participants performed an emotional categorization task that aimed to assess their ability to discriminate between faces displaying emotional expressions and neutral faces. The study included the basic emotions happiness, fear, anger, sadness, disgust and surprise. Behavioral results seem to point to difficulties in recognizing facial expressions of sadness compared with other emotions and neutral facial expressions. Regarding the results obtained for the P100 and N170, these point to the existence of differences in emotional processing at short latencies. At long latencies, some differences between the experimental groups to the emotions of fear and anger were also observed. For the emotion ‘fear’, there were differences between the processing of emotional faces and faces with neutral expression. Overall, it was observed that the right hemisphere showed larger amplitudes than the left. Given the reduced number of studies in the literature concerning the study of externalization and its effects on the processing of facial emotions, it will be important to conduct further research to explore these aspects. In particular it would be interesting to explore how facial processing of emotions occurs in facial stimuli whose emotional intensity is variable. However, the present study already contributes to a better understanding of the behavioral and psychophysiological aspects associated with the processing of emotional expressions in individuals with high externalization

Face adaptation: behavioural and electrophysiological studies on the perception of eye gaze and gender

Whereas the investigation of perceptual aftereffects has a very long tradition in studies on low-level vision, the report and analysis of face-related high-level aftereffects is a very recent line of research. Webster et al. (2004, Nature) first showed a visual aftereffect on the perception of face gender by demonstrating that adaptation to male faces biased the subsequent classification of androgynous faces towards female gender. Similar adaptation effects have also been observed for one of the most important social signals: human eye gaze. Jenkins et al. (2006, Psychological Science) found that adaptation to gaze in one direction virtually eliminated participants’ ability to perceive smaller gaze deviations in the same direction. The present thesis further examined these high-level face aftereffects by determining the temporal characteristics of gaze adaptation and by analysing the neural correlates of eye gaze and gender adaptation. A behavioural study on the temporal decay of gaze adaptation effects shed further light on their basic characteristics: not only was the aftereffect surprisingly long-lasting, but its exponential decay revealed remarkable similarity with the time course of low-level adaptation effects. Further, in a series of event-related potential (ERP) studies it was found that the N170 was only marginally affected by both eye gaze and gender adaptation, whereas pronounced effects of both kinds of adaptation emerged in the P3 component with smaller amplitudes in response to test stimuli similar to the adaptation condition. Finally, gaze adaptation was found to affect ERPs in an earlier time interval ~250-350 ms which appeared to be sensitive to the discrimination between direct vs. averted gaze even when this was only an illusionary percept induced by adaptation. Together, these studies extend previous knowledge of the temporal parameters and the neural correlates of high-level face adaptation

Décoder l’habileté perceptive dans le cerveau humain : contenu représentationnel et computations cérébrales

La capacité à reconnaître les visages de nos collègues, de nos amis et de nos proches est essentielle à notre réussite en tant qu'êtres sociaux. Notre cerveau accomplit cet exploit facilement et rapidement, dans une série d’opérations se déroulant en quelques dizaines de millisecondes à travers un vaste réseau cérébral du système visuel ventral. L’habileté à reconnaître les visages, par contre, varie considérablement d’une personne à l’autre. Certains individus, appelés «super-recognisers», sont capables de reconnaître des visages vus une seule fois dans la rue des années plus tôt. D’autres, appelés «prosopagnosiques», sont incapables de reconnaître le visage de leurs collègues ou leurs proches, même avec une vision parfaite. Une question simple reste encore largement sans réponse : quels mécanismes expliquent que certains individus sont meilleurs à reconnaître des visages? Cette thèse rapporte cinq articles étudiant les mécanismes perceptifs (articles 1, 2, 3) et cérébraux (articles 4, 5) derrière ces variations à travers différentes populations d’individus. L’article 1 décrit le contenu des représentations visuelles faciales chez une population avec un diagnostic de schizophrénie et d’anxiété sociale à l’aide d’une technique psychophysique Bubbles. Nous révélons pour la première fois les mécanismes en reconnaissance des expressions de cette population: un déficit de reconnaissance est accompagné par i) une sous-utilisation de la région des yeux des visages expressifs et ii) une sous-utilisation des détails fins. L’article 2 valide ensuite une nouvelle technique permettant de révéler simultanément le contenu visuel dans trois dimensions psychophysiques centrales pour le système visuel — la position, les fréquences spatiales, et l’orientation. L’article 3 a mesuré, à l'aide de cette nouvelle technique, le contenu représentationnel de 120 individus pendant la discrimination faciale du sexe et des expressions ( >500,000 observations). Nous avons observé de fortes corrélations entre l’habileté à discriminer le sexe et les expressions des visages, ainsi qu'entre l’habileté à discriminer le sexe et l’identité. Crucialement, plus un individu est habile en reconnaissance faciale, plus il utilise un contenu représentationnel similaire entre les tâches. L’article 4 a examiné les computations cérébrales de super-recognisers en utilisant l’électroencéphalographie haute-densité (EEG) et l’apprentissage automatique. Ces outils ont permis de décoder, pour la première fois, l’habileté en reconnaissance faciale à partir du cerveau avec jusqu’à 80% d’exactitude –– et ce à partir d’une seule seconde d’activité cérébrale. Nous avons ensuite utilisé la Representational Similarity Analysis (RSA) pour comparer les représentations cérébrales de nos participants à celles de modèles d’apprentissage profond visuels et langagiers. Les super-recognisers, comparé aux individus avec une habileté typique, ont des représentations cérébrales plus similaires aux computations visuelles et sémantiques de ces modèles optimaux. L’article 5 rapporte une investigation des computations cérébrales chez le cas le plus spécifique et documenté de prosopagnosie acquise, la patiente PS. Les mêmes outils computationnels et d’imagerie que ceux de l’article 4 ont permis i) de décoder les déficits d’identification faciale de PS à partir de son activité cérébrale EEG, et ii) de montrer pour la première fois que la prosopagnosie est associée à un déficit des computations visuelles de haut niveau et des computations cérébrales sémantiques.The ability to recognise the faces of our colleagues, friends, and family members is critical to our success as social beings. Our brains accomplish this feat with astonishing ease and speed, in a series of operations taking place in tens of milliseconds across a vast brain network of the visual system. The ability to recognise faces, however, varies considerably from one person to another. Some individuals, called "super-recognisers", are able to recognise faces seen only once years earlier. Others, called "prosopagnosics", are unable to recognise the faces of their colleagues or relatives, even with perfect vision and typical intelligence. A simple question remains largely unanswered: what mechanisms explain why some individuals are better at recognizing faces? This thesis reports five articles studying the perceptual (article 1, 2, 3) and neural (article 4, 5) mechanisms behind these variations across different populations of individuals. Article 1 describes the content of visual representations of faces in a population with a comorbid diagnosis of schizophrenia and social anxiety disorder using an established psychophysical technique, Bubbles. We reveal for the first time the perceptual mechanisms of expression recognition in this population: a recognition deficit is accompanied by i) an underutilization of the eye region of expressive faces and ii) an underutilization of fine details. Article 2 then validates a new psychophysical technique that simultaneously reveals the visual content in three dimensions central to the visual system — position, spatial frequencies, and orientation. We do not know, however, whether skilled individuals perform well across a variety of facial recognition tasks and, if so, how they accomplish this feat. Article 3 measured, using the technique validated in article 2, the perceptual representations of 120 individuals during facial discrimination of gender and expressions (total of >500,000 trials). We observed strong correlations between the ability to discriminate gender and facial expressions, as well as between the ability to discriminate gender and identify faces. More importantly, we found a positive correlation between individual ability and the similarity of perceptual representations used across these tasks. Article 4 examined differences in brain dynamics between super-recognizers and typical individuals using high-density electroencephalography (EEG) and machine learning. These tools allowed us to decode, for the first time, facial recognition ability from the brain with up to 80% accuracy — using a mere second of brain activity. We then used Representational Similarity Analysis (RSA) to compare our participants' brain representations to those of deep learning models of object and language classification. This showed that super-recognisers, compared to individuals with typical perceptual abilites, had brain representations more similar to the visual and semantic computations of these optimal models. Article 5 reports an investigation of brain computations in the most specific and documented case of acquired prosopagnosia, patient PS. The same computational tools used in article 4 enabled us to decode PS's facial identification deficits from her brain dynamics. Crucially, associations between brain deep learning models showed for the first time that prosopagnosia is associated with deficits in high-level visual and semantic brain computations

Investigating cortical arousal and cognition in schizophrenia and methamphetamine-induced psychotic disorder: an electroencephalography and cytokine study

Introduction: Schizophrenia (SCZ) and methamphetamine-induced psychotic disorder (MPD), are psychotic disorders characterized by positive symptoms (e.g., hallucinations and delusions), negative symptoms (e.g., social withdrawal, apathy), and impaired cognitive function. Despite the overlap in the clinical presentation of SCZ and MPD, no studies have compared electroencephalography (EEG) and inflammation across these two conditions. This study aimed to investigate key differences in brain electrical activity on EEG between SCZ and MPD by investigating; (1) relative frequency (alpha, theta, beta and delta) at rest; (2) cognitive performance and relative frequency activity during the continuous performance task (CPT) and cued target detection task (CTD); (3) differences in the P300 event-related potential waveform (ERP), a measure of attention, during the CPT and CTD; (4) cognitive performance and relative frequency and ERP (N170, P300) during the STROOP task, a measure of working memory and executive function; (5) the associations of (neuro) inflammatory markers with relative frequency and the P300 ERP waveform. Methods: 104 South African individuals, between the ages of 20 and 45 years, participated in this study: 69 outpatients (38 with SCZ (8 females/30 males), 31 with MPD (7 females/24 males)), and 35 healthy controls (CON: 15 females/20 males). All participants underwent a Structured Clinical Interview for Diagnostic Systematic Manual-IV (SCID-DSM-IV), with modifications to include changes made in DSM-5. EEG band frequency oscillations were recorded during baseline conditions: resting eyes open and resting eyes closed, and cognitive tasks (CPT, CTD and the Stroop task). Blood was drawn via venepuncture and serum was used for the analysis of cytokines (interleukin (IL) -1β IL-8, IL-10, IL-12p70, tumour necrosis factor-alpha and interferon-gamma (IFN-γ)) concentrations. Statistical analysis included assessment of normality using the Shapiro- Wilk test, with univariate one-way analysis of variance (ANOVA) of parametric data, and multiple independent Kruskal-Wallis ANOVA for non-parametric data (p