First impressions: A survey on vision-based apparent personality trait analysis

© 2019 IEEE. Personal use of this material is permitted. Permission from IEEE must be obtained for all other uses, in any current or future media, including reprinting/republishing this material for advertising or promotional purposes,creating new collective works, for resale or redistribution to servers or lists, or reuse of any copyrighted component of this work in other works.Personality analysis has been widely studied in psychology, neuropsychology, and signal processing fields, among others. From the past few years, it also became an attractive research area in visual computing. From the computational point of view, by far speech and text have been the most considered cues of information for analyzing personality. However, recently there has been an increasing interest from the computer vision community in analyzing personality from visual data. Recent computer vision approaches are able to accurately analyze human faces, body postures and behaviors, and use these information to infer apparent personality traits. Because of the overwhelming research interest in this topic, and of the potential impact that this sort of methods could have in society, we present in this paper an up-to-date review of existing vision-based approaches for apparent personality trait recognition. We describe seminal and cutting edge works on the subject, discussing and comparing their distinctive features and limitations. Future venues of research in the field are identified and discussed. Furthermore, aspects on the subjectivity in data labeling/evaluation, as well as current datasets and challenges organized to push the research on the field are reviewed.Peer ReviewedPostprint (author's final draft

A survey on perceived speaker traits: personality, likability, pathology, and the first challenge

The INTERSPEECH 2012 Speaker Trait Challenge aimed at a unified test-bed for perceived speaker traits – the first challenge of this kind: personality in the five OCEAN personality dimensions, likability of speakers, and intelligibility of pathologic speakers. In the present article, we give a brief overview of the state-of-the-art in these three fields of research and describe the three sub-challenges in terms of the challenge conditions, the baseline results provided by the organisers, and a new openSMILE feature set, which has been used for computing the baselines and which has been provided to the participants. Furthermore, we summarise the approaches and the results presented by the participants to show the various techniques that are currently applied to solve these classification tasks

Positive emotion processing deficits in schizophrenia

Affective impairments were examined in patients with and without deficit syndrome schizophrenia. A battery of tests designed to measure emotional experience, emotional information processing, and emotional perception were administered to deficit (n = 15) and non-deficit syndrome (n = 26) schizophrenia patients classified according to the Schedule for the Deficit Syndrome, and matched non-patient control subjects (n = 22). As predicted, in comparison to non-deficit patients and controls, deficit syndrome patients reported less frequent and intense experience of positive emotion, recalled significantly fewer positive words, and displayed an impaired ability to accurately identify and judge the valence of pleasant odors. Additionally, deficit patients demonstrated a unique failure to have their attention captured by positive information, as well as less accurate and efficient labeling of positive faces than non-deficit patients or controls. Abnormalities were also associated with negative emotions, such that deficit syndrome patients demonstrated impairment at identifying fearful faces, were less accurate at judging negative smells, had a bias toward recalling anger words, and displayed an elevated attentional lingering effect for negative information. These findings indicate that the deficit syndrome is associated with affective disturbances that impact a number of cognitive and sensory domains, and provide support for the notion that abnormalities may be most severe in relation to the experience and processing of positive emotions. These abnormalities may be due to a mood-congruent processing abnormality, and are consistent with the notion that frontal and limbic system dysfunction may be core to deficit syndrome schizophrenia

Affective dysfunction and affective interference in schizotypy

Affective dysfunction is a core feature of schizophrenia spectrum disorders. Schizophrenic and schizotypal participants report higher levels of unpleasant and lower levels of pleasant trait affect than controls. In response to pleasant stimuli, though, participants often report similar levels of pleasant emotion to controls, but heightened unpleasant emotion, suggesting pleasant experiences may be affected by intrusive unpleasant emotion. An emotional Stroop task was used to examine the relationship between affective interference and trait affect in schizotypy. No significant differences were found between schizotypal participants and controls on e-Stroop performance, but schizotypal participants did self-report more unpleasant trait affect and less pleasant trait affect than controls. Of the schizotypy symptom dimensions, only cognitive disorganization was significantly correlated with unpleasant interference on the e-Stroop. Self-reported trait affect was not correlated with e-Stroop performance, but unpleasant trait affect was correlated with positive schizotypy symptoms and pleasant trait affect was inversely correlated with negative symptoms. Results suggest avenues for future exploration of unpleasant trait bias and cognitive dysfunction in schizophrenia-spectrum disorders

Fact sheet: Automatic Self-Reported Personality Recognition Track

We propose an informed baseline to help disentangle the various contextual factors of influence in this type of case studies. For this purpose, we analysed the correlation between the given metadata and the self-assigned personality trait scores and developed a model based solely on this information. Further, we compared the performance of this informed baseline with models based on state-of-the-art visual, linguistic and audio features. For the present dataset, a model trained solely on simple metadata features (age, gender and number of sessions) proved to have superior or similar performance when compared with simple audio, linguistic or visual features-based systems

Identifying Speaker State from Multimodal Cues

Automatic identification of speaker state is essential for spoken language understanding, with broad potential in various real-world applications. However, most existing work has focused on recognizing a limited set of emotional states using cues from a single modality. This thesis describes my research that addresses these limitations and challenges associated with speaker state identification by studying a wide range of speaker states, including emotion and sentiment, humor, and charisma, using features from speech, text, and visual modalities. The first part of this thesis focuses on emotion and sentiment recognition in speech. Emotion and sentiment recognition is one of the most studied topics in speaker state identification and has gained increasing attention in speech research recently, with extensive emotional speech models and datasets published every year. However, most work focuses only on recognizing a set of discrete emotions in high-resource languages such as English, while in real-life conversations, emotion is changing continuously and exists in all spoken languages. To address the mismatch, we propose a deep neural network model to recognize continuous emotion by combining inputs from raw waveform signals and spectrograms. Experimental results on two datasets show that the proposed model achieves state-of-the-art results by exploiting both waveforms and spectrograms as input. Due to the higher number of existing textual sentiment models than speech models in low-resource languages, we also propose a method to bootstrap sentiment labels from text transcripts and use these labels to train a sentiment classifier in speech. Utilizing the speaker state information shared across modalities, we extend speech sentiment recognition from high-resource languages to low-resource languages. Moreover, using the natural verse-level alignment in the audio Bibles across different languages, we also explore cross-lingual and cross-modality sentiment transfer. In the second part of the thesis, we focus on recognizing humor, whose expression is related to emotion and sentiment but has very different characteristics. Unlike emotion and sentiment that can be identified by crowdsourced annotators, humorous expressions are highly individualistic and cultural-specific, making it hard to obtain reliable labels. This results in the lack of data annotated for humor, and thus we propose two different methods to automatically and reliably label humor. First, we develop a framework for generating humor labels on videos, by learning from extensive user-generated comments. We collect and analyze 100 videos, building multimodal humor detection models using speech, text, and visual features, which achieves an F1-score of 0.76. In addition to humorous videos, we also develop another framework for generating humor labels on social media posts, by learning from user reactions to Facebook posts. We collect 785K posts with humor and non-humor scores and build models to detect humor with performance comparable to human labelers. The third part of the thesis focuses on charisma, a commonly found but less studied speaker state with unique challenges -- the definition of charisma varies a lot among perceivers, and the perception of charisma also varies with speakers' and perceivers' different demographic backgrounds. To better understand charisma, we conduct the first gender-balanced study of charismatic speech, including speakers and raters from diverse backgrounds. We collect personality and demographic information from the rater as well as their own speech, and examine individual differences in the perception and production of charismatic speech. We also extend the work to politicians' speech by collecting speaker trait ratings on representative speech segments of politicians and study how the genre, gender, and the rater's political stance influence the charisma ratings of the segments

Unintentional retrieval of stereotype congruent memories

The present study investigated the prediction that the stereotype physical attractiveness produces automatic memory, this means automatic encoding and retrieval of stereotype congruent information. Furthermore, subjects' relationship between automatic memory and explicit prejudice was explored. Forty-seven subjects participated in a novel implicit memory test. After conceptual priming, they judged the valence of "face-trait word" pairs, in old and new stimuli. Reaction times and self-reported prejudice levels were recorded. Results confirmed automatic memory. However, participants failed to exhibit better automatic memory for stereotype congruent stimuli than for stereotype incongruent stimuli. Significant interactions showed that participants unintentionally retrieved positive and negative traits together with attractive faces faster. A positive trend was found between subjects' automatic memory for stereotype prejudice information and their explicit prejudice attitude. The findings support automaticity of memory

A review of self-processing biases in cognition

When cues in the environment are associated with self (e.g., one’s own name, face, or coffee cup), these items trigger processing biases such as increased attentional focus, perceptual prioritization and memorial support. This paper reviews the existing literature on self-processing biases before introducing a series of studies that provide new insight into the influence of the self on cognition. In particular, the studies examine affective and memorial biases for self-relevant stimuli, and their flexible application in response to different task demands. We conclude that self-processing biases function to ensure that self-relevant information is attended to and preferentially processed because this is a perpetual goal of the self-system. However, contrary task-demands or priming can have an attenuating effect on their influence, speaking to the complexity and dynamism of the self-processing system in cognition

Mentalizing the Self in Adolescence and its Links with Schizotypal Trait Expression

Contemporary research suggests that clinical psychosis is distally linked with schizotypal trait expression and more proximally with the breakdown of psychological processes pertaining to mentalizing. Although previous findings are suggestive of a relationship between trait-vulnerability for psychosis and mentalizing difficulties, they involve adult participants either within or beyond the critical period of illness onset. To date, little is known about the process of mentalizing during the critical developmental period of adolescence or its associations with schizotypal trait dimensions. In a series of empirical studies, the current thesis used novel experimental tasks and self-report measures in samples of typically-developing young people to: (1) examine the nature of associations linking schizotypal trait dimensions in adolescence to disruptions in mentalizing processes involving both the understanding of the self and others; (2) further understand the processes that sustain self-awareness during adolescence by examining the effects that age, cognitive effort and emotional valence may exert on self- and reality-monitoring performance; and (3) prospectively assess the nature of the relation between mentalizing processes sustaining self- (self-monitoring) and other-awareness (ToM) from adolescence to young adulthood. Overall, the findings of the current thesis provide novel data suggesting that he expression of schizotypal traits that impede interpersonal communication with others in adolescence are associated with difficulties in self and other understanding. Regarding the development of psychological processes sustaining self-awareness, current data suggest that although both self- and reality-monitoring abilities may be established in pre-adolescent development, reality-monitoring capacities for emotionally-charged material may undergo further elaboration from adolescence to young adulthood. In addition, the data of the current thesis suggest that increased cognitive effort and emotional valence during memory encoding may respectively lead to self- and reality-monitoring confusions. Finally, the findings of the current thesis suggest that different types of self-monitoring misattributions in adolescence can prospectively predict specific patterns of ToM dysfunction at 5-year follow-up