Computational Emotion Analysis From Images: Recent Advances and Future Directions

Emotions are usually evoked in humans by images. Recently, extensive research efforts have been dedicated to understanding the emotions of images. In this chapter, we aim to introduce image emotion analysis (IEA) from a computational perspective with the focus on summarizing recent advances and suggesting future directions. We begin with commonly used emotion representation models from psychology. We then define the key computational problems that the researchers have been trying to solve and provide supervised frameworks that are generally used for different IEA tasks. After the introduction of major challenges in IEA, we present some representative methods on emotion feature extraction, supervised classifier learning, and domain adaptation. Furthermore, we introduce available datasets for evaluation and summarize some main results. Finally, we discuss some open questions and future directions that researchers can pursue.Comment: Accepted chapter in the book "Human Perception of Visual Information Psychological and Computational Perspective

Affective Image Content Analysis: Two Decades Review and New Perspectives

Images can convey rich semantics and induce various emotions in viewers. Recently, with the rapid advancement of emotional intelligence and the explosive growth of visual data, extensive research efforts have been dedicated to affective image content analysis (AICA). In this survey, we will comprehensively review the development of AICA in the recent two decades, especially focusing on the state-of-the-art methods with respect to three main challenges -- the affective gap, perception subjectivity, and label noise and absence. We begin with an introduction to the key emotion representation models that have been widely employed in AICA and description of available datasets for performing evaluation with quantitative comparison of label noise and dataset bias. We then summarize and compare the representative approaches on (1) emotion feature extraction, including both handcrafted and deep features, (2) learning methods on dominant emotion recognition, personalized emotion prediction, emotion distribution learning, and learning from noisy data or few labels, and (3) AICA based applications. Finally, we discuss some challenges and promising research directions in the future, such as image content and context understanding, group emotion clustering, and viewer-image interaction.Comment: Accepted by IEEE TPAM

APSE: Attention-aware polarity-sensitive embedding for emotion-based image retrieval

With the popularity of social media, an increasing number of people are accustomed to expressing their feelings and emotions online using images and videos. An emotion-based image retrieval (EBIR) system is useful for obtaining visual contents with desired emotions from a massive repository. Existing EBIR methods mainly focus on modeling the global characteristics of visual content without considering the crucial role of informative regions of interest in conveying emotions. Further, they ignore the hierarchical relationships between coarse polarities and fine categories of emotions. In this paper, we design an attention-aware polarity-sensitive embedding (APSE) network to address these issues. First, we develop a hierarchical attention mechanism to automatically discover and model the informative regions of interest. Specifically, both polarity-and emotion-specific attended representations are aggregated for discriminative feature embedding. Second, we propose a generated emotion-pair (GEP) loss to simultaneously consider the inter-and intra-polarity relationships of the emotion labels. Moreover, we adaptively generate negative examples of different hard levels in the feature space guided by the attention module to further improve the performance of feature embedding. Extensive experiments on four popular benchmark datasets demonstrate that the proposed APSE method outperforms the state-of-the-art EBIR approaches by a large margin

PDANet: Polarity-consistent Deep Attention Network for Fine-grained Visual Emotion Regression

Existing methods on visual emotion analysis mainly focus on coarse-grained emotion classification, i.e. assigning an image with a dominant discrete emotion category. However, these methods cannot well reflect the complexity and subtlety of emotions. In this paper, we study the fine-grained regression problem of visual emotions based on convolutional neural networks (CNNs). Specifically, we develop a Polarity-consistent Deep Attention Network (PDANet), a novel network architecture that integrates attention into a CNN with an emotion polarity constraint. First, we propose to incorporate both spatial and channel-wise attentions into a CNN for visual emotion regression, which jointly considers the local spatial connectivity patterns along each channel and the interdependency between different channels. Second, we design a novel regression loss, i.e. polarity-consistent regression (PCR) loss, based on the weakly supervised emotion polarity to guide the attention generation. By optimizing the PCR loss, PDANet can generate a polarity preserved attention map and thus improve the emotion regression performance. Extensive experiments are conducted on the IAPS, NAPS, and EMOTIC datasets, and the results demonstrate that the proposed PDANet outperforms the state-of-the-art approaches by a large margin for fine-grained visual emotion regression. Our source code is released at: https://github.com/ZizhouJia/PDANet.Comment: Accepted by ACM Multimedia 201

Age-related decline in hippocampal tyrosine phosphatase PTPRO is a mechanistic factor in chemotherapy-related cognitive impairment.

Chemotherapy-related cognitive impairment (CRCI) or chemo brain is a devastating neurotoxic sequela of cancer-related treatments, especially for the elderly individuals. Here we show that PTPRO, a tyrosine phosphatase, is highly enriched in the hippocampus, and its level is tightly associated with neurocognitive function but declined significantly during aging. To understand the protective role of PTPRO in CRCI, a mouse model was generated by treating Ptpro-/- female mice with doxorubicin (DOX) because Ptpro-/- female mice are more vulnerable to DOX, showing cognitive impairments and neurodegeneration. By analyzing PTPRO substrates that are neurocognition-associated tyrosine kinases, we found that SRC and EPHA4 are highly phosphorylated/activated in the hippocampi of Ptpro-/- female mice, with increased sensitivity to DOX-induced CRCI. On the other hand, restoration of PTPRO in the hippocampal CA3 region significantly ameliorate CRCI in Ptpro-/- female mice. In addition, we found that the plant alkaloid berberine (BBR) is capable of ameliorating CRCI in aged female mice by upregulating hippocampal PTPRO. Mechanistically, BBR upregulates PTPRO by downregulating miR-25-3p, which directly targeted PTPRO. These findings collectively demonstrate the protective role of hippocampal PTPRO against CRCI

Synergistic Effect of SRY and Its Direct Target, WDR5, on Sox9 Expression

SRY is a sex-determining gene that encodes a transcription factor, which triggers male development in most mammals. The molecular mechanism of SRY action in testis determination is, however, poorly understood. In this study, we demonstrate that WDR5, which encodes a WD-40 repeat protein, is a direct target of SRY. EMSA experiments and ChIP assays showed that SRY could bind to the WDR5 gene promoter directly. Overexpression of SRY in LNCaP cells significantly increased WDR5 expression concurrent with histone H3K4 methylation on the WDR5 promoter. To specifically address whether SRY contributes to WDR5 regulation, we introduced a 4-hydroxy-tamoxifen-inducible SRY allele into LNCaP cells. Conditional SRY expression triggered enrichment of SRY on the WDR5 promoter resulting in induction of WDR5 transcription. We found that WDR5 was self regulating through a positive feedback loop. WDR5 and SRY interacted and were colocalized in cells. In addition, the interaction of WDR5 with SRY resulted in activation of Sox9 while repressing the expression of β-catenin. These results suggest that, in conjunction with SRY, WDR5 plays an important role in sex determination

Separase Phosphosite Mutation Leads to Genome Instability and Primordial Germ Cell Depletion during Oogenesis

To ensure equal chromosome segregation and the stability of the genome during cell division, Separase is strictly regulated primarily by Securin binding and inhibitory phosphorylation. By generating a mouse model that contained a mutation to the inhibitory phosphosite of Separase, we demonstrated that mice of both sexes are infertile. We showed that Separase deregulation leads to chromosome mis-segregation, genome instability, and eventually apoptosis of primordial germ cells (PGCs) during embryonic oogenesis. Although the PGCs of mutant male mice were completely depleted, a population of PGCs from mutant females survived Separase deregulation. The surviving PGCs completed oogenesis but produced deficient initial follicles. These results indicate a sexual dimorphism effect on PGCs from Separase deregulation, which may be correlated with a gender-specific discrepancy of Securin. Our results reveal that Separase phospho-regulation is critical for genome stability in oogenesis. Furthermore, we provided the first evidence of a pre-zygotic mitotic chromosome segregation error resulting from Separase deregulation, whose sex-specific differences may be a reason for the sexual dimorphism of aneuploidy in gametogenesis