Improving Person-Independent Facial Expression Recognition Using Deep Learning

Over the past few years, deep learning, e.g., Convolutional Neural Networks (CNNs) and Generative Adversarial Networks (GANs), have shown promise on facial expression recog- nition. However, the performance degrades dramatically especially in close-to-real-world settings due to high intra-class variations and high inter-class similarities introduced by subtle facial appearance changes, head pose variations, illumination changes, occlusions, and identity-related attributes, e.g., age, race, and gender. In this work, we developed two novel CNN frameworks and one novel GAN approach to learn discriminative features for facial expression recognition. First, a novel island loss is proposed to enhance the discriminative power of learned deep features. Specifically, the island loss is designed to reduce the intra-class variations while enlarging the inter-class differences simultaneously. Experimental results on three posed facial expression datasets and, more importantly, two spontaneous facial expression datasets have shown that the proposed island loss outperforms the baseline CNNs with the traditional softmax loss or the center loss and achieves better or at least comparable performance compared with the state-of-the-art methods. Second, we proposed a novel Probabilistic Attribute Tree-CNN (PAT-CNN) to explic- itly deal with the large intra-class variations caused by identity-related attributes. Specif- ically, a novel PAT module with an associated PAT loss was proposed to learn features in a hierarchical tree structure organized according to identity-related attributes, where the final features are less affected by the attributes. We further proposed a semi-supervised strategy to learn the PAT-CNN from limited attribute-annotated samples to make the best use of available data. Experimental results on three posed facial expression datasets as well as four spontaneous facial expression datasets have demonstrated that the proposed PAT- CNN achieves the best performance compared with state-of-the-art methods by explicitly modeling attributes. Impressively, the PAT-CNN using a single model achieves the best performance on the SFEW test dataset, compared with the state-of-the-art methods using an ensemble of hundreds of CNNs. Last, we present a novel Identity-Free conditional Generative Adversarial Network (IF- GAN) to explicitly reduce high inter-subject variations caused by identity-related attributes, e.g., age, race, and gender, for facial expression recognition. Specifically, for any given in- put facial expression image, a conditional generative model was developed to transform it to an “average” identity expressive face with the same expression as the input face image. Since the generated images have the same synthetic “average” identity, they differ from each other only by the displayed expressions and thus can be used for identity-free facial expression classification. In this work, an end-to-end system was developed to perform facial expression generation and facial expression recognition in the IF-GAN framework. Experimental results on four well-known facial expression datasets including a sponta- neous facial expression dataset have demonstrated that the proposed IF-GAN outperforms the baseline CNN model and achieves the best performance compared with the state-of- the-art methods for facial expression recognition

Modeling Emotion Influence from Images in Social Networks

Images become an important and prevalent way to express users' activities, opinions and emotions. In a social network, individual emotions may be influenced by others, in particular by close friends. We focus on understanding how users embed emotions into the images they uploaded to the social websites and how social influence plays a role in changing users' emotions. We first verify the existence of emotion influence in the image networks, and then propose a probabilistic factor graph based emotion influence model to answer the questions of "who influences whom". Employing a real network from Flickr as experimental data, we study the effectiveness of factors in the proposed model with in-depth data analysis. Our experiments also show that our model, by incorporating the emotion influence, can significantly improve the accuracy (+5%) for predicting emotions from images. Finally, a case study is used as the anecdotal evidence to further demonstrate the effectiveness of the proposed model

Understanding the voluntary moderation practices in live streaming communities

Harmful content, such as hate speech, online abuses, harassment, and cyberbullying, proliferates across various online communities. Live streaming as a novel online community provides ways for thousands of users (viewers) to entertain and engage with a broadcaster (streamer) in real-time in the chatroom. While the streamer has the camera on and the screen shared, tens of thousands of viewers are watching and messaging in real-time, resulting in concerns about harassment and cyberbullying. To regulate harmful content—toxic messages in the chatroom, streamers rely on a combination of automated tools and volunteer human moderators (mods) to block users or remove content, which is termed content moderation. Live streaming as a mixed media contains some unique attributes such as synchronicity and authenticity, making real-time content moderation challenging. Given the high interactivity and ephemerality of live text-based communication in the chatroom, mods have to make decisions with time constraints and little instruction, suffering cognitive overload and emotional toll. While much previous work has focused on moderation in asynchronous online communities and social media platforms, very little is known about human moderation in synchronous online communities with live interaction among users in a timely manner. It is necessary to understand mods’ moderation practices in live streaming communities, considering their roles to support community growth. This dissertation centers on volunteer mods in live streaming communities to explore their moderation practices and relationships with streamers and viewers. Through quantitative and qualitative methods, this dissertation mainly focuses on three aspects: the strategies and tools used by moderators, the mental model and decision-making process applied toward violators, and the conflict management present in the moderation team. This dissertation uses various socio-technical theories to explain mods’ individual and collaborative practices and suggests several design interventions to facilitate the moderation process in live streaming communities

How can drama benefit children’s language learning and moral thinking in a Chinese early years educational context?

In my doctoral research, I set out to explore, as a drama practitioner, how drama can benefit children’s language learning and moral thinking in a Chinese early years educational setting. Since the 1990s, when drama education began to be introduced into mainland China, very little academic research has been carried out to examine its development and the ways it has been introduced into specific educational contexts. This research is intended to help Chinese drama practitioners and researchers by presenting an in-depth study of my own attempts to apply a particular approach to drama education in a specific educational setting. This setting - that of a kindergarten - is one of the key areas in which drama education has begun to be practised. I begin the dissertation by reflecting upon current developments in the field in mainland China and problems that persist, particularly in what I see as some confusions between theoretical justifications and the actual practice of drama in Chinese early years education, with particular reference to the influential writings of two leading professors in the field. In the process, I explain the theories that have shaped my own approach. These are very influenced by the practices of drama in education as developed in the UK and, in particular, those I learned and practised myself during the MA in Drama and Theatre Education that I took at Warwick University. For the research in this study, I have applied reflexive-reflective case study as my methodology, and used a personal journal, video recordings, interviews, children’s drawings and recordings of their storytelling, observations, and the critical comments of a member of staff as co-researcher as my research methods. All my teaching was carried out in a local public kindergarten for 5-6 year-old children near the city of Cheng-du, where I currently live and work. My reflections examine the successes and shortcomings of my planning and my teaching; problems related to deeply held educational beliefs and practices that contradicted or undermined drama practice whilst, on the surface, offering it support; strengths and weaknesses of the research itself; and reflections on the key fields of how drama can relate to both language learning and moral education in the early years classroom, as informed by this study. My aim is to offer a thoroughly researched, first-hand experience to inform current and future Chinese drama practitioners of the problematics and possibilities of introducing drama education into Chinese educational settings