Multi-label Learning with Missing Values using Combined Facial Action Unit Datasets

Facial action units allow an objective, standardized description of facial micro movements which can be used to describe emotions in human faces. Annotating data for action units is an expensive and time-consuming task, which leads to a scarce data situation. By combining multiple datasets from different studies, the amount of training data for a machine learning algorithm can be increased in order to create robust models for automated, multi-label action unit detection. However, every study annotates different action units, leading to a tremendous amount of missing labels in a combined database. In this work, we examine this challenge and present our approach to create a combined database and an algorithm capable of learning under the presence of missing labels without inferring their values. Our approach shows competitive performance compared to recent competitions in action unit detection.Comment: Presented at the first Workshop on the Art of Learning with Missing Values (Artemiss) hosted by the 37th International Conference on Machine Learning (ICML) 202

Deriving Temporal Prototypes from Saliency Map Clusters for the Analysis of Deep-Learning-based Facial Action Unit Classification

Reliably determining the emotional state of a person is a difficult task for both humans as well as machines. Automatic detection and evaluation of facial expressions is particularly important if people are unable to express their emotional state themselves, for example due to cognitive impairments. Identifying the presence of Action Units in a human’s face is a psychologically validated approach of quantifying which emotion is expressed. To automate the detection process of Action Units Neural Networks have been trained. However, the black-box nature of Deep Neural Networks provides no insight on the relevant features identified during the decision process. Approaches of Explainable Artificial Intelligence have to be applied to provide an explanation why the network came to a certain conclusion. In this work "Layer-Wise Relevance Propagation" (LRP) in combination with the meta analysis approach "Spectral Relevance Analysis" (SpRAy) is used to derive temporal prototypes from predictions in video sequences. Temporal prototypes provide an aggregated view on the prediction of the network by grouping together similar frames by considering relevance. Additionally, a specific visualization method for temporal prototypes is presented that highlights the most relevant areas for a prediction of an Action Unit. A quantitative evaluation of our approach shows that temporal prototypes aggregate temporal information well. The proposed method can be used to generate concise visual explanations for a sequence of interpretable saliency maps. Based on the above, this work shall provide the foundation for a new temporal analysis method as well as an explanation approach that is supposed to help researchers and experts to gain a deeper understanding of how the underlying network decides which Action Units are active in a particular emotional state

Unique Class Group Based Multi-Label Balancing Optimizer for Action Unit Detection

Balancing methods for single-label data cannot be applied to multi-label problems as they would also resample the samples with high occurrences. We propose to reformulate this problem as an optimization problem in order to balance multi-label data. We apply this balancing algorithm to training datasets for detecting isolated facial movements, so-called Action Units. Several Action Units can describe combined emotions or physical states such as pain. As datasets in this area are limited and mostly imbalanced, we show how optimized balancing and then augmentation can improve Action Unit detection. At the IEEE Conference on Face and Gesture Recognition 2020, we ranked third in the Affective Behavior Analysis in-the-wild (ABAW) challenge for the Action Unit detection task.Comment: Accepted at the 15th IEEE International Conference on Automatic Face and Gesture Recognition 2020, Workshop "Affect Recognition in-the-wild: Uni/Multi-Modal Analysis & VA-AU-Expression Challenges". arXiv admin note: substantial text overlap with arXiv:2002.0323

Female, white, 27? : Bias Evaluation on Data and Algorithms for Affect Recognition in Faces

Nowadays, Artificial Intelligence (AI) algorithms show a strong performance for many use cases, making them desirable for real-world scenarios where the algorithms provide high-impact decisions. However, one major drawback of AI algorithms is their susceptibility to bias and resulting unfairness. This has a huge influence for their application, as they have a higher failure rate for certain subgroups. In this paper, we focus on the field of affective computing and particularly on the detection of bias for facial expressions. Depending on the deployment scenario, bias in facial expression models can have a disadvantageous impact and it is therefore essential to evaluate the bias and limitations of the model. In order to analyze the metadata distribution in affective computing datasets, we annotate several benchmark training datasets, containing both Action Units and categorical emotions, with age, gender, ethnicity, glasses, and beards. We show that there is a significantly skewed distribution, particularly for ethnicity and age. Based on this metadata annotation, we evaluate two trained state-of-the-art affective computing algorithms. Our evaluation shows that the strongest bias is in age, with the best performance for persons under 34 and a sharp decrease for older persons. Furthermore, we see an ethnicity bias with varying direction depending on the algorithm, a slight gender bias and worse performance for facial parts occluded by glasses

Female, white, 27? : Bias Evaluation on Data and Algorithms for Affect Recognition in Faces

Nowadays, Artificial Intelligence (AI) algorithms show a strong performance for many use cases, making them desirable for real-world scenarios where the algorithms provide high-impact decisions. However, one major drawback of AI algorithms is their susceptibility to bias and resulting unfairness. This has a huge influence for their application, as they have a higher failure rate for certain subgroups. In this paper, we focus on the field of affective computing and particularly on the detection of bias for facial expressions. Depending on the deployment scenario, bias in facial expression models can have a disadvantageous impact and it is therefore essential to evaluate the bias and limitations of the model. In order to analyze the metadata distribution in affective computing datasets, we annotate several benchmark training datasets, containing both Action Units and categorical emotions, with age, gender, ethnicity, glasses, and beards. We show that there is a significantly skewed distribution, particularly for ethnicity and age. Based on this metadata annotation, we evaluate two trained state-of-the-art affective computing algorithms. Our evaluation shows that the strongest bias is in age, with the best performance for persons under 34 and a sharp decrease for older persons. Furthermore, we see an ethnicity bias with varying direction depending on the algorithm, a slight gender bias and worse performance for facial parts occluded by glasses

Deriving Temporal Prototypes from Saliency Map Clusters for the Analysis of Deep-Learning-based Facial Action Unit Classification

Reliably determining the emotional state of a person is a difficult task for both humans as well as machines. Automatic detection and evaluation of facial expressions is particularly important if people are unable to express their emotional state themselves, for example due to cognitive impairments. Identifying the presence of Action Units in a human’s face is a psychologically validated approach of quantifying which emotion is expressed. To automate the detection process of Action Units Neural Networks have been trained. However, the black-box nature of Deep Neural Networks provides no insight on the relevant features identified during the decision process. Approaches of Explainable Artificial Intelligence have to be applied to provide an explanation why the network came to a certain conclusion. In this work "Layer-Wise Relevance Propagation" (LRP) in combination with the meta analysis approach "Spectral Relevance Analysis" (SpRAy) is used to derive temporal prototypes from predictions in video sequences. Temporal prototypes provide an aggregated view on the prediction of the network by grouping together similar frames by considering relevance. Additionally, a specific visualization method for temporal prototypes is presented that highlights the most relevant areas for a prediction of an Action Unit. A quantitative evaluation of our approach shows that temporal prototypes aggregate temporal information well. The proposed method can be used to generate concise visual explanations for a sequence of interpretable saliency maps. Based on the above, this work shall provide the foundation for a new temporal analysis method as well as an explanation approach that is supposed to help researchers and experts to gain a deeper understanding of how the underlying network decides which Action Units are active in a particular emotional state

Machine-based emotion-assessment in waiting rooms – a feasibility and acceptance study

Background: Due to an aging society and changing health behaviors, emergency room crowding has become a major problem in western health care systems. Empowering patients and health care workers to assess necessary and relevant information is critical to streamline clinical workflows. Health kiosks, designed for services like self-check-in or (ideally contactless) health self-assessment may be instrumental in solving this issue. Based on the collected data, automated workflows such as flagging critical patients, inducing specific diagnostics or early symptomatic treatment could be implemented. Objective: Using an AI-supported software, which visually analyzes and categorizes facial expressions, the emotional status of hemato-oncologic patients in a German oncology outpatient clinic was examined. Additionally a survey was conducted, evaluating the acceptance of such a self-assessment solution. Results: 98% of the participants were not stressed by the real-time emotion analysis. However, the current set of registered emotion categories was found to be only partially sufficient to adequately describe the emotional status of the patients. More importantly, 88% of the participants found such a system to be meaningful. Also, 84% of the participants agreed that such a self-analysis could be of potential assistance. No relevant generation- or gender-specific differences could be observed. Discussion: Automated analysis of patients’ emotional status can be a first step toward a more comprehensive assessment of the respective health status. Patients, in particular the elderly, approve to the vision and development of such a system. Next steps are a further improvement of the AI-based emotion recognition software with respect to more emotional states as well as the definition, inclusion and ideally contactless acquisition of physical biomarkers (as e.g. heart rate or respiratory rate) determining physical and mental well-being

Constitutive nuclear factor-kappa B activity preserves homeostasis of quiescent mature lymphocytes and granulocytes by controlling the expression of distinct Bcl-2 family proteins

Constitutive nuclear factor kappaB (NFkappaB) activity protects quiescent mature Immune cells from spontaneous apoptosis. Here, we examined whether NF-kappaB exerts its antiapoptotic function in these cells through the control of Bcl-2 family proteins. Specific pharmacologic inhibitors of NF-kappaB were used to achieve total NF-kappaB inactivation In quiescent human blood lymphocytes, granulocytes, and monocytes. NF-kappaB inhibition induced drastic lymphocyte and granulocyte apoptosis, but only moderate monocyte apoptosis. T- and B-cell apoptosis was slow and associated with a gradual down-regulation of the prosurvival Bcl-2 family proteins Bcl-X-L and BcI-2, respectively. By contrast, granulocyte apoptosis was fast and accompanied by a rapid cellular accumulation of Bcl-x(s), the proapoptotic Bcl-x isoform that is generated from alternative splicing of the bcl-x pre-mRNA. Finally, antisense bci-x(L) and bcl-2 knockdown in T and B cells, respectively, and induction of Bcl-xs expression in granulocytes through antisense oligonucleotide-mediated redirection of bcl-x pre-mRNA splicing were sufficient to induce significant apoptosis in these cells. Taken together, these results reveal that basal NF-kappaB activity preserves homeostasis of quiescent mature lymphocytes and granulocytes through regulation of distinct members of the Bcl-2 family. This study sheds light on the constitutive mechanisms by which NF-kappaB maintains defense integrity. (C) 2002 by The American Society of Hematology