Explainable automated recognition of emotional states from canine facial expressions: the case of positive anticipation and frustration.

In animal research, automation of affective states recognition has so far mainly addressed pain in a few species. Emotional states remain uncharted territories, especially in dogs, due to the complexity of their facial morphology and expressions. This study contributes to fill this gap in two aspects. First, it is the first to address dog emotional states using a dataset obtained in a controlled experimental setting, including videos from (n = 29) Labrador Retrievers assumed to be in two experimentally induced emotional states: negative (frustration) and positive (anticipation). The dogs' facial expressions were measured using the Dogs Facial Action Coding System (DogFACS). Two different approaches are compared in relation to our aim: (1) a DogFACS-based approach with a two-step pipeline consisting of (i) a DogFACS variable detector and (ii) a positive/negative state Decision Tree classifier; (2) An approach using deep learning techniques with no intermediate representation. The approaches reach accuracy of above 71% and 89%, respectively, with the deep learning approach performing better. Secondly, this study is also the first to study explainability of AI models in the context of emotion in animals. The DogFACS-based approach provides decision trees, that is a mathematical representation which reflects previous findings by human experts in relation to certain facial expressions (DogFACS variables) being correlates of specific emotional states. The deep learning approach offers a different, visual form of explainability in the form of heatmaps reflecting regions of focus of the network's attention, which in some cases show focus clearly related to the nature of particular DogFACS variables. These heatmaps may hold the key to novel insights on the sensitivity of the network to nuanced pixel patterns reflecting information invisible to the human eye

The Explanation Matters: Enhancing AI Adoption in Human Resource Management

Artificial intelligence (AI) has ubiquitous applications in companies, permeating multiple business divisions like human resource management (HRM). Yet, in these high-stakes domains where transparency and interpretability of results are of utmost importance, the black-box characteristic of AI is even more of a threat to AI adoption. Hence, explainable AI (XAI), which is regular AI equipped with or complemented by techniques to explain it, comes in. We present a systematic literature review of n=62 XAI in HRM papers. Further, we conducted an experiment among a German sample (n=108) of HRM personnel regarding a turnover prediction task with or without (X)AI-support. We find that AI-support leads to better task performance, self-assessment accuracy and response characteristics toward the AI, and XAI, i.e., transparent models allow for more accurate self-assessment of one’s performance. Future studies could enhance our research by employing local explanation techniques on real-world data with a larger and international sample

First impressions: A survey on vision-based apparent personality trait analysis

© 2019 IEEE. Personal use of this material is permitted. Permission from IEEE must be obtained for all other uses, in any current or future media, including reprinting/republishing this material for advertising or promotional purposes,creating new collective works, for resale or redistribution to servers or lists, or reuse of any copyrighted component of this work in other works.Personality analysis has been widely studied in psychology, neuropsychology, and signal processing fields, among others. From the past few years, it also became an attractive research area in visual computing. From the computational point of view, by far speech and text have been the most considered cues of information for analyzing personality. However, recently there has been an increasing interest from the computer vision community in analyzing personality from visual data. Recent computer vision approaches are able to accurately analyze human faces, body postures and behaviors, and use these information to infer apparent personality traits. Because of the overwhelming research interest in this topic, and of the potential impact that this sort of methods could have in society, we present in this paper an up-to-date review of existing vision-based approaches for apparent personality trait recognition. We describe seminal and cutting edge works on the subject, discussing and comparing their distinctive features and limitations. Future venues of research in the field are identified and discussed. Furthermore, aspects on the subjectivity in data labeling/evaluation, as well as current datasets and challenges organized to push the research on the field are reviewed.Peer ReviewedPostprint (author's final draft

Integrating audio and visual modalities for multimodal personality trait recognition via hybrid deep learning

Recently, personality trait recognition, which aims to identify people’s first impression behavior data and analyze people’s psychological characteristics, has been an interesting and active topic in psychology, affective neuroscience and artificial intelligence. To effectively take advantage of spatio-temporal cues in audio-visual modalities, this paper proposes a new method of multimodal personality trait recognition integrating audio-visual modalities based on a hybrid deep learning framework, which is comprised of convolutional neural networks (CNN), bi-directional long short-term memory network (Bi-LSTM), and the Transformer network. In particular, a pre-trained deep audio CNN model is used to learn high-level segment-level audio features. A pre-trained deep face CNN model is leveraged to separately learn high-level frame-level global scene features and local face features from each frame in dynamic video sequences. Then, these extracted deep audio-visual features are fed into a Bi-LSTM and a Transformer network to individually capture long-term temporal dependency, thereby producing the final global audio and visual features for downstream tasks. Finally, a linear regression method is employed to conduct the single audio-based and visual-based personality trait recognition tasks, followed by a decision-level fusion strategy used for producing the final Big-Five personality scores and interview scores. Experimental results on the public ChaLearn First Impression-V2 personality dataset show the effectiveness of our method, outperforming other used methods

Explainable automated recognition of emotional states from canine facial expressions: the case of positive anticipation and frustration

In animal research, automation of affective states recognition has so far mainly addressed pain in a few species. Emotional states remain uncharted territories, especially in dogs, due to the complexity of their facial morphology and expressions. This study contributes to fill this gap in two aspects. First, it is the first to address dog emotional states using a dataset obtained in a controlled experimental setting, including videos from (n = 29) Labrador Retrievers assumed to be in two experimentally induced emotional states: negative (frustration) and positive (anticipation). The dogs’ facial expressions were measured using the Dogs Facial Action Coding System (DogFACS). Two different approaches are compared in relation to our aim: (1) a DogFACS-based approach with a two-step pipeline consisting of (i) a DogFACS variable detector and (ii) a positive/negative state Decision Tree classifier; (2) An approach using deep learning techniques with no intermediate representation. The approaches reach accuracy of above 71% and 89%, respectively, with the deep learning approach performing better. Secondly, this study is also the first to study explainability of AI models in the context of emotion in animals. The DogFACS-based approach provides decision trees, that is a mathematical representation which reflects previous findings by human experts in relation to certain facial expressions (DogFACS variables) being correlates of specific emotional states. The deep learning approach offers a different, visual form of explainability in the form of heatmaps reflecting regions of focus of the network’s attention, which in some cases show focus clearly related to the nature of particular DogFACS variables. These heatmaps may hold the key to novel insights on the sensitivity of the network to nuanced pixel patterns reflecting information invisible to the human eye

Comparing Approaches for Explaining DNN-Based Facial Expression Classifications

Classifying facial expressions is a vital part of developing systems capable of aptly interacting with users. In this field, the use of deep-learning models has become the standard. However, the inner workings of these models are unintelligible, which is an important issue when deploying them to high-stakes environments. Recent efforts to generate explanations for emotion classification systems have been focused on this type of models. In this work, an alternative way of explaining the decisions of a more conventional model based on geometric features is presented. We develop a geometric-features-based deep neural network (DNN) and a convolutional neural network (CNN). Ensuring a sufficient level of predictive accuracy, we analyze explainability using both objective quantitative criteria and a user study. Results indicate that the fidelity and accuracy scores of the explanations approximate the DNN well. From the performed user study, it becomes clear that the explanations increase the understanding of the DNN and that they are preferred over the explanations for the CNN, which are more commonly used. All scripts used in the study are publicly available

Explainable Strategic Optimisation of Grand Scale Problems

Explainable Strategic Optimisation of grand scale problems aims to identify solutions that provide long term planning advantages to problems that cannot undergo traditional optimisation techniques due to their level of complexity. Usually, optimisation tasks focus on improving a limited number of objectives in the pursuit of obviously immediate target. However, this methodology, when applied to grand scale problems is found to be insufficient; a major reason for this is the inherent complexities typical of problems such as utility optimisation and massive logistical operations. One approach to these problems is Generational Expansion Planning that typically addresses long-term planning of country/county-wide utility problems. This thesis draws influence from the Generational Expansion Planning field; a significant field in relation to this work as it typically focuses on large scale optimisation problems. Problems such as the improvement and maintenance of national utility operations. However, this thesis takes a novel approach that places empathises on an abstract strategic planning method that concerns itself with the extraction of design insights that can guide an experts understanding of an unrelentingly complex problem. The proposed system was developed with data from British Telecom (BT) and was developed within their organisation in which its deployment is being planned. The techniques behind the proposed systems presented in this thesis are shown to improve the popular many-objective Non-Dominated Sorting Genetic Algorithm II in a series of experiments in which the improved Type-2 dominance method outperformed the traditional dominance method by 59%. Several component parts are brought together within this thesis so that the unique optimisation of varied regions that exist inside the United Kingdom’s Access Network can be explored. The proposed system places great import on the interpretability of the system and the solutions that it produces. As such, an Explainable Artificial Intelligent (XAI) system has been implemented in the hope that with greater interpretability, AI systems will be able to provide solutions with greater context, nuance, and confidence, particularly when the decision of an AI model has a direct impact on a person or business. This thesis will explore the related material and will explore the proposed framework; which brings together a multitude of technologies, such as, novel fuzzy many-objective optimisation, fuzzy explainable artificial intelligence, and strategic analysis. These technologies have been approached and combined in order to develop a novel system capable of dealing with complex grand scale problems, which traditionally are tackled as piecemeal optimisation problems. The proposed systems were shown to improve the optimisation of focused scenarios; in these experiments the proposed system was able to provide solutions for the optimisation of telecommunication networks that outperformed the current methodology for the planning/upgrading of the access network. The proposed systems were tested on rural, mixed, and urban regions of a simulated United Kingdom; it was observed that when the proposed systems were used the network solutions produced were 51.99% cheaper for rural regions, in which a combination of technologies were used as opposed to only FTTP. It was also observed that solutions produced by the proposed system in mixed regions were 54.16% cheaper while still providing the customer broadband requirements. These results identify how an expansive system such as the novel system proposed in this thesis is able to provide sound business solutions to complex real-world problems that consists of an ever growing number of variables, constraints, and objectives. Additionally, the proposed systems are capable of producing greater understanding of design principles/choices in network solutions, which in turn provides BT and users with a greater level of trust in the solutions and the system. This is a major obstacle that must be overcome when the problem domain that is being considered is incredible vast, uncertain, and extremely vital to the success of a company. The results of this thesis identify how the proposed systems can be developed and implemented to provide an insight into the planning and execution of an access network not required for decades to come. This is a significant change from the current reactive approach to a proactive approach that provides insight into the ever changing variables and needs of the network. The proposed systems are able to instil the confidence that allow a more thoughtful approach to be taken that is beneficial to both company and customer