Emotion Detection Using Noninvasive Low Cost Sensors

Emotion recognition from biometrics is relevant to a wide range of application domains, including healthcare. Existing approaches usually adopt multi-electrodes sensors that could be expensive or uncomfortable to be used in real-life situations. In this study, we investigate whether we can reliably recognize high vs. low emotional valence and arousal by relying on noninvasive low cost EEG, EMG, and GSR sensors. We report the results of an empirical study involving 19 subjects. We achieve state-of-the- art classification performance for both valence and arousal even in a cross-subject classification setting, which eliminates the need for individual training and tuning of classification models.Comment: To appear in Proceedings of ACII 2017, the Seventh International Conference on Affective Computing and Intelligent Interaction, San Antonio, TX, USA, Oct. 23-26, 201

Automated Semantic Understanding of Human Emotions in Writing and Speech

Affective Human Computer Interaction (A-HCI) will be critical for the success of new technologies that will prevalent in the 21st century. If cell phones and the internet are any indication, there will be continued rapid development of automated assistive systems that help humans to live better, more productive lives. These will not be just passive systems such as cell phones, but active assistive systems like robot aides in use in hospitals, homes, entertainment room, office, and other work environments. Such systems will need to be able to properly deduce human emotional state before they determine how to best interact with people. This dissertation explores and extends the body of knowledge related to Affective HCI. New semantic methodologies are developed and studied for reliable and accurate detection of human emotional states and magnitudes in written and spoken speech; and for mapping emotional states and magnitudes to 3-D facial expression outputs. The automatic detection of affect in language is based on natural language processing and machine learning approaches. Two affect corpora were developed to perform this analysis. Emotion classification is performed at the sentence level using a step-wise approach which incorporates sentiment flow and sentiment composition features. For emotion magnitude estimation, a regression model was developed to predict evolving emotional magnitude of actors. Emotional magnitudes at any point during a story or conversation are determined by 1) previous emotional state magnitude; 2) new text and speech inputs that might act upon that state; and 3) information about the context the actors are in. Acoustic features are also used to capture additional information from the speech signal. Evaluation of the automatic understanding of affect is performed by testing the model on a testing subset of the newly extended corpus. To visualize actor emotions as perceived by the system, a methodology was also developed to map predicted emotion class magnitudes to 3-D facial parameters using vertex-level mesh morphing. The developed sentence level emotion state detection approach achieved classification accuracies as high as 71% for the neutral vs. emotion classification task in a test corpus of children’s stories. After class re-sampling, the results of the step-wise classification methodology on a test sub-set of a medical drama corpus achieved accuracies in the 56% to 84% range for each emotion class and polarity. For emotion magnitude prediction, the developed recurrent (prior-state feedback) regression model using both text-based and acoustic based features achieved correlation coefficients in the range of 0.69 to 0.80. This prediction function was modeled using a non-linear approach based on Support Vector Regression (SVR) and performed better than other approaches based on Linear Regression or Artificial Neural Networks

desarrollo de una metodología para el reconocimiento de emociones basado en un enfoque multimodal mediante la extracción y selección discriminante de características

Las emociones pueden ser entendidas como respuestas automáticas de diferentes sistemas biológicos del cuerpo humano ante un determinado estímulo interno o externo. A partir de diferentes estudios, se han propuesto esquemas de clasificación de emociones, siendo los más comúnmente utilizados los espacios discretos, donde a cada emoción se le asigna una determinada etiqueta y espacios dimensionales en los cuáles una emoción se determina como una combinación de dos índices que están relacionados con una interpretación, medición y cuantificación más precisa de las emociones. Basado en un determinado espacio de clasificación, se ha buscado a través del desarrollo de diferentes trabajos de investigación, darle la capacidad a una máquina para detectar y reconocer las emociones del ser humano, teniendo en cuenta que entender y diferenciar las emociones expresadas por las personas es una tarea compleja aún hasta para los mismos seres humanos. Debido a que las emociones producen respuestas de diferentes sistemas biológicos del cuerpo humano, es intuitivo pensar en el desarrollo de sistemas que incorporen la información de dichos sistemas biológicos para realizar un reconocimiento efectivo de las emociones. El enfoque multimodal para el reconocimiento de emociones comprende entonces aquellas metodologías que combinan características de diferentes señales obtenidas del cuerpo humano ante un determinado estímulo que causa un estado emocional específico. Las señales que se utilizan comúnmente en el desarrollo de sistemas multimodales son las información de video, de audio, a partir del Electroencefalograma (EEG) y diversas señales fisiológicas como el ritmo cardiaco, la respuesta galvánica de la piel, la señal respiratoria y la de temperatura.En el presente trabajo, se propone una metodología de selección de características dentro de un enfoque multimodal de clasificación de estados emocionales en un espacio dimensional, a partir de modelos discriminativos, mediante la implementación de dos algoritmos de selección conocidos como Eliminación Recursiva de Características (RFE) y Eliminación de características basado en la Maximización del Margen (MFE). Con estos métodos de selección se busca reducir un conjunto original de características a partir de los análisis de la linealidad y no linealidad de las diferentes señales. El análisis no lineal está basado en ciertas métricas obtenidas a partir de una técnica conocida como gráficos de recurrencia que ha conseguido dentro del estado del arte mejorar los resultados de clasificación. Se utilizan dos bases de datos reconocidas dentro del estado del arte para la extracción y selección de características dentro del esquema propuesto.Emotions can be understood as automatic responses from different biological systems of the human body to a particular internal or external stimulus. From different studies, there have been proposed emotions classification schemes as the discrete space, where every emotion is assigned a particular tag and dimensional spaces in which an emotion is determined as a combination of two indices which are related to a more accurate measurement and quantification of emotions. Based on a given classification space there has been developed different research projects in order to gave a machine the ability to detect and recognize the emotions of humans, considering that understand and differentiate the emotions expressed by people is still a complex task even for human beings themselves. Because emotions produce different biological responses of the human body, it is intuitive to think of the development of systems that incorporate biological information from these systems to make effective recognition of emotions. The multimodal approach for emotion recognition then comprises those methodologies that combine characteristics of different signals obtained from the human body to a given stimulus that causes a specific emotional state. The signals that are commonly used in the development of multimodal systems are the video information, the audio signal, the electroencephalogram (EEG) and various physiological signals such as heart rate, galvanic skin response , the respiratory signal and the temperature. In this paper we propose a feature selection methodology within a multimodal approach for the classification of emotional states in a dimensional space, based on discriminative models by implementing two selection algorithms known as Recursive Feature Elimination (RFE) and margin-maximizing feature elimination (MFE). With these approaches for feature reduce the original set of features is reduced from the analysis of the linearity and nonlinearity of the different signals. Nonlinear analysis is based on certain metrics obtained from a technique known as recurrence plots (RP) that have showed to improve the classification results in some state of art works. We use two databases recognized within the state of the art for the extraction and selection of features within the proposed scheme

Affective and Implicit Tagging using Facial Expressions and Electroencephalography.

PhDRecent years have seen an explosion of user-generated, untagged multimedia data, generating a need for efficient search and retrieval of this data. The predominant method for content-based tagging is through manual annotation. Consequently, automatic tagging is currently the subject of intensive research. However, it is clear that the process will not be fully automated in the foreseeable future. We propose to involve the user and investigate methods for implicit tagging, wherein users' responses to the multimedia content are analysed in order to generate descriptive tags. We approach this problem through the modalities of facial expressions and EEG signals. We investigate tag validation and affective tagging using EEG signals. The former relies on the detection of event-related potentials triggered in response to the presentation of invalid tags alongside multimedia material. We demonstrate significant differences in users' EEG responses for valid versus invalid tags, and present results towards single-trial classification. For affective tagging, we propose methodologies to map EEG signals onto the valence-arousal space and perform both binary classification as well as regression into this space. We apply these methods in a real-time affective recommendation system. We also investigate the analysis of facial expressions for implicit tagging. This relies on a dynamic texture representation using non-rigid registration that we first evaluate on the problem of facial action unit recognition. We present results on well-known datasets (with both posed and spontaneous expressions) comparable to the state of the art in the field. Finally, we present a multi-modal approach that fuses both modalities for affective tagging. We perform classification in the valence-arousal space based on these modalities and present results for both feature-level and decision-level fusion. We demonstrate improvement in the results when using both modalities, suggesting the modalities contain complementary information

Developing Driving Behaviour Models Incorporating the Effects of Stress

Driving is a complex task and several factors influence drivers’ decisions and performance including traffic conditions, attributes of vehicles, network and environmental characteristics, and last but not least characteristics of the drivers themselves. in an effort to better explain and represent driving behaviour, several driving behaviour models have been suggested over the years. In the existing literature, there are two main streams of driving behaviour models that can be found. The first is approaching driving behaviour from a human factors and cognitive perspective while the second is engineering-based. Driving behaviour models of the latter category are mathematical representations of drivers’ behaviour at the individual level, mostly focussing on acceleration/deceleration, lane-change and gap-acceptance decisions. Many of these factors are captured by existing driving behaviour models used in microscopic simulation tools. However, while the vast majority of existing models is approximating driving behaviour, primarily focusing on the effects of traffic conditions, little attention has been given to the impact of drivers’ characteristics. The aim of the current thesis is to investigate the effects of stress on driving behaviour and quantify its impact using an econometric modelling framework. This main research question emerged as a result of a widely acknowledged research gap in existing engineering-based driving behaviour models related to the incorporation of human factors and drivers’ characteristics within the model specification. The research was based on data collected using the University of Leeds Driving Simulator. Two main scenarios were presented to participants, while they were also deliberately subjected to stress induced by time pressure and various scenarios. At the same time, stress levels were measured via physiological indicators. Sociodemographic and trait data was also collected in the form of surveys. The data has been initially analysed for each main scenario and several statistics are extracted. The results show a clear effect of time pressure in favour of speeding, however relations related to physiological responses are not always clear. Moreover, two driving behaviour models are developed, a gap-acceptance and a car-following model. In the former model, increase in physiological responses is related to higher probability of accepting a gap and time pressure has a positive effect of gap-acceptance probability as well. In the car-following model, stress is associated with increased acceleration and potentially a more aggressive driving style. The aforementioned analysis is based on data collected in a driving simulator. Given the potential differences in driving behaviour between real and simulated driving, the transferability of a model based on the latter data to field traffic setting is also investigated. Results indicate significant differences in parameters estimated from a video and the simulator dataset, however these differences can be significantly reduced after applying parameter updating techniques. The findings in this thesis show that stress and drivers’ characteristics can influence driving behaviour and thus should be considered in the driving behaviour models for microscopic simulation applications. However, for real life applications, it is suggested that the extent of these effects should be treated with caution and ideally rescaled based on real traffic observations