Multimodal Emotion Recognition Model using Physiological Signals

As an important field of research in Human-Machine Interactions, emotion recognition based on physiological signals has become research hotspots. Motivated by the outstanding performance of deep learning approaches in recognition tasks, we proposed a Multimodal Emotion Recognition Model that consists of a 3D convolutional neural network model, a 1D convolutional neural network model and a biologically inspired multimodal fusion model which integrates multimodal information on the decision level for emotion recognition. We use this model to classify four emotional regions from the arousal valence plane, i.e., low arousal and low valence (LALV), high arousal and low valence (HALV), low arousal and high valence (LAHV) and high arousal and high valence (HAHV) in the DEAP and AMIGOS dataset. The 3D CNN model and 1D CNN model are used for emotion recognition based on electroencephalogram (EEG) signals and peripheral physiological signals respectively, and get the accuracy of 93.53% and 95.86% with the original EEG signals in these two datasets. Compared with the single-modal recognition, the multimodal fusion model improves the accuracy of emotion recognition by 5% ~ 25%, and the fusion result of EEG signals (decomposed into four frequency bands) and peripheral physiological signals get the accuracy of 95.77%, 97.27% and 91.07%, 99.74% in these two datasets respectively. Integrated EEG signals and peripheral physiological signals, this model could reach the highest accuracy about 99% in both datasets which shows that our proposed method demonstrates certain advantages in solving the emotion recognition tasks.Comment: 10 pages, 10 figures, 6 table

EMOTION RECOGNITION BASED ON VARIOUS PHYSIOLOGICAL SIGNALS - A REVIEW

Emotion recognition is one of the biggest challenges in human-human and human-computer interaction. There are various approaches to recognize emotions like facial expression, audio signals, body poses, and gestures etc. Physiological signals play vital role in emotion recognition as they are not controllable and are of immediate response type. In this paper, we discuss the research done on emotion recognition using skin conductance, skin temperature, electrocardiogram (ECG), electromyography (EMG), and electroencephalogram (EEG) signals. Altogether, the same methodology has been adopted for emotion recognition techniques based upon various physiological signals. After survey, it is understood that none of these methods are fully efficient standalone but the efficiency can be improved by using combination of physiological signals. The study of this paper provides an insight on the current state of research and challenges faced during emotion recognition using physiological signals, so that research can be advanced for better recognition

Multi-scale Entropy and Multiclass Fisher’s Linear Discriminant for Emotion Recognition Based on Multimodal Signal

Emotion recognition using physiological signals has been a special topic frequently discussed by researchers and practitioners in the past decade. However, the use of SpO2 and Pulse rate signals for emotion recognitionisvery limited and the results still showed low accuracy. It is due to the low complexity of SpO2 and Pulse rate signals characteristics. Therefore, this study proposes a Multiscale Entropy and Multiclass Fisher’s Linear Discriminant Analysis for feature extraction and dimensional reduction of these physiological signals for improving emotion recognition accuracy in elders.  In this study, the dimensional reduction process was grouped into three experimental schemes, namely a dimensional reduction using only SpO2 signals, pulse rate signals, and multimodal signals (a combination feature vectors of SpO2 and Pulse rate signals). The three schemes were then classified into three emotion classes (happy, sad, and angry emotions) using Support Vector Machine and Linear Discriminant Analysis Methods. The results showed that Support Vector Machine with the third scheme achieved optimal performance with an accuracy score of 95.24%. This result showed a significant increase of more than 22%from the previous works

Cross-Subject Emotion Recognition with Sparsely-Labeled Peripheral Physiological Data Using SHAP-Explained Tree Ensembles

There are still many challenges of emotion recognition using physiological data despite the substantial progress made recently. In this paper, we attempted to address two major challenges. First, in order to deal with the sparsely-labeled physiological data, we first decomposed the raw physiological data using signal spectrum analysis, based on which we extracted both complexity and energy features. Such a procedure helped reduce noise and improve feature extraction effectiveness. Second, in order to improve the explainability of the machine learning models in emotion recognition with physiological data, we proposed Light Gradient Boosting Machine (LightGBM) and SHapley Additive exPlanations (SHAP) for emotion prediction and model explanation, respectively. The LightGBM model outperformed the eXtreme Gradient Boosting (XGBoost) model on the public Database for Emotion Analysis using Physiological signals (DEAP) with f1-scores of 0.814, 0.823, and 0.860 for binary classification of valence, arousal, and liking, respectively, with cross-subject validation using eight peripheral physiological signals. Furthermore, the SHAP model was able to identify the most important features in emotion recognition, and revealed the relationships between the predictor variables and the response variables in terms of their main effects and interaction effects. Therefore, the results of the proposed model not only had good performance using peripheral physiological data, but also gave more insights into the underlying mechanisms in recognizing emotions

Multi-modal fusion methods for robust emotion recognition using body-worn physiological sensors in mobile environments

High-accuracy physiological emotion recognition typically requires participants to wear or attach obtrusive sensors (e.g., Electroencephalograph). To achieve precise emotion recognition using only wearable body-worn physiological sensors, my doctoral work focuses on researching and developing a robust sensor fusion system among different physiological sensors. Developing such fusion system has three problems: 1) how to pre-process signals with different temporal characteristics and noise models, 2) how to train the fusion system with limited labeled data and 3) how to fuse multiple signals with inaccurate and inexact ground truth. To overcome these challenges, I plan to explore semi-supervised, weakly supervised and unsupervised machine learning methods to obtain precise emotion recognition in mobile environments. By developing such techniques, we can measure the user engagement with larger amounts of participants and apply the emotion recognition techniques in a variety of scenarios such as mobile video watching and online education

A study of physiological signals-based emotion recognition systems.

The use of physiological signals is relatively recent development in human emotion recognition. Interest in this field has been motivated by the unbiased nature of such signals, which are generated autonomously from the central nervous system. Generally, these signals can be collected from the cardiovascular system, respiratory system, electrodermal activities, muscular system and brain activities. This paper presents an overview of emotion recognition using physiological signals. The main components of a physiological signals-based emotion recognition system are explained, including discussion regarding the concepts and problems about the various stages involved in its framework

Multiple Instance Learning for Emotion Recognition using Physiological Signals

The problem of continuous emotion recognition has been the subject of several studies. The proposed affective computing approaches employ sequential machine learning algorithms for improving the classification stage, accounting for the time ambiguity of emotional responses. Modeling and predicting the affective state over time is not a trivial problem because continuous data labeling is costly and not always feasible. This is a crucial issue in real-life applications, where data labeling is sparse and possibly captures only the most important events rather than the typical continuous subtle affective changes that occur. In this work, we introduce a framework from the machine learning literature called Multiple Instance Learning, which is able to model time intervals by capturing the presence or absence of relevant states, without the need to label the affective responses continuously (as required by standard sequential learning approaches). This choice offers a viable and natural solution for learning in a weakly supervised setting, taking into account the ambiguity of affective responses. We demonstrate the reliability of the proposed approach in a gold-standard scenario and towards real-world usage by employing an existing dataset (DEAP) and a purposely built one (Consumer). We also outline the advantages of this method with respect to standard supervised machine learning algorithms

Classification of Physiological Signals for Emotion Recognition using IoT

Emotion recognition gains huge popularity now a days. Physiological signals provides an appropriate way to detect human emotion with the help of IoT. In this paper, a novel system is proposed which is capable of determining the emotional status using physiological parameters, including design specification and software implementation of the system. This system may have a vivid use in medicine (especially for emotionally challenged people), smart home etc. Various Physiological parameters to be measured includes, heart rate (HR), galvanic skin response (GSR), skin temperature etc. To construct the proposed system the measured physiological parameters were feed to the neural networks which further classify the data in various emotional states, mainly in anger, happy, sad, joy. This work recognized the correlation between human emotions and change in physiological parameters with respect to their emotion

On-body Sensing and Signal Analysis for User Experience Recognition in Human-Machine Interaction

—In this paper, a new algorithm is proposed for recognition of user experience through emotion detection using physiological signals, for application in human-machine interaction. The algorithm recognizes user’s emotion quality and intensity in a two dimensional emotion space continuously. The continuous recognition of the user’s emotion during human-machine interaction will enable the machine to adapt its activity based on the user’s emotion in a real-time manner, thus improving user experience. The emotion model underlying the proposed algorithm is one of the most recent emotion models, which models emotion’s intensity and quality in a continuous two-dimensional space of valance and arousal axes. Using only two physiological signals, which are correlated to the valance and arousal axes of the emotion space, is among the contributions of this paper. Prediction of emotion through physiological signals has the advantage of elimination of social masking and making the prediction more reliable. The key advantage of the proposed algorithm over other algorithms presented to date is the use of the least number of modalities (only two physiological signals) to predict the quality and intensity of emotion continuously in time, and using the most recent widely accepted emotion model

Emotion Recognition with Pre-Trained Transformers Using Multimodal Signals

In this paper, we address the problem of multimodal emotion recognition from multiple physiological signals. We demonstrate that a Transformer-based approach is suitable for this task. In addition, we present how such models may be pretrained in a multimodal scenario to improve emotion recognition performances. We evaluate the benefits of using multimodal inputs and pre-training with our approach on a state-ofthe-art dataset