Multimodal Emotion Recognition Model using Physiological Signals

As an important field of research in Human-Machine Interactions, emotion recognition based on physiological signals has become research hotspots. Motivated by the outstanding performance of deep learning approaches in recognition tasks, we proposed a Multimodal Emotion Recognition Model that consists of a 3D convolutional neural network model, a 1D convolutional neural network model and a biologically inspired multimodal fusion model which integrates multimodal information on the decision level for emotion recognition. We use this model to classify four emotional regions from the arousal valence plane, i.e., low arousal and low valence (LALV), high arousal and low valence (HALV), low arousal and high valence (LAHV) and high arousal and high valence (HAHV) in the DEAP and AMIGOS dataset. The 3D CNN model and 1D CNN model are used for emotion recognition based on electroencephalogram (EEG) signals and peripheral physiological signals respectively, and get the accuracy of 93.53% and 95.86% with the original EEG signals in these two datasets. Compared with the single-modal recognition, the multimodal fusion model improves the accuracy of emotion recognition by 5% ~ 25%, and the fusion result of EEG signals (decomposed into four frequency bands) and peripheral physiological signals get the accuracy of 95.77%, 97.27% and 91.07%, 99.74% in these two datasets respectively. Integrated EEG signals and peripheral physiological signals, this model could reach the highest accuracy about 99% in both datasets which shows that our proposed method demonstrates certain advantages in solving the emotion recognition tasks.Comment: 10 pages, 10 figures, 6 table

Emotion Recognition from Electroencephalogram Signals based on Deep Neural Networks

Emotion recognition using deep learning methods through electroencephalogram (EEG) analysis has marked significant progress. Nevertheless, the complexities and time-intensive nature of EEG analysis present challenges. This study proposes an efficient EEG analysis method that foregoes feature extraction and sliding windows, instead employing one-dimensional Neural Networks for emotion classification. The analysis utilizes EEG signals from the Database for Emotion Analysis using Physiological Signals (DEAP) and focuses on thirteen EEG electrode positions closely associated with emotion changes. Three distinct Neural Models are explored for emotion classification: two Convolutional Neural Networks (CNN) and a combined approach using Convolutional Neural Networks and Long Short-Term Memory (CNN-LSTM). Additionally, two emotion labels are considered: four emotional ranges encompassing low arousal and low valence (LALV), low arousal and high valence (LAHV), high arousal and high valence (HAHV), and high arousal and low valence (HALV); and high valence (HV) and low valence (LV). Results demonstrate CNN_1 achieving an average accuracy of 97.7% for classifying four emotional ranges, CNN_2 with 97.1%, and CNN-LSTM reaching an impressive 99.5%. Notably, in classifying HV and LV labels, our methods attained remarkable accuracies of 100%, 98.8%, and 99.7% for CNN_1, CNN_2, and CNN-LSTM, respectively. The performance of our models surpasses that of previously reported studies, showcasing their potential as highly effective classifiers for emotion recognition using EEG signals

Using Deep Convolutional Neural Network for Emotion Detection on a Physiological Signals Dataset (AMIGOS)

Recommender systems have been based on context and content, and now the technological challenge of making personalized recommendations based on the user emotional state arises through physiological signals that are obtained from devices or sensors. This paper applies the deep learning approach using a deep convolutional neural network on a dataset of physiological signals (electrocardiogram and galvanic skin response), in this case, the AMIGOS dataset. The detection of emotions is done by correlating these physiological signals with the data of arousal and valence of this dataset, to classify the affective state of a person. In addition, an application for emotion recognition based on classic machine learning algorithms is proposed to extract the features of physiological signals in the domain of time, frequency, and non-linear. This application uses a convolutional neural network for the automatic feature extraction of the physiological signals, and through fully connected network layers, the emotion prediction is made. The experimental results on the AMIGOS dataset show that the method proposed in this paper achieves a better precision of the classification of the emotional states, in comparison with the originally obtained by the authors of this dataset.This research project is financed by theGovernment of Colombia, Colciencias and the Governorateof Boyac

EEG-based Deep Emotional Diagnosis: A Comparative Study

Emotion is an important part of people's daily life, particularly relevant to the mental health of people. Emotional diagnosis is closely related to the nervous system, which can well reflect people's mental conditions in response to the surrounding environment or the development of various neurodegenerative diseases. Emotion recognition can help the medical diagnosis of mental health. In recent years, emotion recognition based on EEG has attracted the attention of many researchers accompanying with the continuous development of artificial intelligence and brain computer interface technology. In this paper, we carried out a comparison on the performance of three deep learning techniques on EEG classification, including DNN, CNN and CNN-LSTM. DEAP data set was used in our experiments. EEG signals were transformed from time domain to frequency domain first, and then features are extracted to classify emotions. From our research, it shows these deep learning techniques can achieve good accuracy on emotional diagnosis

TACOformer:Token-channel compounded Cross Attention for Multimodal Emotion Recognition

Recently, emotion recognition based on physiological signals has emerged as a field with intensive research. The utilization of multi-modal, multi-channel physiological signals has significantly improved the performance of emotion recognition systems, due to their complementarity. However, effectively integrating emotion-related semantic information from different modalities and capturing inter-modal dependencies remains a challenging issue. Many existing multimodal fusion methods ignore either token-to-token or channel-to-channel correlations of multichannel signals from different modalities, which limits the classification capability of the models to some extent. In this paper, we propose a comprehensive perspective of multimodal fusion that integrates channel-level and token-level cross-modal interactions. Specifically, we introduce a unified cross attention module called Token-chAnnel COmpound (TACO) Cross Attention to perform multimodal fusion, which simultaneously models channel-level and token-level dependencies between modalities. Additionally, we propose a 2D position encoding method to preserve information about the spatial distribution of EEG signal channels, then we use two transformer encoders ahead of the fusion module to capture long-term temporal dependencies from the EEG signal and the peripheral physiological signal, respectively. Subject-independent experiments on emotional dataset DEAP and Dreamer demonstrate that the proposed model achieves state-of-the-art performance.Comment: Accepted by IJCAI 2023- AI4TS worksho

STILN: A Novel Spatial-Temporal Information Learning Network for EEG-based Emotion Recognition

The spatial correlations and the temporal contexts are indispensable in Electroencephalogram (EEG)-based emotion recognition. However, the learning of complex spatial correlations among several channels is a challenging problem. Besides, the temporal contexts learning is beneficial to emphasize the critical EEG frames because the subjects only reach the prospective emotion during part of stimuli. Hence, we propose a novel Spatial-Temporal Information Learning Network (STILN) to extract the discriminative features by capturing the spatial correlations and temporal contexts. Specifically, the generated 2D power topographic maps capture the dependencies among electrodes, and they are fed to the CNN-based spatial feature extraction network. Furthermore, Convolutional Block Attention Module (CBAM) recalibrates the weights of power topographic maps to emphasize the crucial brain regions and frequency bands. Meanwhile, Batch Normalizations (BNs) and Instance Normalizations (INs) are appropriately combined to relieve the individual differences. In the temporal contexts learning, we adopt the Bidirectional Long Short-Term Memory Network (Bi-LSTM) network to capture the dependencies among the EEG frames. To validate the effectiveness of the proposed method, subject-independent experiments are conducted on the public DEAP dataset. The proposed method has achieved the outstanding performance, and the accuracies of arousal and valence classification have reached 0.6831 and 0.6752 respectively