Discrete wavelet packet transform for electroencephalogram based valence-arousal emotion recognition

Electroencephalogram (EEG) based emotion recognition has received considerable attention as it is a non-invasive method of acquiring physiological signals from the brain and it could directly reflect emotional states. However, the challenging issues regarding EEG-based emotional state recognition is that it requires well-designed methods and algorithms to extract necessary features from the complex, chaotic, and multichannel EEG signal in order to achieve optimum classification performance. The aim of this study is to discover the feature extraction method and the combination of electrode channels that optimally implements EEG-based valencearousal emotion recognition. Based on this, two emotion recognition experiments were performed to classify human emotional states into high/low valence or high/low arousal. The first experiment was aimed to evaluate the performance of Discrete Wavelet Packet Transform (DWPT) as a feature extraction method. The second experiment was aimed at identifying the combination of electrode channels that optimally recognize emotions based on the valence-arousal model in EEG emotion recognition. In order to evaluate the results of this study, a benchmark EEG dataset was used to implement the emotion classification. In the first experiment, the entropy features of the theta, alpha, beta, and gamma bands through the 10 EEG channels Fp1, Fp2, F3, F4, T7, T8, P3, P4, O1, and O2 were extracted using DWPT and Radial Basis Function-Support Vector Machine (RBF-SVM) was used as the classifier. In the second experiment, the classification experiments were repeated using the 4 EEG frontal channels Fp1, Fp2, F3, and F4. The result of the first experiment showed that entropy features extracted using DWPT are better than bandpower features. While the result of the second classification experiment shows that the combination of the 4 frontal channels is more significant than the combination of the 10 channel

Deep learning framework for subject-independent emotion detection using wireless signals.

Emotion states recognition using wireless signals is an emerging area of research that has an impact on neuroscientific studies of human behaviour and well-being monitoring. Currently, standoff emotion detection is mostly reliant on the analysis of facial expressions and/or eye movements acquired from optical or video cameras. Meanwhile, although they have been widely accepted for recognizing human emotions from the multimodal data, machine learning approaches have been mostly restricted to subject dependent analyses which lack of generality. In this paper, we report an experimental study which collects heartbeat and breathing signals of 15 participants from radio frequency (RF) reflections off the body followed by novel noise filtering techniques. We propose a novel deep neural network (DNN) architecture based on the fusion of raw RF data and the processed RF signal for classifying and visualising various emotion states. The proposed model achieves high classification accuracy of 71.67% for independent subjects with 0.71, 0.72 and 0.71 precision, recall and F1-score values respectively. We have compared our results with those obtained from five different classical ML algorithms and it is established that deep learning offers a superior performance even with limited amount of raw RF and post processed time-sequence data. The deep learning model has also been validated by comparing our results with those from ECG signals. Our results indicate that using wireless signals for stand-by emotion state detection is a better alternative to other technologies with high accuracy and have much wider applications in future studies of behavioural sciences

Affective level design for a role-playing videogame evaluated by a brain\u2013computer interface and machine learning methods

Game science has become a research field, which attracts industry attention due to a worldwide rich sell-market. To understand the player experience, concepts like flow or boredom mental states require formalization and empirical investigation, taking advantage of the objective data that psychophysiological methods like electroencephalography (EEG) can provide. This work studies the affective ludology and shows two different game levels for Neverwinter Nights 2 developed with the aim to manipulate emotions; two sets of affective design guidelines are presented, with a rigorous formalization that considers the characteristics of role-playing genre and its specific gameplay. An empirical investigation with a brain\u2013computer interface headset has been conducted: by extracting numerical data features, machine learning techniques classify the different activities of the gaming sessions (task and events) to verify if their design differentiation coincides with the affective one. The observed results, also supported by subjective questionnaires data, confirm the goodness of the proposed guidelines, suggesting that this evaluation methodology could be extended to other evaluation tasks

Approaches, applications, and challenges in physiological emotion recognition — a tutorial overview

An automatic emotion recognition system can serve as a fundamental framework for various applications in daily life from monitoring emotional well-being to improving the quality of life through better emotion regulation. Understanding the process of emotion manifestation becomes crucial for building emotion recognition systems. An emotional experience results in changes not only in interpersonal behavior but also in physiological responses. Physiological signals are one of the most reliable means for recognizing emotions since individuals cannot consciously manipulate them for a long duration. These signals can be captured by medical-grade wearable devices, as well as commercial smart watches and smart bands. With the shift in research direction from laboratory to unrestricted daily life, commercial devices have been employed ubiquitously. However, this shift has introduced several challenges, such as low data quality, dependency on subjective self-reports, unlimited movement-related changes, and artifacts in physiological signals. This tutorial provides an overview of practical aspects of emotion recognition, such as experiment design, properties of different physiological modalities, existing datasets, suitable machine learning algorithms for physiological data, and several applications. It aims to provide the necessary psychological and physiological backgrounds through various emotion theories and the physiological manifestation of emotions, thereby laying a foundation for emotion recognition. Finally, the tutorial discusses open research directions and possible solutions

Classification of Affective Data to Evaluate the Level Design in a Role-Playing Videogame

This paper presents a novel approach to evaluate game level design strategies, applied to role playing games. Following a set of well defined guidelines, two game levels were designed for Neverwinter Nights 2 to manipulate particular emotions like boredom or flow, and tested by 13 subjects wearing a brain computer interface helmet. A set of features was extracted from the affective data logs and used to classify different parts of the gaming sessions, to verify the correspondence of the original level aims and the effective results on people emotions. The very interesting correlations observed, suggest that the technique is extensible to other similar evaluation tasks

Exploratory psychometric validation and efficacy assessment study of an agoraphobia treatment based on virtual reality serious games and biofeedback

Tese de mestrado integrado em Engenharia Biomédica e Biofísica (Sinais e Imagens Médicas), Universidade de Lisboa, Faculdade de Ciências, 2020Uma fobia é um tipo de perturbação ansiosa, definida por um medo persistente e excessivo em relação a um objeto ou situação, tendo um impacto bastante limitativo na vida do doente fóbico. Atualmente, as perturbações mentais são ainda vistas como tabu, e existe uma elevada incidência de pessoas que sofrem de fobias (~ 83M na União Europeia, e 32M no Estados Unidos da América). Os métodos atuais para tratamento de perturbações mentais baseiam-se sobretudo em psicoterapia e em farmacologia. Especificamente, o tratamento de perturbações fóbicas baseia-se em terapia por exposição in vivo. Esta técnica foca-se em alterar a resposta do doente ao objeto ou situação que é alvo de medo, através de exposição repetida ao mesmo. A título de exemplo, um doente de fobia de elevadores pode iniciar a sua terapia apenas a pensar em entrar num elevador. De seguida, o terapeuta pode levar o doente a andar de elevador apenas de um andar para o seguinte, a andar de elevador durante vários andares, e a entrar num elevador muito lotado. Este método designa-se assim por dessensibilização fóbica. No entanto, os métodos convencionais possuem várias limitações. Nomeadamente, o tratamento de fobias peca pela falta de quantificação (não são retiradas quaisquer métricas de avaliação), personalização ao doente (a personalização é apenas dependente da opinião subjetiva do terapeuta), ritmo terapêutico gradual e controlado, e segurança, visto que se baseia em terapia por exposição in vivo. A investigação tem demonstrado a eficácia da utilização alternativa de terapia por exposição virtual, baseada em jogos sérios em Realidade Virtual (RV). No entanto, apesar deste método permitir um ritmo terapêutico gradual e em segurança (permitindo uma exposição fóbica virtual no ambiente clínico seguro, ao invés de in vivo – em estágios terapêuticos iniciais e intermédios), não soluciona a falta de quantificação e personalização. Assim, surgiu a hipótese de adicionar biofeedback – uma técnica emergente que utiliza sinais vitais para controlar diretamente a adaptação de um dado sistema, amplamente aplicada em sistemas de treino cerebral – para personalizar e quantificar a terapia por exposição virtual. A técnica é bastante utilizada recorrendo a sinais cerebrais (Neurofeedback), no entanto também são utilizados sinais cardiovasculares, por exemplo. É atualmente conhecido que, a nível fisiológico, as emoções (especificamente o medo) e a ansiedade são correlacionáveis com respostas fisiológicas do sistema cardiovascular. Por exemplo, o sistema cardiovascular responde ao stress, em conjunto com o sistema endócrino, com elevados níveis de cortisol e com um ritmo cardíaco e uma pressão sanguínea aumentados. O presente estudo avalia a eficácia de um novo método de tratamento de agorafobia (ansiedade/medo extremos de espaços abertos ou fechados com multidões) baseado em jogos sérios em RV e biofeedback, como complemento aos métodos terapêuticos convencionais. A adição de uma técnica complementar de relaxamento em RV – seguinte à exposição fóbica – é também avaliada. Como primeiro passo, o estudo avalia se a ansiedade suscita respostas cardíacas e cerebrais diferenciadas, isto é, se é possível retirar biomarcadores da ansiedade. Seguindo investigação prévia, uma experiência preliminar foi conduzida com 156 pessoas saudáveis que assistiram a um conjunto de videoclipes que suscitavam respostas emocionais diferentes, enquanto que as suas atividades cerebrais e cardíacas foram monitorizadas através de sensores de Eletroencefalografia (EEG) e Fotopletismografia (PPG), respetivamente. Foram estudadas seis categorias emocionais: Medo, Alegria, Raiva, Nojo, Neutro, Tristeza, e Ternura. A categoria emocional de Ansiedade/Medo suscitou respostas diferenciadas nos sinais fisiológicos, sugerindo que componentes podem ser utilizadas como biomarcadores da ansiedade. A categoria emocional Ternura foi também alvo de uma análise detalhado, dado que esta é uma emoção ainda pouco conhecida, não sendo consensual a sua natureza. De seguida, um teste de prova-de-conceito de 8 sessões foi conduzido com 5 doentes de agorafobia, dos quais 3 deles foram submetidos ao protocolo terapêutico convencional com a adição do novo método de RV + biofeedback, enquanto que os restantes 2 doentes foram submetidos apenas ao protocolo convencional (psicoterapia e farmacologia). O protocolo do novo método inicia-se com questionários de auto-avaliação de agorafobia e de ansiedade, de seguida passa para o cenário RV de exposição fóbica, e finalmente para o cenário RV de relaxamento. O cenário RV de exposição fóbica consiste numa sala de cinema, na qual o número de pessoas varia de sessão para sessão consoante os resultados fisiológicos e auto-reportados do doente na sessão anterior (biofeedback manual). Por outro lado, o cenário RV de relaxamento consiste numa praia paradísica numa ilha, na qual a turbulência das ondas do mar varia automaticamente consoante os resultados fisiológicos e auto-reportados do doente na sessão anterior (biofeedback preliminar). O objetivo do cenário RV de exposição fóbica é assim expor o doente gradualmente ao cenário de fobia, personalizando o nível de exposição em cada sessão consoante a sua resposta, através do biofeedback. Ou seja, se na sessão anterior o doente conseguiu estar confortável no cenário, o que se traduz nos seus sinais fisiológicos, na sessão seguinte é exposto a um cenário de maior intensidade, obrigando-o assim a habituar-se gradualmente ao cenário. Contrariamente, o objetivo do cenário RV de relaxamento é relaxar gradualmente o doente, expondo-o a um cenário cujo caráter relaxante se intensifica à medida que o doente relaxa, seguindo uma metodologia de biofeedback semelhante. Os resultados mostraram que a diminuição dos sintomas ansiosos e agorafóbicos, entre a primeira e a última sessões, no grupo experimental, foi 3,28 e 5,02 vezes mais elevada, respetivamente, do que a diminuição desses sintomas no grupo de controlo. Resultados relativos a potenciais biomarcadores de estados ansiosos e relaxados foram também adquiridos. De um modo geral, estas conclusões sugerem e quantificam o valor acrescentado deste novo método terapêutico, como complemento à psicoterapia convencional, demonstrando, como foi conjeturado, que a abordagem mista de exposição fóbica + relaxamento permite uma maior redução sintomática. O presente trabalho constitui assim um avanço no estado-da-arte, dado que se retiraram resultados conclusivos relativamente à eficácia e valor acrescentado de um método terapêutico inovador, ainda não explorado na literatura. Trabalho futuro irá avaliar a eficácia e o valor acrescentado do biofeedback, através de um grupo de controlo com biofeedback placebo. Os jogos sérios poderão também ser melhorados, desenvolvendo outros cenários e também os triggers da ansiedade, alvo do controlo automático via biofeedback. Pretende-se também otimizar o biofeedback, através de algoritmos de Aprendizagem Automática de reconhecimento emocional, nomeadamente utilizando algoritmos de redes neuronais. Especificamente, o presente trabalho servirá de base para o desenvolvimento e avaliação de um classificador de estados ansiosos baseado nos biomarcadores encontrados. Por fim, planeia-se ainda avaliar o valor acrescentado de uma abordagem terapêutica no domicílio, como complemento à terapêutica clínica.Current treatment methods for mental disorders – namely, psychotherapy and pharmacology – have several limitations. Specifically, phobias’ treatment lack quantification, personalization to the patient, a gradual and controlled therapy pace, and safety, since it relies on in vivo exposure therapy (phobic desensitization). Research has shown the efficacy of using virtual exposure therapy, based on Virtual Reality (VR) serious games. However, although this method enables a gradual and controlled therapy pace, it does not solve quantification and personalization. Thus, the hypothesis to add biofeedback – an emergent technique that uses vital signs to directly control a system’s adaptation, widely used for brain training systems – to personalize and quantify virtual exposure therapy, arose. This study aims to assess the efficacy of a novel agoraphobia (extreme anxiety/fear of crowded open or closed spaces) treatment method based on VR serious games and biofeedback, as a complement to the conventional methods. The addition of a complementary VR relaxation technique – following the phobic exposure – was also studied. As a first step, the study aims at evaluating if anxiety elicits differentiated brain and heart activity responses, i.e., it aims at evaluating if anxiety biomarkers can be retrieved. Following previous research, a preliminary study was conducted with 156 healthy subjects that watched a set of videoclips that elicited different emotional responses, while their brain and heart activity was monitored through Electroencephalography (EEG) and Photoplethysmography (PPG) sensors, respectively. The fear/anxious emotional category elicited differentiated responses on heart and brain activity, suggesting that certain features can be used as anxiety biomarkers. It was retrieved conclusions regarding the best anxiety biomarker performer, showing the strongest correlations with the self-reporting emotional states, suggesting that the Anxiety/Fear emotional state elicited most differentiated responses on the cardiovascular system, rather than on the Central Nervous System and providing insights into the not yet consensual literature on the topic. Then, a proof-of-concept trial of 8 sessions was conducted with 5 agoraphobic patients, in which 3 of them underwent the conventional treatment protocol with the addition of the novel VR + biofeedback method, while the other 2 only underwent the conventional protocol. The novel method’s protocol begins with agoraphobia and anxiety self-assessment questionnaires, then moves to the VR phobic exposure scenario, and, lastly, to the VR relaxation scenario. Results showed that the decrease of anxious and agoraphobic symptoms, between the initial and last sessions, in the experimental group was 3.28 and 5.02 times greater, respectively, than the decrease of those symptoms in the control group. Results regarding anxious and relaxed states’ biomarkers were also retrieved. Overall, these findings show and quantify the added-value of this novel therapy method – innovative in the literature –, as a complement to the conventional psychotherapy, showing that, as hypothesized, the mixed exposure + relaxation approach enables a more significant symptom reduction. Future work will assess the efficiency and added-value of biofeedback, using Machine Learning methods, as well as of a home-based approach

EEG based assessment of emotional wellbeing in smart environment

Abstract. Smart technologies are frequently united and automated in our everyday settings and commonplace task by linking computers and other devices. While there has been a necessity to build smart environments for an easy and comfortable life, research on measuring wellbeing in this environment becomes increasingly intensive. Emotion is one of the decisive aspects of wellbeing that encourages us to work effectively, manage, and cope with stress, and affect our physical health. This work evaluates the EEG signal to measure individuals the different emotional states in a smart space by creating a computer gaming scenario. EEG, a physiological signal which provides details on mental, physiological, and emotional states, EEG frequency bands are strongly correlated with positive and negative emotional responses. Since brain left frontal cortical area is responsible for positive emotion and the right frontal region associate, therefore, we choose two pairs of EEG electrodes F3-F4, and F7-F8 to assess the game player emotional states during the gaming situations. We measure the EEG frontal alpha asymmetry (FAA) by comparing variations in the alpha band power levels in the left and right frontal cortex, corresponding to positive and negative emotions. Our experiment outcome reveals considerable support with the emotional variance of the test participants. We note that multiple interruptions during the gaming situation create irritation to the test subjects. These findings also confirm that F3 and F4 EEG channels are the most sensitive to human emotional responses compared to F7 and F8 channels

Stress and heart rate: significant parameters and their variations

The aim of this paper is to identify heart rate parameters with higher significant values when a set of people are performing a task under stress condition. In order to accomplish this, one computer application with arithmetic and memory activities which lets drive the subjects to different stages of activity and stress has been designed. Tests are formed by initial and final rest periods and three task phases with incremental stressful level. Electrocardiogram is measured in each state and parameters are extracted from it. A statistical study using analysis of variance (ANOVA) is done to see which ones are the most significant. It is concluded that the median of RR segments is the parameter to best determine the state of stress.Regional Government of Andalusia (p08-TIC-3631

Emotion and Stress Recognition Related Sensors and Machine Learning Technologies

This book includes impactful chapters which present scientific concepts, frameworks, architectures and ideas on sensing technologies and machine learning techniques. These are relevant in tackling the following challenges: (i) the field readiness and use of intrusive sensor systems and devices for capturing biosignals, including EEG sensor systems, ECG sensor systems and electrodermal activity sensor systems; (ii) the quality assessment and management of sensor data; (iii) data preprocessing, noise filtering and calibration concepts for biosignals; (iv) the field readiness and use of nonintrusive sensor technologies, including visual sensors, acoustic sensors, vibration sensors and piezoelectric sensors; (v) emotion recognition using mobile phones and smartwatches; (vi) body area sensor networks for emotion and stress studies; (vii) the use of experimental datasets in emotion recognition, including dataset generation principles and concepts, quality insurance and emotion elicitation material and concepts; (viii) machine learning techniques for robust emotion recognition, including graphical models, neural network methods, deep learning methods, statistical learning and multivariate empirical mode decomposition; (ix) subject-independent emotion and stress recognition concepts and systems, including facial expression-based systems, speech-based systems, EEG-based systems, ECG-based systems, electrodermal activity-based systems, multimodal recognition systems and sensor fusion concepts and (x) emotion and stress estimation and forecasting from a nonlinear dynamical system perspective