Dimensional affect recognition from HRV: an approach based on supervised SOM and ELM

Dimensional affect recognition is a challenging topic and current techniques do not yet provide the accuracy necessary for HCI applications. In this work we propose two new methods. The first is a novel self-organizing model that learns from similarity between features and affects. This method produces a graphical representation of the multidimensional data which may assist the expert analysis. The second method uses extreme learning machines, an emerging artificial neural network model. Aiming for minimum intrusiveness, we use only the heart rate variability, which can be recorded using a small set of sensors. The methods were validated with two datasets. The first is composed of 16 sessions with different participants and was used to evaluate the models in a classification task. The second one was the publicly available Remote Collaborative and Affective Interaction (RECOLA) dataset, which was used for dimensional affect estimation. The performance evaluation used the kappa score, unweighted average recall and the concordance correlation coefficient. The concordance coefficient on the RECOLA test partition was 0.421 in arousal and 0.321 in valence. Results show that our models outperform state-of-the-art models on the same data and provides new ways to analyze affective states

On-body Sensing and Signal Analysis for User Experience Recognition in Human-Machine Interaction

—In this paper, a new algorithm is proposed for recognition of user experience through emotion detection using physiological signals, for application in human-machine interaction. The algorithm recognizes user’s emotion quality and intensity in a two dimensional emotion space continuously. The continuous recognition of the user’s emotion during human-machine interaction will enable the machine to adapt its activity based on the user’s emotion in a real-time manner, thus improving user experience. The emotion model underlying the proposed algorithm is one of the most recent emotion models, which models emotion’s intensity and quality in a continuous two-dimensional space of valance and arousal axes. Using only two physiological signals, which are correlated to the valance and arousal axes of the emotion space, is among the contributions of this paper. Prediction of emotion through physiological signals has the advantage of elimination of social masking and making the prediction more reliable. The key advantage of the proposed algorithm over other algorithms presented to date is the use of the least number of modalities (only two physiological signals) to predict the quality and intensity of emotion continuously in time, and using the most recent widely accepted emotion model

A Review on EEG Signals Based Emotion Recognition

Emotion recognition has become a very controversial issue in brain-computer interfaces (BCIs). Moreover, numerous studies have been conducted in order to recognize emotions. Also, there are several important definitions and theories about human emotions. In this paper we try to cover important topics related to the field of emotion recognition. We review several studies which are based on analyzing electroencephalogram (EEG) signals as a biological marker in emotion changes. Considering low cost, good time and spatial resolution, EEG has become very common and is widely used in most BCI applications and studies. First, we state some theories and basic definitions related to emotions. Then some important steps of an emotion recognition system like different kinds of biologic measurements (EEG, electrocardiogram [EEG], respiration rate, etc), offline vs online recognition methods, emotion stimulation types and common emotion models are described. Finally, the recent and most important studies are reviewed

A Review on the Computational Methods for Emotional State Estimation from the Human EEG

A growing number of affective computing researches recently developed a computer system that can recognize an emotional state of the human user to establish affective human-computer interactions. Various measures have been used to estimate emotional states, including self-report, startle response, behavioral response, autonomic measurement, and neurophysiologic measurement. Among them, inferring emotional states from electroencephalography (EEG) has received considerable attention as EEG could directly reflect emotional states with relatively low costs and simplicity. Yet, EEG-based emotional state estimation requires well-designed computational methods to extract information from complex and noisy multichannel EEG data. In this paper, we review the computational methods that have been developed to deduct EEG indices of emotion, to extract emotion-related features, or to classify EEG signals into one of many emotional states. We also propose using sequential Bayesian inference to estimate the continuous emotional state in real time. We present current challenges for building an EEG-based emotion recognition system and suggest some future directions.open

Approaches, applications, and challenges in physiological emotion recognition — a tutorial overview

An automatic emotion recognition system can serve as a fundamental framework for various applications in daily life from monitoring emotional well-being to improving the quality of life through better emotion regulation. Understanding the process of emotion manifestation becomes crucial for building emotion recognition systems. An emotional experience results in changes not only in interpersonal behavior but also in physiological responses. Physiological signals are one of the most reliable means for recognizing emotions since individuals cannot consciously manipulate them for a long duration. These signals can be captured by medical-grade wearable devices, as well as commercial smart watches and smart bands. With the shift in research direction from laboratory to unrestricted daily life, commercial devices have been employed ubiquitously. However, this shift has introduced several challenges, such as low data quality, dependency on subjective self-reports, unlimited movement-related changes, and artifacts in physiological signals. This tutorial provides an overview of practical aspects of emotion recognition, such as experiment design, properties of different physiological modalities, existing datasets, suitable machine learning algorithms for physiological data, and several applications. It aims to provide the necessary psychological and physiological backgrounds through various emotion theories and the physiological manifestation of emotions, thereby laying a foundation for emotion recognition. Finally, the tutorial discusses open research directions and possible solutions

Emotion and Stress Recognition Related Sensors and Machine Learning Technologies

This book includes impactful chapters which present scientific concepts, frameworks, architectures and ideas on sensing technologies and machine learning techniques. These are relevant in tackling the following challenges: (i) the field readiness and use of intrusive sensor systems and devices for capturing biosignals, including EEG sensor systems, ECG sensor systems and electrodermal activity sensor systems; (ii) the quality assessment and management of sensor data; (iii) data preprocessing, noise filtering and calibration concepts for biosignals; (iv) the field readiness and use of nonintrusive sensor technologies, including visual sensors, acoustic sensors, vibration sensors and piezoelectric sensors; (v) emotion recognition using mobile phones and smartwatches; (vi) body area sensor networks for emotion and stress studies; (vii) the use of experimental datasets in emotion recognition, including dataset generation principles and concepts, quality insurance and emotion elicitation material and concepts; (viii) machine learning techniques for robust emotion recognition, including graphical models, neural network methods, deep learning methods, statistical learning and multivariate empirical mode decomposition; (ix) subject-independent emotion and stress recognition concepts and systems, including facial expression-based systems, speech-based systems, EEG-based systems, ECG-based systems, electrodermal activity-based systems, multimodal recognition systems and sensor fusion concepts and (x) emotion and stress estimation and forecasting from a nonlinear dynamical system perspective

Autonomic Nervous System Dynamics for Mood and Emotional-State Recognition: Wearable Systems, Modeling, and Advanced Biosignal Processing

This thesis aims at investigating how electrophysiological signals related to the autonomic nervous system (ANS) dynamics could be source of reliable and effective markers for mood state recognition and assessment of emotional responses. In-depth methodological and applicative studies of biosignals such as electrocardiogram, electrodermal response, and respiration activity along with information coming from the eyes (gaze points and pupil size variation) were performed. Supported by the current literature, I found that nonlinear signal processing techniques play a crucial role in understanding the underlying ANS physiology and provide important quantifiers of cardiovascular control dynamics with prognostic value in both healthy subjects and patients. Two main applicative scenarios were identified: the former includes a group of healthy subjects who was presented with sets of images gathered from the International Affective Picture System hav- ing five levels of arousal and five levels of valence, including both a neutral reference level. The latter was constituted by bipolar patients who were followed for a period of 90 days during which psychophysical evaluations were performed. In both datasets, standard signal processing techniques as well as nonlinear measures have been taken into account to automatically and accurately recognize the elicited levels of arousal and valence and mood states, respectively. A novel probabilistic approach based on the point-process theory was also successfully applied in order to model and characterize the instantaneous ANS nonlinear dynamics in both healthy subjects and bipolar patients. According to the reported evidences on ANS complex behavior, experimental results demonstrate that an accurate characterization of the elicited affective levels and mood states is viable only when non- linear information are retained. Moreover, I demonstrate that the instantaneous ANS assessment is effective in both healthy subjects and patients. Besides mathematics and signal processing, this thesis also contributes to pragmatic issues such as emotional and mood state mod- eling, elicitation, and noninvasive ANS monitoring. Throughout the dissertation, a critical review on the current state-of-the-art is reported leading to the description of dedicated experimental protocols, reliable mood models, and novel wearable systems able to perform ANS monitoring in a naturalistic environment

Reconocimiento de Estados Afectivos a partir de Señales Biomédicas

Las emociones constituyen una parte fundamental de los individuos, influyendo en sucomunicación diaria, la toma de decisiones y el foco de atención. La incorporación de las emociones en la tecnología ha avanzado en losúltimos años, desde estudios exploratorios en la respuesta a los estímulos, a aplicaciones comerciales en interfaces hombre-máquina. Una de las fuentes paraidentificar estados emocionales es la respuesta fisiológica, registrada medianteseñales biomédicas. El uso de estas señales permitiría el desarrollo de dispositivos poco invasivos, como por ejemplo una pulsera, que puedan registrarseñales continuamente, en diferentes condiciones, y manteniendo la privacidad delos usuarios. Existen numerosos enfoques para el reconocimiento de afectos, condiferentes señales, técnicas de procesamiento de la señal y métodos deaprendizaje automático. Entre ellos, la combinación demúltiples señales se utilizó ampliamente para mejorar las tasas de reconocimiento,pero resulta inviable en la práctica por su invasividad. Los desafíosactuales requieren clasificadores que puedan funcionar en tiempo real, enaplicaciones interactivas, y con mayor comodidad para el usuario. En esta tesis doctoral se aborda el desafío del reconocimiento de estadosafectivos en varios aspectos. Se revisan las propiedades de cada señalfisiológica en términos de su practicidad y potencial. Se propone un método paraadaptar un clasificador a nuevos usuarios, estimando parámetros fisiológicosbasales. Luego se presentan dos métodos originales paramejorar las tasas de reconocimiento. El primero es un método supervisado basadoen mapas auto-organizativos (sSOM). Este método permite representar los espacios de características fisiológicas ymodelos emocionales, para analizar las relaciones en los datos. El otro estabasado en máquinas de aprendizaje extremo (ELM),una novedosa familia de redes neuronales artificiales que tiene gran poder degeneralización y puede entrenarse con pocos datos. Los métodos fueron evaluados y comparados con los del estadodel arte, en corpus realistas y de acceso libre. Los resultados obtenidos muestran avances en relación al estado del arte para el problema. Elmétodo de adaptación permite, a partir de pocos segundos,mejorar las tasas de reconocimiento en tiempo real, aproximando los resultados delreconocimiento que se podría hacer con posterioridad, sobre los registros completos. Utilizando una única señal de actividad cardiovascular, en particularla variabilidad del ritmo cardíaco (HRV), se lograron avances prometedores, con diferencias significativasen relación a los resultados obtenidos por los métodos del estado del arte. LasELM obtuvieron excelentes resultados y con bajo costo computacional, por lo queserían útiles para aplicaciones móviles. El sSOMlogra resultados similares, con la ventaja de proveer a la vez una herramientapara representar y analizar los espacios complejos de la fisiología y lasemociones, en una forma compacta.Fil: Bugnon, Leandro Ariel. Universidad Nacional del Litoral; Argentin