Neurophysiological Assessment of Affective Experience

In the field of Affective Computing the affective experience (AX) of the user during the interaction with computers is of great interest. The automatic recognition of the affective state, or emotion, of the user is one of the big challenges. In this proposal I focus on the affect recognition via physiological and neurophysiological signals. Long‐standing evidence from psychophysiological research and more recently from research in affective neuroscience suggests that both, body and brain physiology, are able to indicate the current affective state of a subject. However, regarding the classification of AX several questions are still unanswered. The principal possibility of AX classification was repeatedly shown, but its generalisation over different task contexts, elicitating stimuli modalities, subjects or time is seldom addressed. In this proposal I will discuss a possible agenda for the further exploration of physiological and neurophysiological correlates of AX over different elicitation modalities and task contexts

The Appreciative Heart: The Psychophysiology of Positive Emotions and Optimal Functioning

This monograph is an overview of Institute of HeartMath's research on the physiological correlates of positive emotions and the science underlying two core HeartMath techniques which supports Heart-Based Living. The heart's connection with love and other positive emotions has survived throughout millennia and across many diverse cultures. New empirical research is providing scientific validation for this age-old association. This 21-page monograph offers a comprehensive understanding of the Institute of HeartMath's cutting-edge research exploring the heart's central role in emotional experience. Described in detail is physiological coherence, a distinct mode of physiological functioning, which is generated during sustained positive emotions and linked with beneficial health and performance-related outcomes. The monograph also provides steps and applications of two HeartMath techniques, Freeze-Frame(R) and Heart Lock-In(R), which engage the heart to help transform stress and produce sustained states of coherence. Data from outcome studies are presented, which suggest that these techniques facilitate a beneficial repatterning process at the mental, emotional and physiological levels

A pediatric near-infrared spectroscopy brain-computer interface based on the detection of emotional valence

Brain-computer interfaces (BCIs) are being investigated as an access pathway to communication for individuals with physical disabilities, as the technology obviates the need for voluntary motor control. However, to date, minimal research has investigated the use of BCIs for children. Traditional BCI communication paradigms may be suboptimal given that children with physical disabilities may face delays in cognitive development and acquisition of literacy skills. Instead, in this study we explored emotional state as an alternative access pathway to communication. We developed a pediatric BCI to identify positive and negative emotional states from changes in hemodynamic activity of the prefrontal cortex (PFC). To train and test the BCI, 10 neurotypical children aged 8-14 underwent a series of emotion-induction trials over four experimental sessions (one offline, three online) while their brain activity was measured with functional near-infrared spectroscopy (fNIRS). Visual neurofeedback was used to assist participants in regulating their emotional states and modulating their hemodynamic activity in response to the affective stimuli. Child-specific linear discriminant classifiers were trained on cumulatively available data from previous sessions and adaptively updated throughout each session. Average online valence classification exceeded chance across participants by the last two online sessions (with 7 and 8 of the 10 participants performing better than chance, respectively, in Sessions 3 and 4). There was a small significant positive correlation with online BCI performance and age, suggesting older participants were more successful at regulating their emotional state and/or brain activity. Variability was seen across participants in regards to BCI performance, hemodynamic response, and discriminatory features and channels. Retrospective offline analyses yielded accuracies comparable to those reported in adult affective BCI studies using fNIRS. Affective fNIRS-BCIs appear to be feasible for school-aged children, but to further gauge the practical potential of this type of BCI, replication with more training sessions, larger sample sizes, and end-users with disabilities is necessary

Toward Emotion Recognition From Physiological Signals in the Wild: Approaching the Methodological Issues in Real-Life Data Collection

Emotion, mood, and stress recognition (EMSR) has been studied in laboratory settings for decades. In particular, physiological signals are widely used to detect and classify affective states in lab conditions. However, physiological reactions to emotional stimuli have been found to differ in laboratory and natural settings. Thanks to recent technological progress (e.g., in wearables) the creation of EMSR systems for a large number of consumers during their everyday activities is increasingly possible. Therefore, datasets created in the wild are needed to insure the validity and the exploitability of EMSR models for real-life applications. In this paper, we initially present common techniques used in laboratory settings to induce emotions for the purpose of physiological dataset creation. Next, advantages and challenges of data collection in the wild are discussed. To assess the applicability of existing datasets to real-life applications, we propose a set of categories to guide and compare at a glance different methodologies used by researchers to collect such data. For this purpose, we also introduce a visual tool called Graphical Assessment of Real-life Application-Focused Emotional Dataset (GARAFED). In the last part of the paper, we apply the proposed tool to compare existing physiological datasets for EMSR in the wild and to show possible improvements and future directions of research. We wish for this paper and GARAFED to be used as guidelines for researchers and developers who aim at collecting affect-related data for real-life EMSR-based applications

Breaking Down the Barriers To Operator Workload Estimation: Advancing Algorithmic Handling of Temporal Non-Stationarity and Cross-Participant Differences for EEG Analysis Using Deep Learning

This research focuses on two barriers to using EEG data for workload assessment: day-to-day variability, and cross- participant applicability. Several signal processing techniques and deep learning approaches are evaluated in multi-task environments. These methods account for temporal, spatial, and frequential data dependencies. Variance of frequency- domain power distributions for cross-day workload classification is statistically significant. Skewness and kurtosis are not significant in an environment absent workload transitions, but are salient with transitions present. LSTMs improve day- to-day feature stationarity, decreasing error by 59% compared to previous best results. A multi-path convolutional recurrent model using bi-directional, residual recurrent layers significantly increases predictive accuracy and decreases cross-participant variance. Deep learning regression approaches are applied to a multi-task environment with workload transitions. Accounting for temporal dependence significantly reduces error and increases correlation compared to baselines. Visualization techniques for LSTM feature saliency are developed to understand EEG analysis model biases

Variability in heart and brain activity across the adult lifespan

The world population is rapidly aging. In Germany for example, the percentage of individuals 60 years and older is projected to be 38% in 20501. Longer lifetimes entail more progressive impairment of brain and body. It is therefore a crucial question how to assess and quantify these frequently occurring alterations associated with aging. In order to address this question, the overarching goal of this dissertation is to explore and characterize bodily and neural signals which reflect effects of aging across the adult lifespan. To this end, I performed two studies as lead investigator and contributed to three more large-scale collaborative studies. In Study 1 (Kumral et al., 2019), I investigated the relationship of heart rate variability (HRV) to brain structure (gray matter) and resting state (rs) brain activity (functional connectivity) in a well-characterized sample of healthy subjects across the adult lifespan (N=388). For Study 2 (Koenig et al., 2020), I contributed to a mega analysis testing the association between cortical thickness and heart-rate variability (HRV) at rest, also across the lifespan (N=1218). In Study 3 (Kumral et al., 2020), I examined whether different measures of brain signal variability – identified with hemodynamic (functional magnetic resonance imaging; fMRI) or electrophysiological (EEG) methods – reflect the same underlying physiology in healthy younger and older adults (N=189). Lastly, during my dissertation work, I was part of the Mind-Body-Emotion group in Leipzig, which established two publicly available – and now widely used – datasets (Datasets 1 and 2; Babayan et al., 2019, Mendes et al., 2019), which include structural and functional MRI, EEG data as well as a range of physiological and behavioral measures. In Study 1, I showed that age-related decreases in resting HRV are accompanied by age-dependent and age-invariant alterations in brain function, particularly located along cortical midline structures. In Study 2, we found that the age-related decrease of resting HRV was associated with cortical thinning in prefrontal brain structures. In Study 3, I demonstrated age differences in brain signal variability obtained with rs-fMRI and rs-EEG, respectively. Surprisingly, the two measures of neural variability showed no significant correlation, but rather seemed to provide complementary information on the state of the aging brain. The present dissertation provides evidence that measures of cardiovascular and neural signal variability may be useful biomarkers for neurocognitive health (and disease) in aging. With these measures, we can further specify the dynamic interplay of the human body and the brain in relation to individual health-related factors

Inner speech as language process and cognitive tool.

Many people report a form of internal language known as inner speech (IS). This review examines recent growth of research interest in the phenomenon, which has broadly supported a theoretical model in which IS is a functional language process that can confer benefits for cognition in a range of domains. A key insight to have emerged in recent years is that IS is an embodied experience characterized by varied subjective qualities, which can be usefully modeled in artificial systems and whose neural signals have the potential to be decoded through advancing brain-computer interface technologies. Challenges for future research include understanding individual differences in IS and mapping form to function across IS subtypes

Dispositional Mindfulness as a Moderator of Electrocortical and Behavioral Responses to Affective Social Stimuli

Numerous studies have linked dispositional mindfulness to enhanced emotion regulation. The present research examined dispositional mindfulness as a predictor of emotion regulation in social affective contexts. Participants completed passive viewing and Emotional Go/No-Go tasks involving social affective stimuli (happy, neutral, and fearful facial expressions). Event-related potentials (ERPs) and behavioral responses were examined to discern whether dispositional mindfulness predicted differential neural and behavioral responses indexing attention to, awareness of, and inhibitory control over automatic responses to affective social stimuli. Dispositional mindfulness predicted larger (more negative) N100, N200 and No-Go N200 amplitudes during the Emotional Go/No-Go task, but was not associated with amplitude of the Late Positive Potential during the passive viewing task. Dispositional mindfulness also predicted faster response times (RT) to target stimuli that were not attributable to a speed-accuracy tradeoff. No relations were found between mindfulness and RT variability nor accuracy. Implications for understanding mindfulness and early processes of social emotion regulation are discussed

Reconocimiento de Estados Afectivos a partir de Señales Biomédicas

Las emociones constituyen una parte fundamental de los individuos, influyendo en sucomunicación diaria, la toma de decisiones y el foco de atención. La incorporación de las emociones en la tecnología ha avanzado en losúltimos años, desde estudios exploratorios en la respuesta a los estímulos, a aplicaciones comerciales en interfaces hombre-máquina. Una de las fuentes paraidentificar estados emocionales es la respuesta fisiológica, registrada medianteseñales biomédicas. El uso de estas señales permitiría el desarrollo de dispositivos poco invasivos, como por ejemplo una pulsera, que puedan registrarseñales continuamente, en diferentes condiciones, y manteniendo la privacidad delos usuarios. Existen numerosos enfoques para el reconocimiento de afectos, condiferentes señales, técnicas de procesamiento de la señal y métodos deaprendizaje automático. Entre ellos, la combinación demúltiples señales se utilizó ampliamente para mejorar las tasas de reconocimiento,pero resulta inviable en la práctica por su invasividad. Los desafíosactuales requieren clasificadores que puedan funcionar en tiempo real, enaplicaciones interactivas, y con mayor comodidad para el usuario. En esta tesis doctoral se aborda el desafío del reconocimiento de estadosafectivos en varios aspectos. Se revisan las propiedades de cada señalfisiológica en términos de su practicidad y potencial. Se propone un método paraadaptar un clasificador a nuevos usuarios, estimando parámetros fisiológicosbasales. Luego se presentan dos métodos originales paramejorar las tasas de reconocimiento. El primero es un método supervisado basadoen mapas auto-organizativos (sSOM). Este método permite representar los espacios de características fisiológicas ymodelos emocionales, para analizar las relaciones en los datos. El otro estabasado en máquinas de aprendizaje extremo (ELM),una novedosa familia de redes neuronales artificiales que tiene gran poder degeneralización y puede entrenarse con pocos datos. Los métodos fueron evaluados y comparados con los del estadodel arte, en corpus realistas y de acceso libre. Los resultados obtenidos muestran avances en relación al estado del arte para el problema. Elmétodo de adaptación permite, a partir de pocos segundos,mejorar las tasas de reconocimiento en tiempo real, aproximando los resultados delreconocimiento que se podría hacer con posterioridad, sobre los registros completos. Utilizando una única señal de actividad cardiovascular, en particularla variabilidad del ritmo cardíaco (HRV), se lograron avances prometedores, con diferencias significativasen relación a los resultados obtenidos por los métodos del estado del arte. LasELM obtuvieron excelentes resultados y con bajo costo computacional, por lo queserían útiles para aplicaciones móviles. El sSOMlogra resultados similares, con la ventaja de proveer a la vez una herramientapara representar y analizar los espacios complejos de la fisiología y lasemociones, en una forma compacta.Fil: Bugnon, Leandro Ariel. Universidad Nacional del Litoral; Argentin

Physiological indicators and subjective restorativeness with audio-visual interactions in urban soundscapes

The present study aimed to identify the trends of changes in physiological indicators and subjective restorativeness in response to audio-visual interactions in the environment. Four scenarios types were presented using four different modalities (video-sound, image-sound, sound-only, and video-only; each modality was evaluated by independent groups of subjects). The physiological responses and subjective restoration of subjects were measured to assess the interactions between the audio-visual modalities. These data were also analysed to determine the physiological and subjective differences between dynamic and static visual presentations. We found that with visual modalities, the heart rate (HR), heart rate variability (HRV) calculated using the standard deviation of the NN intervals (SDNN[sbnd]HRV), high-frequency band in the HRV power spectrum (HF[sbnd]HRV), alpha reactivity on electroencephalography, and skin temperature (ST) decreased; however, the beta reactivity on EEG and skin conductance level (SCL) increased. With auditory modalities, the SDNN[sbnd]HRV, HF-HRV, ST, and respiration depth decreased; however, the respiration rate and SCL increased. Use of static images and sound to reproduce the natural environment evoked more physiological comfort and subjective restorativeness. These findings could provide physiological insights for the theory of the restorative environment