Multimodal assessment of emotional responses by physiological monitoring: novel auditory and visual elicitation strategies in traditional and virtual reality environments

This doctoral thesis explores novel strategies to quantify emotions and listening effort through monitoring of physiological signals. Emotions are a complex aspect of the human experience, playing a crucial role in our survival and adaptation to the environment. The study of emotions fosters important applications, such as Human-Computer and Human-Robot interaction or clinical assessment and treatment of mental health conditions such as depression, anxiety, stress, chronic anger, and mood disorders. Listening effort is also an important area of study, as it provides insight into the listeners’ challenges that are usually not identified by traditional audiometric measures. The research is divided into three lines of work, each with a unique emphasis on the methods of emotion elicitation and the stimuli that are most effective in producing emotional responses, with a specific focus on auditory stimuli. The research fostered the creation of three experimental protocols, as well as the use of an available online protocol for studying emotional responses including monitoring of both peripheral and central physiological signals, such as skin conductance, respiration, pupil dilation, electrocardiogram, blood volume pulse, and electroencephalography. An emotional protocol was created for the study of listening effort using a speech-in-noise test designed to be short and not induce fatigue. The results revealed that the listening effort is a complex problem that cannot be studied with a univariate approach, thus necessitating the use of multiple physiological markers to study different physiological dimensions. Specifically, the findings demonstrate a strong association between the level of auditory exertion, the amount of attention and involvement directed towards stimuli that are readily comprehensible compared to those that demand greater exertion. Continuing with the auditory domain, peripheral physiological signals were studied in order to discriminate four emotions elicited in a subject who listened to music for 21 days, using a previously designed and publicly available protocol. Surprisingly, the processed physiological signals were able to clearly separate the four emotions at the physiological level, demonstrating that music, which is not typically studied extensively in the literature, can be an effective stimulus for eliciting emotions. Following these results, a flat-screen protocol was created to compare physiological responses to purely visual, purely auditory, and combined audiovisual emotional stimuli. The results show that auditory stimuli are more effective in separating emotions at the physiological level. The subjects were found to be much more attentive during the audio-only phase. In order to overcome the limitations of emotional protocols carried out in a laboratory environment, which may elicit fewer emotions due to being an unnatural setting for the subjects under study, a final emotional elicitation protocol was created using virtual reality. Scenes similar to reality were created to elicit four distinct emotions. At the physiological level, it was noted that this environment is more effective in eliciting emotions. To our knowledge, this is the first protocol specifically designed for virtual reality that elicits diverse emotions. Furthermore, even in terms of classification, the use of virtual reality has been shown to be superior to traditional flat-screen protocols, opening the doors to virtual reality for the study of conditions related to emotional control

Neurofeedback Treatment in Attention Deficit Hyperactivity Disorder and Comorbit Aggression

The present thesis focused on specificity and long-term effects of slow-cortical potential neurofeedback (SCP-NF) treatment for children with ADHD in a large multicenter randomized controlled trial, on its relation to aggressive behaviors as a common comorbidity of ADHD, and on neuroimaging and psychophysiological subtypes of aggression. We assessed clinical efficacy on ADHD and comorbid aggression in comparison to a semi-active control group which controlled for unspecific effects. The role of self-regulation and learning of SCPs was systematically evaluated. Additionally, we investigated amygdala-specific activity in aggression subtypes in a large multicenter cohort, which might provide a possible putative NF target. The first two studies assessed 150 children aged 7–9 years diagnosed with ADHD which were randomized to 25 sessions of feedback of SCPs (NF) or feedback of coordination of the supraspinatus muscles (EMG). The primary outcome was the change in ADHD symptoms rated by parents four weeks and six-month after treatment end. Slow-cortical potential neurofeedback showed significant superiority over the semi-active control condition with medium effect sizes four weeks after treatment. This superiority of SCP-NF over the semi-active control group became non-significant 6 months after treatment end. However, taking together all assessments, SCP-NF showed a stable improvement with large effect sizes following treatment and EMG-BF showed worsening of symptoms one month after treatment, with subsequent remission at follow-up, leading to non-significant group differences six months after treatment end. Assessment of self-regulation showed significant ability to self-regulate slow-cortical potential when direct feedback is given and improvement of self-regulation skills indicate specificity of SCP-NF for selected subscales after training, but not at follow-up. In sum, these findings suggest shared specific and unspecific effects contributing to this clinical outcome. The third study aimed to disentangle aggression-related subtypes at a neural level. In total 177 participants (n=108 cases with aggression-related disorders and n= 69 typically developing peers), aged 8-18 years were assessed across nine sites in Europa during a well-established emotional face-matching fMRI task. Additionally, simultaneous skin conductance recordings were acquired in a subsample (n=64). Children and adolescents with aggression-related problems showed higher amygdala activity in response to negative faces compared to typically developing peers. Further, we showed distinct amygdala activity for subtypes of aggression. Callous-unemotional traits showed to moderate both central (amygdala) and peripheral (SC) responses. These findings increase insights which could be used for personalized diagnostics and treatments

Biometric storyboards: a games user research approach for improving qualitative evaluations of player experience

Developing video games is an iterative and demanding process. It is difficult to achieve the goal of most video games — to be enjoyable, engaging and to create revenue for game developers — because of many hard-to-evaluate factors, such as the different ways players can interact with the game. Understanding how players behave during gameplay is of vital importance to developers and can be uncovered in user tests as part of game development. This can help developers to identify and resolve any potential problem areas before release, leading to a better player experience and possibly higher game review scores and sales. However, traditional user testing methods were developed for function and efficiency oriented applications. Hence, many traditional user testing methods cannot be applied in the same way for video game evaluation. This thesis presents an investigation into the contributions of physiological measurements in user testing within games user research (GUR). GUR specifically studies the interaction between a game and users (players) with the aim to provide feedback for developers to help them to optimise the game design of their title. An evaluation technique called Biometric Storyboards is developed, which visualises the relationships between game events, player feedback and changes in a player’s physiological state. Biometric Storyboards contributes to the field of human-computer interaction and GUR in three important areas: (1) visualising mixedmeasures of player experience, (2) deconstructing game design by analysing game events and pace, (3) incremental improvement of classic user research techniques (such as interviews and physiological measurements). These contributions are described in practical case studies, interviews with game developers and laboratory experiments. The results show this evaluation approach can enable games user researchers to increase the plausibility and persuasiveness of their reports and facilitate developers to better deliver their design goals. Biometric Storyboards is not aimed at replacing existing methods, but to extend them with mixed methods visualisations, to provide powerful tools for games user researchers and developers to better understand and communicate player needs, interactions and experiences. The contributions of this thesis are directly applicable for user researchers and game developers, as well as for researchers in user experience evaluation in entertainment systems

Affective Computing

This book provides an overview of state of the art research in Affective Computing. It presents new ideas, original results and practical experiences in this increasingly important research field. The book consists of 23 chapters categorized into four sections. Since one of the most important means of human communication is facial expression, the first section of this book (Chapters 1 to 7) presents a research on synthesis and recognition of facial expressions. Given that we not only use the face but also body movements to express ourselves, in the second section (Chapters 8 to 11) we present a research on perception and generation of emotional expressions by using full-body motions. The third section of the book (Chapters 12 to 16) presents computational models on emotion, as well as findings from neuroscience research. In the last section of the book (Chapters 17 to 22) we present applications related to affective computing

Animal models of exercise therapy: mechanisms of activity-induced angiogenesis

Skeletal muscle is spatially heterogeneous in muscle fibre type composition and microvascular supply. The capacity to quantify this heterogeneity in skeletal muscle is not routinely performed for it’s a laborious and time consuming technique. We have developed a high throughput data pipeline that utilises the simultaneous immunohistochemical labelling of muscle fibre type and microvascular supply, as an input for a semi-automated analysis software package that allows for the analysis of fine morphometric indices of fibre type composition and the interactions with microvascular supply. We have successfully shown that regional variation in fibre type composition impacts the functional characteristics of a muscle. After successful characterisation of regional heterogeneity in both structure and function we sought to establish their influence in physiological (adaptive) angiogenesis. Utilising animal angiogenic models we have shown that shear stress driven angiogenesis is principally a stochastic response that does not promote improved oxygen delivery when we analyse the spatial heterogeneity of the neovasculature. Conversely, skeletal muscle overload (abluminal stretch of microvasculature) increases the homogeneity of the oxygen supply area of the capillary bed, suggesting a tissue driven angiogenic response that is not evident in shear stress. Spinal cord injury induced rarefaction of the capillary bed attempts to maintain a homogeneous distribution of fibre size and capillary supply. The combination therapy of epidural stimulation and locomotor training can ameliorate the phenotypic change and rarefied capillary bed seen with spinal cord injury to that of intact levels. Endurance and resistance exercise have a largely similar global genomic response following a chronic training regime, which we are able to replicate in animal models of exercise through indirect electrical stimulation. The shear stress and muscle overload driven angiogenic response have distinctly different angiogenic pathways that contain no commonly expressed networks

Animal models of exercise therapy: mechanisms of activity-induced angiogenesis

Skeletal muscle is spatially heterogeneous in muscle fibre type composition and microvascular supply. The capacity to quantify this heterogeneity in skeletal muscle is not routinely performed for it’s a laborious and time consuming technique. We have developed a high throughput data pipeline that utilises the simultaneous immunohistochemical labelling of muscle fibre type and microvascular supply, as an input for a semi-automated analysis software package that allows for the analysis of fine morphometric indices of fibre type composition and the interactions with microvascular supply. We have successfully shown that regional variation in fibre type composition impacts the functional characteristics of a muscle. After successful characterisation of regional heterogeneity in both structure and function we sought to establish their influence in physiological (adaptive) angiogenesis. Utilising animal angiogenic models we have shown that shear stress driven angiogenesis is principally a stochastic response that does not promote improved oxygen delivery when we analyse the spatial heterogeneity of the neovasculature. Conversely, skeletal muscle overload (abluminal stretch of microvasculature) increases the homogeneity of the oxygen supply area of the capillary bed, suggesting a tissue driven angiogenic response that is not evident in shear stress. Spinal cord injury induced rarefaction of the capillary bed attempts to maintain a homogeneous distribution of fibre size and capillary supply. The combination therapy of epidural stimulation and locomotor training can ameliorate the phenotypic change and rarefied capillary bed seen with spinal cord injury to that of intact levels. Endurance and resistance exercise have a largely similar global genomic response following a chronic training regime, which we are able to replicate in animal models of exercise through indirect electrical stimulation. The shear stress and muscle overload driven angiogenic response have distinctly different angiogenic pathways that contain no commonly expressed networks

Advanced Signal Processing in Wearable Sensors for Health Monitoring

Smart, wearables devices on a miniature scale are becoming increasingly widely available, typically in the form of smart watches and other connected devices. Consequently, devices to assist in measurements such as electroencephalography (EEG), electrocardiogram (ECG), electromyography (EMG), blood pressure (BP), photoplethysmography (PPG), heart rhythm, respiration rate, apnoea, and motion detection are becoming more available, and play a significant role in healthcare monitoring. The industry is placing great emphasis on making these devices and technologies available on smart devices such as phones and watches. Such measurements are clinically and scientifically useful for real-time monitoring, long-term care, and diagnosis and therapeutic techniques. However, a pertaining issue is that recorded data are usually noisy, contain many artefacts, and are affected by external factors such as movements and physical conditions. In order to obtain accurate and meaningful indicators, the signal has to be processed and conditioned such that the measurements are accurate and free from noise and disturbances. In this context, many researchers have utilized recent technological advances in wearable sensors and signal processing to develop smart and accurate wearable devices for clinical applications. The processing and analysis of physiological signals is a key issue for these smart wearable devices. Consequently, ongoing work in this field of study includes research on filtration, quality checking, signal transformation and decomposition, feature extraction and, most recently, machine learning-based methods