Quantitative Multidimensional Stress Assessment from Facial Videos

Stress has a significant impact on the physical and mental health of an individual and is a growing concern for society, especially during the COVID-19 pandemic. Facial video-based stress evaluation from non-invasive cameras has proven to be a significantly more efficient method to evaluate stress in comparison to approaches that use questionnaires or wearable sensors. Plenty of classification models have been built for stress detection. However, most do not consider individual differences. Also, the results for such models are limited by a uni-dimensional definition of stress levels lacking a comprehensive quantitative definition of stress. The dissertation focuses on building a framework that utilizes the multilevel video frame representations from deep learning and the remote photoplethysmography signals extracted from the facial videos for stress assessment. The fusion model takes the inputs of a baseline video and a target video of the subject. The physiological features such as heart rate and heart rate variability are used with the initial stress scores generated from deep learning are used to predict the stress scores in cognitive anxiety, somatic anxiety, and self-confidence. To generate stress scores with better accuracy, the signal extraction method is improved by introducing the CWT-SNR method that uses the signal-to-noise ratio to assist the adaptive bandpass filtering in the post-processing of the signals. A study on phase space reconstruction features is performed and the results show the potential for additional accuracy improvement for the heart rate variability detection. To select the best deep learning architecture, multiple deep learning architectures are tested to build the deep learning model. Support Vector Regression is used to generate the output stress score results. Testing with the data from the UBFC-Phys dataset, the fusion model shows a strong correlation between ground truth and the predicted results

Hyperspectral imaging for the remote sensing of blood oxygenation and emotions

This PhD project is a basic research and it concerns with how human’s physiological features, such as tissue oxygen saturation (StO2), can be captured from a stand-off distance and then to understand how this remotely acquired physiological feature can be deployed for biomedical and other applications. This work utilises Hyperspectral Imaging (HSI) within the diffuse optical scattering framework, to assess the StO2 in a contactless remote sensing manner. The assessment involves a detailed investigation about the wavelength dependence of diffuse optical scattering from the skin as well as body tissues, under various forms of optical absorption models. It is concluded that the threechromophore extended Beer Lambert Law model is better suited for assessing the palm and facial tissue oxygenations, especially when spectral data in the wavelengths region of [516-580]nm is used for the analysis. A first attempt of using the facial StO2 to detect and to classify people’s emotional state is initiated in this project. The objective of this work is to understand how strong emotions, such as distress that caused by mental or physical stimulations, can be detected using physiological feature such as StO2. Based on data collected from ~20 participants, it is found that the forehead StO2 is elevated upon the onset of strong emotions that triggered by mental stimulation. The StO2 pattern in the facial region upon strong emotions that are initiated by physical stimulations is quite complicated, and further work is needed for a better understanding of the interplays between bodily physique, individual’s health condition and blood transfusion control mechanism. Most of this work has already been published and future research to follow up when the author returns back to China is highlighted

Remote sensing of strong emotions using electro-optical imaging technique

©Cranfield UniversityThis thesis reports a summary of the PhD programme for the assessment of person‘s emotional anxiety using Electro-optical technology. The thesis focuses mainly on the understanding of fundamental properties of physiological responses to emotional anxiety and how they can be captured by using Electro-optical (EO) imaging methods such as hyperspectral imaging (HSI) and thermal imaging (TI) techniques. The thesis summarises three main areas of work that have been undertaken by the author in the programme: (a) Experimental set up including HSI system and data acquisition software design and implementation, (b) fundamental understanding of physiological responses to emotional anxiety from the EO perspective and (c) the development of a novel remote sensing technique for the assessment of emotions without the requirement of base line information. One of our main results is to provide evidence to prove that the mean temperature in the periorbital region remains the same within 0.2°C during emotional anxiety. Furthermore, we have shown that it is the high temperature pixels within the periorbital, which increases in numbers by a huge amount after 2 minutes of the onset of anxiety. We have also developed techniques to allow the assessment anxiety without the need of base line information. The method has been tested using a sample size of about 40 subjects, and achieved promising result. Technologies for the remote sensing of heart beat rate has been in great demand, this study also involves the development of heart beat detection using TI system. Moreover, we have also attempted for the first time to sense glucose concentration from the blood sample in-vivo using HSI technique remotely

A review of biophysiological and biochemical indicators of stress for connected and preventive healthcare

Stress is a known contributor to several life-threatening medical conditions and a risk factor for triggering acute cardiovascular events, as well as a root cause of several social problems. The burden of stress is increasing globally and, with that, is the interest in developing effective stress-monitoring solutions for preventive and connected health, particularly with the help of wearable sensing technologies. The recent development of miniaturized and flexible biosensors has enabled the development of connected wearable solutions to monitor stress and intervene in time to prevent the progression of stress-induced medical conditions. This paper presents a review of the literature on different physiological and chemical indicators of stress, which are commonly used for quantitative assessment of stress, and the associated sensing technologies

Learning deep physiological models of affect

Feature extraction and feature selection are crucial phases in the process of affective modeling. Both, however, incorporate substantial limitations that hinder the development of reliable and accurate models of affect. For the purpose of modeling affect manifested through physiology, this paper builds on recent advances in machine learning with deep learning (DL) approaches. The efficiency of DL algorithms that train artificial neural network models is tested and compared against standard feature extraction and selection approaches followed in the literature. Results on a game data corpus — containing players’ physiological signals (i.e. skin conductance and blood volume pulse) and subjective self-reports of affect — reveal that DL outperforms manual ad-hoc feature extraction as it yields significantly more accurate affective models. Moreover, it appears that DL meets and even outperforms affective models that are boosted by automatic feature selection, for several of the scenarios examined. As the DL method is generic and applicable to any affective modeling task, the key findings of the paper suggest that ad-hoc feature extraction and selection — to a lesser degree — could be bypassed.The authors would like to thank Tobias Mahlmann for his work on the development and administration of the cluster used to run the experiments. Special thanks for proofreading goes to Yana Knight. Thanks also go to the Theano development team, to all participants in our experiments, and to Ubisoft, NSERC and Canada Research Chairs for funding. This work is funded, in part, by the ILearnRW (project no: 318803) and the C2Learn (project no. 318480) FP7 ICT EU projects.peer-reviewe

Feature Space Augmentation: Improving Prediction Accuracy of Classical Problems in Cognitive Science and Computer Vison

The prediction accuracy in many classical problems across multiple domains has seen a rise since computational tools such as multi-layer neural nets and complex machine learning algorithms have become widely accessible to the research community. In this research, we take a step back and examine the feature space in two problems from very different domains. We show that novel augmentation to the feature space yields higher performance. Emotion Recognition in Adults from a Control Group: The objective is to quantify the emotional state of an individual at any time using data collected by wearable sensors. We define emotional state as a mixture of amusement, anger, disgust, fear, sadness, anxiety and neutral and their respective levels at any time. The generated model predicts an individual’s dominant state and generates an emotional spectrum, 1x7 vector indicating levels of each emotional state and anxiety. We present an iterative learning framework that alters the feature space uniquely to an individual’s emotion perception, and predicts the emotional state using the individual specific feature space. Hybrid Feature Space for Image Classification: The objective is to improve the accuracy of existing image recognition by leveraging text features from the images. As humans, we perceive objects using colors, dimensions, geometry and any textual information we can gather. Current image recognition algorithms rely exclusively on the first 3 and do not use the textual information. This study develops and tests an approach that trains a classifier on a hybrid text based feature space that has comparable accuracy to the state of the art CNN’s while being significantly inexpensive computationally. Moreover, when combined with CNN’S the approach yields a statistically significant boost in accuracy. Both models are validated using cross validation and holdout validation, and are evaluated against the state of the art

Stress level assessment with non-intrusive sensors

Mención Internacional en el título de doctorStress is an involuntary reaction where the human body changes from a calm state to an excited state in order to preserve the integrity of the organism. Small amount of stress should be good to became entrepreneur and learn new ways of thinking, but continuous stress can carry an array of daily risks, such as, cardiovascular diseases, hair loss, diabetes or immune dysregulation. Recognize how, when and where it occurs has become a step in stress assessment. Stress recognition starts from 1973 until now. This disease has become a problem in recent years because has increased the number of cases, especially in workers where his/her performance decreases. Stress reactions are provoked for the Autonomous Nervous System (ANS) and one way to estimate it could be found in physiological signals. A list of a variety wearable sensor is presented to capture these reactions, trying to minimize the risk of distraction due to external factors. The aim of this work thesis is to detect stress for level assessment. A combination of different physiological signals is selected to extract stress feature an classify in a rating scale from relax to breakdown situations. This thesis proposes a new feature extraction model to understand physiological Galvanic Skin Response (GSR) reactions. Last methods conclude in incongruent results that are not interpretable. This model propose a robust algorithm that can be used in real-time (low time computability) and results are sparse in time to obtain an easily statistical and graphical interpretation. Signal processing methods of heart rhythm and hormone cortisol are included to develop a robust feature extraction method of stress reactions. A combination of electrodermal, heart and hormone analysis is presented to know in real-time the state of the individual. These features have been selected because the acquisition is non-intrusive avoiding other factor such as distractions. This thesis is application-focused and highly multidisciplinary. A complete feature extraction model is presented including the new electrodermal model named and usual heart rhythm techniques. Three experiments were evaluated: a) a feature selection model using neurocognitive games, b) a stress classifier in time during public talks, and c) a real-time stress assessment classifier in a five-star rating scale. This thesis improve stress detection overcoming a system to capture physiological responses, analyze and conclude a stress assessment decision. We discussed past state of the art and propose a new method of feature extraction using signal processing improvements. Three different scenarios were evaluated to confirm the achievement of aims proposed.Programa Oficial de Doctorado en Multimedia y ComunicacionesPresidente: Joaquín Míguez Arenas.- Secretario: Luis Ignacio Santamaría Caballero.- Vocal: Mª Isabel Valera Martíne

How to Relax in Stressful Situations: A Smart Stress Reduction System

Stress is an inescapable element of the modern age. Instances of untreated stress may lead to a reduction in the individual's health, well-being and socio-economic situation. Stress management application development for wearable smart devices is a growing market. The use of wearable smart devices and biofeedback for individualized real-life stress reduction interventions has received less attention. By using our unobtrusive automatic stress detection system for use with consumer-grade smart bands, we first detected stress levels. When a high stress level is detected, our system suggests the most appropriate relaxation method by analyzing the physical activity-based contextual information. In more restricted contexts, physical activity is lower and mobile relaxation methods might be more appropriate, whereas in free contexts traditional methods might be useful. We further compared traditional and mobile relaxation methods by using our stress level detection system during an eight day EU project training event involving 15 early stage researchers (mean age 28; gender 9 Male, 6 Female). Participants' daily stress levels were monitored and a range of traditional and mobile stress management techniques was applied. On day eight, participants were exposed to a 'stressful' event by being required to give an oral presentation. Insights about the success of both traditional and mobile relaxation methods by using the physiological signals and collected self-reports were provided