DeepBreath: Deep learning of breathing patterns for automatic stress recognition using low-cost thermal imaging in unconstrained settings

We propose DeepBreath, a deep learning model which automatically recognises people’s psychological stress level (mental overload) from their breathing patterns. Using a low cost thermal camera, we track a person’s breathing patterns as temperature changes around his/her nostril. The paper’s technical contribution is threefold. First of all, instead of creating hand-crafted features to capture aspects of the breathing patterns, we transform the uni-dimensional breathing signals into two dimensional respiration variability spectrogram (RVS) sequences. The spectrograms easily capture the complexity of the breathing dynamics. Second, a spatial pattern analysis based on a deep Convolutional Neural Network (CNN) is directly applied to the spectrogram sequences without the need of hand-crafting features. Finally, a data augmentation technique, inspired from solutions for over-fitting problems in deep learning, is applied to allow the CNN to learn with a small-scale dataset from short-term measurements (e.g., up to a few hours). The model is trained and tested with data collected from people exposed to two types of cognitive tasks (Stroop Colour Word Test, Mental Computation test) with sessions of different difficulty levels. Using normalised self-report as ground truth, the CNN reaches 84.59% accuracy in discriminating between two levels of stress and 56.52% in discriminating between three levels. In addition, the CNN outperformed powerful shallow learning methods based on a single layer neural network. Finally, the dataset of labelled thermal images will be open to the community

Detecting Stressful Social Interactions Using Wearable Physiological and Inertial Sensors

Stress is unavoidable in everyday life which can result in several health related short and long-term adverse consequences. Previous research found that most of the stress events occur due to interpersonal tension followed by work related stress. Enabling automated detection of stressful social interactions using wearable technology will help trigger just-in-time interventions which can help the user cope with the stressful situation. In this dissertation, we show the feasibility of differentiating stressful social interactions from other stressors i.e., work and commute.However, collecting reliable ground truth stressor data in the natural environment is challenging. This dissertation addresses this challenge by designing a Day Reconstruction Method (DRM) based contextual stress visualization that highlights the continuous stress inferences from a wearable sensor with surrounding activities such as conversation, physical activity, and location on a timeline diagram. This dissertation proposes a Conditional Random Field, Context-Free Grammar (CRF-CFG) model to detect conversation from breathing patterns to support the visualization. The advantage of breathing signal is that it does not capture the content of the conversation and hence, is more privacy preserving compared to audio. It proposes a framework to systematically analyze the breathing data collected in the natural environment. However, it requires wearing of chest worn sensor. This dissertation aims to determine stressful social interaction without wearing chest worn sensor or without requiring any conversation model which is privacy sensitive. Therefore, it focuses on detecting stressful social interactions directly from stress time-series only which can be captured using increasingly available wrist worn sensor.This dissertation proposes a framework to systematically analyze the respiration data collected in the natural environment. The analysis includes screening the low-quality data, segmenting the respiration time-series by cycles, and develop time-domain features. It proposes a Conditional Random Field, Context-Free Grammar (CRF-CFG) model to detect conversation episodes from breathing patterns. This system is validated against audio ground-truth in the field with an accuracy of 71.7\%.This dissertation introduces the stress cycle concept to capture the cyclical patterns and identifies novel features from stress time-series data. Furthermore, wrist-worn accelerometry data shows that hand gestures have a distinct pattern during stressful social interactions. The model presented in this dissertation augments accelerometry patterns with the stress cycle patterns for more accurate detection. Finally, the model is trained and validated using data collected from 38 participants in free-living conditions. The model can detect the stressful interactions with an F1-score of 0.83 using stress cycle features and enable the delivery of stress intervention within 3.9 minutes since the onset of a stressful social interaction

NON-VERBAL COMMUNICATION WITH PHYSIOLOGICAL SENSORS. THE AESTHETIC DOMAIN OF WEARABLES AND NEURAL NETWORKS

Historically, communication implies the transfer of information between bodies, yet this phenomenon is constantly adapting to new technological and cultural standards. In a digital context, it’s commonplace to envision systems that revolve around verbal modalities. However, behavioural analysis grounded in psychology research calls attention to the emotional information disclosed by non-verbal social cues, in particular, actions that are involuntary. This notion has circulated heavily into various interdisciplinary computing research fields, from which multiple studies have arisen, correlating non-verbal activity to socio-affective inferences. These are often derived from some form of motion capture and other wearable sensors, measuring the ‘invisible’ bioelectrical changes that occur from inside the body. This thesis proposes a motivation and methodology for using physiological sensory data as an expressive resource for technology-mediated interactions. Initialised from a thorough discussion on state-of-the-art technologies and established design principles regarding this topic, then applied to a novel approach alongside a selection of practice works to compliment this. We advocate for aesthetic experience, experimenting with abstract representations. Atypically from prevailing Affective Computing systems, the intention is not to infer or classify emotion but rather to create new opportunities for rich gestural exchange, unconfined to the verbal domain. Given the preliminary proposition of non-representation, we justify a correspondence with modern Machine Learning and multimedia interaction strategies, applying an iterative, human-centred approach to improve personalisation without the compromising emotional potential of bodily gesture. Where related studies in the past have successfully provoked strong design concepts through innovative fabrications, these are typically limited to simple linear, one-to-one mappings and often neglect multi-user environments; we foresee a vast potential. In our use cases, we adopt neural network architectures to generate highly granular biofeedback from low-dimensional input data. We present the following proof-of-concepts: Breathing Correspondence, a wearable biofeedback system inspired by Somaesthetic design principles; Latent Steps, a real-time auto-encoder to represent bodily experiences from sensor data, designed for dance performance; and Anti-Social Distancing Ensemble, an installation for public space interventions, analysing physical distance to generate a collective soundscape. Key findings are extracted from the individual reports to formulate an extensive technical and theoretical framework around this topic. The projects first aim to embrace some alternative perspectives already established within Affective Computing research. From here, these concepts evolve deeper, bridging theories from contemporary creative and technical practices with the advancement of biomedical technologies.Historicamente, os processos de comunicação implicam a transferência de informação entre organismos, mas este fenómeno está constantemente a adaptar-se a novos padrões tecnológicos e culturais. Num contexto digital, é comum encontrar sistemas que giram em torno de modalidades verbais. Contudo, a análise comportamental fundamentada na investigação psicológica chama a atenção para a informação emocional revelada por sinais sociais não verbais, em particular, acções que são involuntárias. Esta noção circulou fortemente em vários campos interdisciplinares de investigação na área das ciências da computação, dos quais surgiram múltiplos estudos, correlacionando a actividade nãoverbal com inferências sócio-afectivas. Estes são frequentemente derivados de alguma forma de captura de movimento e sensores “wearable”, medindo as alterações bioeléctricas “invisíveis” que ocorrem no interior do corpo. Nesta tese, propomos uma motivação e metodologia para a utilização de dados sensoriais fisiológicos como um recurso expressivo para interacções mediadas pela tecnologia. Iniciada a partir de uma discussão aprofundada sobre tecnologias de ponta e princípios de concepção estabelecidos relativamente a este tópico, depois aplicada a uma nova abordagem, juntamente com uma selecção de trabalhos práticos, para complementar esta. Defendemos a experiência estética, experimentando com representações abstractas. Contrariamente aos sistemas de Computação Afectiva predominantes, a intenção não é inferir ou classificar a emoção, mas sim criar novas oportunidades para uma rica troca gestual, não confinada ao domínio verbal. Dada a proposta preliminar de não representação, justificamos uma correspondência com estratégias modernas de Machine Learning e interacção multimédia, aplicando uma abordagem iterativa e centrada no ser humano para melhorar a personalização sem o potencial emocional comprometedor do gesto corporal. Nos casos em que estudos anteriores demonstraram com sucesso conceitos de design fortes através de fabricações inovadoras, estes limitam-se tipicamente a simples mapeamentos lineares, um-para-um, e muitas vezes negligenciam ambientes multi-utilizadores; com este trabalho, prevemos um potencial alargado. Nos nossos casos de utilização, adoptamos arquitecturas de redes neurais para gerar biofeedback altamente granular a partir de dados de entrada de baixa dimensão. Apresentamos as seguintes provas de conceitos: Breathing Correspondence, um sistema de biofeedback wearable inspirado nos princípios de design somaestético; Latent Steps, um modelo autoencoder em tempo real para representar experiências corporais a partir de dados de sensores, concebido para desempenho de dança; e Anti-Social Distancing Ensemble, uma instalação para intervenções no espaço público, analisando a distância física para gerar uma paisagem sonora colectiva. Os principais resultados são extraídos dos relatórios individuais, para formular um quadro técnico e teórico alargado para expandir sobre este tópico. Os projectos têm como primeiro objectivo abraçar algumas perspectivas alternativas às que já estão estabelecidas no âmbito da investigação da Computação Afectiva. A partir daqui, estes conceitos evoluem mais profundamente, fazendo a ponte entre as teorias das práticas criativas e técnicas contemporâneas com o avanço das tecnologias biomédicas

Connecting Brains and Bodies: Applying Physiological Computing to Support Social Interaction

Physiological and affective computing propose methods to improve human-machine interactions by adapting machines to the users' states. Recently, social signal processing (SSP) has proposed to apply similar methods to human-human interactions with the hope of better understanding and modeling social interactions. Most of the social signals employed are facial expressions, body movements and speech, but studies using physiological signals remain scarce. In this paper, we motivate the use of physiological signals in the context of social interactions. Specifically, we review studies which have investigated the relationship between various physiological indices and social interactions. We then propose two main directions to apply physiological SSP: using physiological signals of individual users as new social cues displayed in the group and using inter-user physiology to measure properties of the interactions such as conflict and social presence. We conclude that physiological measures have the potential to enhance social interactions and to connect peopl

Impact of Temperament Types and Anger Intensity on Drivers\u27 EEG Power Spectrum and Sample Entropy: An On-road Evaluation Toward Road Rage Warning

"Road rage", also called driving anger, is becoming an increasingly common phenomenon affecting road safety in auto era as most of previous driving anger detection approaches based on physiological indicators are often unreliable due to the less consideration of drivers\u27 individual differences. This study aims to explore the impact of temperament types and anger intensity on drivers\u27 EEG characteristics. Thirty-two drivers with valid license were enrolled to perform on-road experiments on a particularly busy route on which a variety of provoking events like cutting in line of surrounding vehicle, jaywalking, occupying road of non-motor vehicle and traffic congestion frequently happened. Then, muti-factor analysis of variance (ANOVA) and post hoc analysis were utilized to study the impact of temperament types and anger intensity on drivers\u27 power spectrum and sample entropy of θ and β waves extracted from EEG signals. The study results firstly indicated that right frontal region of the brain has close relationship with driving anger. Secondly, there existed significant main effects of temperament types on power spectrum and sample entropy of β wave while significant main effects of anger intensity on power spectrum and sample entropy of θ and β wave were all observed. Thirdly, significant interactions between temperament types and anger intensity for power spectrum and sample entropy of β wave were both noted. Fourthly, with the increase of anger intensity, the power spectrum and sample entropy both decreased sufficiently for θ wave while increased remarkably for β wave. The study results can provide a theoretical support for designing a personalized and hierarchical warning system for road rage