ThermSense: Smartphone-based Breathing Sensing Platform using Noncontact Low-Cost Thermal Camera

The ability of sensing breathing is becoming an increasingly important function for technology that aims at supporting both psychological and physical wellbeing. We demonstrate ThermSense, a new breathing sensing platform based on smartphone technology and low-cost thermal camera, which allows a user to measure his/her breathing pattern in a contact-free manner. With the designed key functions of Thermal Voxel Integration-based breathing estimation and respiration variability spectrogram (RVS, bi-dimensional representation of breathing dynamics), the developed platform provides scalability and flexibility for gathering respiratory physiological measurements ubiquitously. The functionality could be used for a variety of applications from stress monitoring to respiration training

Technology-assisted emotion recognition for autism spectrum disorder (ASD) children: a systematic literature review

The information about affective states in individuals with autism spectrum disorder (ASD) is difficult to obtain as they usually suffer from deficits in facial expression. Affective state conditions of individuals with ASD were associated with impaired regulation of speech, communication, and social skills leading towards poor socio-emotion interaction. It is conceivable that the advance of technology could offer a psychophysiological alternative modality, particularly useful in persons who cannot verbally communicate their emotions as affective states such as individuals with ASD. The study is focusing on the investigation of technology-assisted approach and its relationship to affective states recognition. A systematic review was executed to summarize relevant research that involved technology-assisted implementation to identify the affective states of individuals with ASD using Preferred Reporting Items for Systematic Reviews and MetaAnalyses (PRISMA) approach. The output from the online search process obtained from six publication databases on relevant studies published up to 31 July 2020 was analyzed. Out of 391 publications retrieved, 20 papers met the inclusion and exclusion criteria set in prior. Data were synthesized narratively despite methodological and heterogeneity variations. In this review, some research methods, systems, equipment and models to address all the related issues to the technology-assisted and affective states concerned were presented. As for the consequence, it can be assumed that the emotion recognition with assisted by technology, for evaluating and classifying affective states could help to improve efficacy in therapy sessions between therapists and individuals with ASD. This review will serve as a concise reference for providing general overviews of the current state-of-the-art studies in this area for practitioners, as well as for experienced researchers who are searching for a new direction for future works

Automated mental stress recognition through mobile thermal imaging

Mental stress is a critical problem in our modern society. This form of stress strongly affects our well being, and technology is needed to help us to manage health problems. The ability to automatically recognize a person’s mental stress can be fundamental in supporting stress and health management. This research focuses on the use of mobile thermal imaging, a new and less explored sensor, to merge the measurement of multiple physiological signatures into one sensor and to build a reliable mental stress automatic recognition model. Mobile thermal imaging has greater potentials for real-world applications given that it is small and light weight, and requires low computation cost. To make mobile thermal imaging a robust multimodal stress sensor, we have so far contributed: i) a new robust respiration tracking method; and ii) a novel respiration-based automatic stress recognition model that works in ubiquitous settings. We are currently investigating new thermal signatures from underexplored body regions (i.e. trapezius muscle) and formulating a research framework to fuse multiple thermal signatures for more reliable stress recognition outcomes

Your blush gives you away: detecting hidden mental states with remote photoplethysmography and thermal imaging

Multimodal emotion recognition techniques are increasingly essential for assessing mental states. Image-based methods, however, tend to focus predominantly on overt visual cues and often overlook subtler mental state changes. Psychophysiological research has demonstrated that HR and skin temperature are effective in detecting ANS activities, thereby revealing these subtle changes. However, traditional HR tools are generally more costly and less portable, while skin temperature analysis usually necessitates extensive manual processing. Advances in remote-PPG and automatic thermal ROI detection algorithms have been developed to address these issues, yet their accuracy in practical applications remains limited. This study aims to bridge this gap by integrating r-PPG with thermal imaging to enhance prediction performance. Ninety participants completed a 20-minute questionnaire to induce cognitive stress, followed by watching a film aimed at eliciting moral elevation. The results demonstrate that the combination of r-PPG and thermal imaging effectively detects emotional shifts. Using r-PPG alone, the prediction accuracy was 77% for cognitive stress and 61% for moral elevation, as determined by SVM. Thermal imaging alone achieved 79% accuracy for cognitive stress and 78% for moral elevation, utilizing a RF algorithm. An early fusion strategy of these modalities significantly improved accuracies, achieving 87% for cognitive stress and 83% for moral elevation using RF. Further analysis, which utilized statistical metrics and explainable machine learning methods including SHAP, highlighted key features and clarified the relationship between cardiac responses and facial temperature variations. Notably, it was observed that cardiovascular features derived from r-PPG models had a more pronounced influence in data fusion, despite thermal imaging's higher predictive accuracy in unimodal analysis.Comment: 28 pages, 6 figure

Facial Thermal and Blood Perfusion Patterns of Human Emotions: Proof-of-Concept

In this work, a preliminary study of proof-of-concept was conducted to evaluate the performance of the thermographic and blood perfusion data when emotions of positive and negative valence are applied, where the blood perfusion data are obtained from the thermographic data. The images were obtained for baseline, positive, and negative valence according to the protocol of the Geneva Affective Picture Database. Absolute and percentage differences of average values of the data between the valences and the baseline were calculated for different regions of interest (forehead, periorbital eyes, cheeks, nose and upper lips). For negative valence, a decrease in temperature and blood perfusion was observed in the regions of interest, and the effect was greater on the left side than on the right side. In positive valence, the temperature and blood perfusion increased in some cases, showing a complex pattern. The temperature and perfusion of the nose was reduced for both valences, which is indicative of the arousal dimension. The blood perfusion images were found to be greater contrast; the percentage differences in the blood perfusion images are greater than those obtained in thermographic images. Moreover, the blood perfusion images, and vasomotor answer are consistent, therefore, they can be a better biomarker than thermographic analysis in identifying emotions.Comment: 22 pages, 9 figure

DeepBreath: Deep learning of breathing patterns for automatic stress recognition using low-cost thermal imaging in unconstrained settings

We propose DeepBreath, a deep learning model which automatically recognises people’s psychological stress level (mental overload) from their breathing patterns. Using a low cost thermal camera, we track a person’s breathing patterns as temperature changes around his/her nostril. The paper’s technical contribution is threefold. First of all, instead of creating hand-crafted features to capture aspects of the breathing patterns, we transform the uni-dimensional breathing signals into two dimensional respiration variability spectrogram (RVS) sequences. The spectrograms easily capture the complexity of the breathing dynamics. Second, a spatial pattern analysis based on a deep Convolutional Neural Network (CNN) is directly applied to the spectrogram sequences without the need of hand-crafting features. Finally, a data augmentation technique, inspired from solutions for over-fitting problems in deep learning, is applied to allow the CNN to learn with a small-scale dataset from short-term measurements (e.g., up to a few hours). The model is trained and tested with data collected from people exposed to two types of cognitive tasks (Stroop Colour Word Test, Mental Computation test) with sessions of different difficulty levels. Using normalised self-report as ground truth, the CNN reaches 84.59% accuracy in discriminating between two levels of stress and 56.52% in discriminating between three levels. In addition, the CNN outperformed powerful shallow learning methods based on a single layer neural network. Finally, the dataset of labelled thermal images will be open to the community

Correlation between measured stress and attention with thermal imaging techniques

Thermal imaging techniques enable an easier way to detect interaction compared to RGB or depth cameras and are therefore more comprehensively used in Human Computer Interaction (HCI) as a sensory system enabling novel interactive systems with the help of computer vision techniques. This makes it possible to use a thermal camera which operates in the Far Infrared (FIR) spectrum for the processing of humans skin temperature which is indicative of stress and therefore suits for detecting stress. Stress is considered of being one major cause for several sicknesses. Human's health can suffer because of stress and this includes also one's performance. Stress can affect the attention to different tasks. The evolution of techniques especially the computer technology overwhelmed a lot of people with an increasing need for computers in every department of work. This enables multi-tasking with several tasks being done simultaneously quite easily but also enabling a surge of stress. Having several tasks simultaneously to do can affect the outcome of them because a lack of concentration can occur or attention to the important task drops because of stress. The correlation between stress and attention can be investigated and based on a positive relation between them one can point to stress decreasing mechanisms in the case of performance slumps. In this thesis we show that through thermal imaging techniques it is possible to detect objects of interest. We investigate how thermal cameras can be used for detecting one's stress and which areas of a human are suitable for measuring stress. We moreover discuss the relation between stress and attention. Based on this researches an application will be implemented which will be able to detect stress. Using this application in a user study, where subjects perform several different stress generating tasks, we examine how stress affects attention and vice versa. Furthermore some areas which could benefit from the examined stress and attention relation are named.Wärmebildkameras machen es möglich Interaktionen einfacher als RGB Kameras zu erkennen und werden daher in der Mensch Computer Interaktion (HCI) als ein Sensorsystem benutzt, wodurch neuartige interaktive Systeme mit Hilfe von Computer Vision Techniken möglich werden. Dies macht es möglich eine Wärmebildkamera, die im fernen Infrarot Spektrum operiert, zu benutzen um die Hauttemperatur von Menschen zu verarbeiten, die hinweisend für Stress ist und daher gut zur Stresserkennung dient. Stress wird als eine der Hauptursachen für viele Krankheiten angesehen wobei dadurch die Gesundheit und Leistungsfähigkeit eines Menschen leiden kann. Stress kann die Aufmerksamkeit auf verschiedene Aufgaben beeinflussen. Die Technologieentwicklung, vor allem die der Computertechnologie, überforderte viele Menschen mit einem größeren Computerbedarf in allen Arbeitsbereichen. Das machte Multitasking auch am Arbeitsplatz leichter möglich. Diese Arbeitsweise kann aber durch einen Konzentrations- oder Aufmerksamkeitsverlust aufgrund von Stress das Ergebnis beeinflussen. Die Beziehung zwischen Stress und Aufmerksamkeit kann untersucht werden und bei einem positiven Ergebnis stressreduzierende Maßnahmen vorgeschlagen werden falls sich ein Leistungsabfall bemerkbar macht. In dieser Abschlussarbeit werden wir zeigen, dass es durch Wärmebildtechniken möglich ist gewünschte Objekte erkennen zu lassen. Wir werden untersuchen wie Wärmebildkameras benutzt werden können um Stress zu erkennen und welche Körperteile eines Menschen dafür geeignet sind. Darüber hinaus werden wir die Beziehung zwischen Stress und Aufmerksamkeit untersuchen. Basierend auf die Untersuchungen werden wir eine passende Applikation programmieren, die Stress erkennen kann. Diese Applikation wird in einer Nutzerstudie eingesetzt, in der die Teilnehmer viele verschiedene stresserzeugende Aufgaben bearbeiten um zu untersuchen wie Stress und Aufmerksamkeit zusammenhängen. Des Weiteren werden einige Anwendungsgebiete aufgezählt welche von den Ergebnissen profitieren können