Emotions in context: examining pervasive affective sensing systems, applications, and analyses

Pervasive sensing has opened up new opportunities for measuring our feelings and understanding our behavior by monitoring our affective states while mobile. This review paper surveys pervasive affect sensing by examining and considering three major elements of affective pervasive systems, namely; “sensing”, “analysis”, and “application”. Sensing investigates the different sensing modalities that are used in existing real-time affective applications, Analysis explores different approaches to emotion recognition and visualization based on different types of collected data, and Application investigates different leading areas of affective applications. For each of the three aspects, the paper includes an extensive survey of the literature and finally outlines some of challenges and future research opportunities of affective sensing in the context of pervasive computing

Semi-Supervised Generative Adversarial Network for Stress Detection Using Partially Labeled Physiological Data

Physiological measurements involves observing variables that attribute to the normative functioning of human systems and subsystems directly or indirectly. The measurements can be used to detect affective states of a person with aims such as improving human-computer interactions. There are several methods of collecting physiological data, but wearable sensors are a common, non-invasive tool for accurate readings. However, valuable information is hard to extract from the raw physiological data, especially for affective state detection. Machine Learning techniques are used to detect the affective state of a person through labeled physiological data. A clear problem with using labeled data is creating accurate labels. An expert is needed to analyze a form of recording of participants and mark sections with different states such as stress and calm. While expensive, this method delivers a complete dataset with labeled data that can be used in any number of supervised algorithms. An interesting question arises from the expensive labeling: how can we reduce the cost while maintaining high accuracy? Semi-Supervised learning (SSL) is a potential solution to this problem. These algorithms allow for machine learning models to be trained with only a small subset of labeled data (unlike unsupervised which use no labels). They provide a way of avoiding expensive labeling. This paper compares a fully supervised algorithm to a SSL on the public WESAD (Wearable Stress and Affect Detection) Dataset for stress detection. This paper shows that Semi-Supervised algorithms are a viable method for inexpensive affective state detection systems with accurate results.Comment: 12 page

On the Development of Machine Learning Based Real-Time Stress Monitoring : A Pilot Study

During specific environmental changes, the human body regulates itself through emotional, physical or mental responses. One such response is stress. The psychological and physical stability of an individual may be affected by recurrent occurrences of acute stress. This often leads to anxiety disorder, other psychological illnesses, hypertension, and other physiological disorders. The work performance of the individual is also negatively affected due to long-term stress. Across various age groups, the global population is primarily influenced by anxiety, depression and psychological stress. The long-term adverse effects of stress can be mitigated by effectively monitoring and managing stress through a cost-efficient and reliable stress detection system.  This paper mainly focuses on stress detection using a machine-learning approach. Wearable sensor data from electroencephalogram (EEG) and electrocardiogram (ECG) are considered during exposure to stress and the level of stress undergone by the participant is further analyzed. This approach helps in stress detection, analysis and mitigation, which in turn improves the quality life of people. Machining Learning technique k-means clustering algorithm is used after removal of artifacts to obtain case-specific clusters that segregate features pointing to non-stress and stress periods.  The results of the proposed K-means clustering algorithm are compared to state-of-the-art techniques such as Artificial Neural Network (ANN), Decision Tree (DT), Random Forest (RF) and Support Vector Machine (SVM). From the results, it was concluded that the proposed algorithm outperformed the other with an accuracy of 96% in the overall analysis

Machine Learning for Stress Monitoring from Wearable Devices: A Systematic Literature Review

Introduction. The stress response has both subjective, psychological and objectively measurable, biological components. Both of them can be expressed differently from person to person, complicating the development of a generic stress measurement model. This is further compounded by the lack of large, labeled datasets that can be utilized to build machine learning models for accurately detecting periods and levels of stress. The aim of this review is to provide an overview of the current state of stress detection and monitoring using wearable devices, and where applicable, machine learning techniques utilized. Methods. This study reviewed published works contributing and/or using datasets designed for detecting stress and their associated machine learning methods, with a systematic review and meta-analysis of those that utilized wearable sensor data as stress biomarkers. The electronic databases of Google Scholar, Crossref, DOAJ and PubMed were searched for relevant articles and a total of 24 articles were identified and included in the final analysis. The reviewed works were synthesized into three categories of publicly available stress datasets, machine learning, and future research directions. Results. A wide variety of study-specific test and measurement protocols were noted in the literature. A number of public datasets were identified that are labeled for stress detection. In addition, we discuss that previous works show shortcomings in areas such as their labeling protocols, lack of statistical power, validity of stress biomarkers, and generalization ability. Conclusion. Generalization of existing machine learning models still require further study, and research in this area will continue to provide improvements as newer and more substantial datasets become available for study.Comment: 50 pages, 8 figure

Wearable Biosensors to Understand Construction Workers' Mental and Physical Stress

Occupational stress is defined as harmful physical and mental responses when job requirements are greater than a worker's capacity. Construction is one of the most stressful occupations because it involves physiologically and psychologically demanding tasks performed in a hazardous environment this stress can jeopardize construction safety, health, and productivity. Various instruments, such as surveys and interviews, have been used for measuring workers’ perceived mental and physical stress. However valuable, such instruments are limited by their invasiveness, which prevents them from being used for continuous stress monitoring. The recent advancement of wearable biosensors has opened a new door toward the non-invasive collection of a field worker’s physiological signals that can be used to assess their mental and physical status. Despite these advancements, challenges remain: acquiring physiological signals from wearable biosensors can be easily contaminated from diverse sources of signal noise. Further, the potential of these devices to assess field workers’ mental and physical status has not been examined in the naturalistic work environment. To address these issues, this research aims to propose and validate a comprehensive and efficient stress-measurement framework that recognizes workers mental and physical stress in a naturalistic environment. The focus of this research is on two wearable biosensors. First, a wearable EEG headset, which is a direct measurement of brain waves with the minimal time lag, but it is highly vulnerable to various artifacts. Second, a very convenient wristband-type biosensor, which may be used as a means for assessing both mental and physical stress, but there is a time lag between when subjects are exposed to stressors and when their physiological signals change. To achieve this goal, five interrelated and interdisciplinary studies were performed to; 1) acquire high-quality EEG signals from the job site; 2) assess construction workers’ emotion by measuring the valence and arousal level by analyzing the patterns of construction workers’ brainwaves; 3) recognize mental stress in the field based on brain activities by applying supervised-learning algorithms;4) recognize real-time mental stress by applying Online Multi-Task Learning (OMTL) algorithms; and 5) assess workers’ mental and physical stress using signals collected from a wristband biosensor. To examine the performance of the proposed framework, we collected physiological signals from 21 workers at five job sites. Results yielded a high of 80.13% mental stress-recognition accuracy using an EEG headset and 90.00% physical stress-recognition accuracy using a wristband sensor. These results are promising given that stress recognition with wired physiological devices within a controlled lab setting in the clinical domain has, at best, a similar level of accuracy. The proposed wearable biosensor-based, stress-recognition framework is expected to help us better understand workplace stressors and improve worker safety, health, and productivity through early detection and mitigation of stress at human-centered, smart and connected construction sites.PHDCivil EngineeringUniversity of Michigan, Horace H. Rackham School of Graduate Studieshttps://deepblue.lib.umich.edu/bitstream/2027.42/149965/1/hjebelli_1.pd

Rethinking Eye-blink: Assessing Task Difficulty through Physiological Representation of Spontaneous Blinking

Continuous assessment of task difficulty and mental workload is essential in improving the usability and accessibility of interactive systems. Eye tracking data has often been investigated to achieve this ability, with reports on the limited role of standard blink metrics. Here, we propose a new approach to the analysis of eye-blink responses for automated estimation of task difficulty. The core module is a time-frequency representation of eye-blink, which aims to capture the richness of information reflected on blinking. In our first study, we show that this method significantly improves the sensitivity to task difficulty. We then demonstrate how to form a framework where the represented patterns are analyzed with multi-dimensional Long Short-Term Memory recurrent neural networks for their non-linear mapping onto difficulty-related parameters. This framework outperformed other methods that used hand-engineered features. This approach works with any built-in camera, without requiring specialized devices. We conclude by discussing how Rethinking Eye-blink can benefit real-world applications.Comment: [Accepted version] In Proceedings of CHI Conference on Human Factors in Computing Systems (CHI '21), May 8-13, 2021, Yokohama, Japan. ACM, New York, NY, USA. 19 Pages. https://doi.org/10.1145/3411764.344557

Rethinking Eye-blink: Assessing Task Difficulty through Physiological Representation of Spontaneous Blinking

Continuous assessment of task difficulty and mental workload is essential in improving the usability and accessibility of interactive systems. Eye tracking data has often been investigated to achieve this ability, with reports on the limited role of standard blink metrics. Here, we propose a new approach to the analysis of eye-blink responses for automated estimation of task difficulty. The core module is a time-frequency representation of eye-blink, which aims to capture the richness of information reflected on blinking. In our first study, we show that this method significantly improves the sensitivity to task difficulty. We then demonstrate how to form a framework where the represented patterns are analyzed with multi-dimensional Long Short-Term Memory recurrent neural networks for their non-linear mapping onto difficulty-related parameters. This framework outperformed other methods that used hand-engineered features. This approach works with any built-in camera, without requiring specialized devices. We conclude by discussing how Rethinking Eye-blink can benefit real-world applications