Emotions in context: examining pervasive affective sensing systems, applications, and analyses

Pervasive sensing has opened up new opportunities for measuring our feelings and understanding our behavior by monitoring our affective states while mobile. This review paper surveys pervasive affect sensing by examining and considering three major elements of affective pervasive systems, namely; “sensing”, “analysis”, and “application”. Sensing investigates the different sensing modalities that are used in existing real-time affective applications, Analysis explores different approaches to emotion recognition and visualization based on different types of collected data, and Application investigates different leading areas of affective applications. For each of the three aspects, the paper includes an extensive survey of the literature and finally outlines some of challenges and future research opportunities of affective sensing in the context of pervasive computing

Biosensing and Actuation—Platforms Coupling Body Input-Output Modalities for Affective Technologies

Research in the use of ubiquitous technologies, tracking systems and wearables within mental health domains is on the rise. In recent years, affective technologies have gained traction and garnered the interest of interdisciplinary fields as the research on such technologies matured. However, while the role of movement and bodily experience to affective experience is well-established, how to best address movement and engagement beyond measuring cues and signals in technology-driven interactions has been unclear. In a joint industry-academia effort, we aim to remodel how affective technologies can help address body and emotional self-awareness. We present an overview of biosignals that have become standard in low-cost physiological monitoring and show how these can be matched with methods and engagements used by interaction designers skilled in designing for bodily engagement and aesthetic experiences. Taking both strands of work together offers unprecedented design opportunities that inspire further research. Through first-person soma design, an approach that draws upon the designer’s felt experience and puts the sentient body at the forefront, we outline a comprehensive work for the creation of novel interactions in the form of couplings that combine biosensing and body feedback modalities of relevance to affective health. These couplings lie within the creation of design toolkits that have the potential to render rich embodied interactions to the designer/user. As a result we introduce the concept of “orchestration”. By orchestration, we refer to the design of the overall interaction: coupling sensors to actuation of relevance to the affective experience; initiating and closing the interaction; habituating; helping improve on the users’ body awareness and engagement with emotional experiences; soothing, calming, or energising, depending on the affective health condition and the intentions of the designer. Through the creation of a range of prototypes and couplings we elicited requirements on broader orchestration mechanisms. First-person soma design lets researchers look afresh at biosignals that, when experienced through the body, are called to reshape affective technologies with novel ways to interpret biodata, feel it, understand it and reflect upon our bodies

MITIGATING PUBLIC SPEAKING ANXIETY USING VIRTUAL REALITY AND POPULATION-SPECIFIC MODELS

In the education and workplace landscape of the 21st century, it is often said that a person is only as valuable as the ideas s/he has and can share. Public speaking skills are essential to help people effectively exchange ideas, persuade, inform their audiences as well as make a tangible impact. They also plays a vital role in one’s academic and professional success. However, research shows that public speaking anxiety (PSA) ranks as a top social phobia among many people and tends to be aggravated in minorities, first generation students, and non-native speakers. This research aims at mitigating this anxiety by utilizing physiological (cardiovascular activity, electrodermal activity etc.) and acoustic (pitch, intonation, etc.) indices captured from wearable devices and virtual reality (VR) interfaces to quantify and predict PSA. This work also examines the significance of individual-specific factors, such as general trait anxiety and personality metrics, as well as contextual factors, such as age, gender, highest education, and native language, receny of public speaking in moderating the association between bio-behavioural (physiological and acoustic) indices and PSA. The individual-specific information is used to develop population-specific machine learning models of PSA. Results of this research highlight the importance of including such factors for detecting PSA with the proposed population-based PSA models yielding Spearman’s correlation of 0.55 n(p < 0.05) between the actual and predicted state-based scores. This work further analyzes whether systematic exposure to public speaking tasks in a VR environment can help alleviate PSA. Results indicate that systematic exposure to public speaking in VR can alleviate PSA in terms of both self-reported (p < 0.05) and physiological (p < 0.05) indices. Findings of this study will enable researchers to better understand antedecedents and causes of PSA as well as lay the foundation toward developing adaptive behavioural interventions for social communication disorders using systematic exposure (e.g., through VR stimuli), relaxation feedback, and cognitive restructuring

Frisson Waves: Exploring Automatic Detection, Triggering and Sharing of Aesthetic Chills in Music Performances

Frisson is the feeling and experience of physical reactions such as shivers, tingling skin, and goosebumps. Using entrainment through facilitating interpersonal transmissions of embodied sensations, we present "Frisson Waves" with the aim to enhance live music performance experiences. "Frisson Waves" is an exploratory real-time system to detect, trigger and share frisson in a wave-like pattern over audience members during music performances. The system consists of a physiological sensing wristband for detecting frisson and a thermo-haptic neckband for inducing frisson. In a controlled environment, we evaluate detection (n=19) and triggering of frisson (n=15). Based on our findings, we conducted an in-the-wild music concert with 48 audience members using our system to share frisson. This paper summarizes a framework for accessing, triggering and sharing frisson. We report our research insights, lessons learned, and limitations of "Frisson Waves". Yan He, George Chernyshov, Jiawen Han, Dingding Zheng, Ragnar Thomsen, Danny Hynds, Muyu Liu, Yuehui Yang, Yulan Ju, Yun Suen Pai, Kouta Minamizawa, Kai Kunze, and Jamie A War

Ecological Momentary Assessment in Internet-Delivered Psychological Treatments using Wearable Technology

The growing prevalence of mental health problems is a global concern. Current psychological treatments are effective for a wide range of mental health problems. Yet, treatments today fall short with regards to scalability and struggle to meet the demand for help. To treat patients in a more cost-effective, accessible, and scalable manner, InternetDelivered Psychological Treatment (IDPT) has posed as a promising solution. Although, IDPT has shown encouraging results, the technology falls short in some regards. One such shortcoming is low user adherence. Adaptive IDPT that allow for personalizing treatment to patient needs may help solve the issue of high drop-out rates in IDPT as they are thought to aid in increasing user adherence. Yet, to adapt and personalize treatment there is a need of meaningful data about patients. In this thesis, we have created an artifact for the use of wearable data in IDPT. More specifically, our artifact can be split in two parts: (1) an extension of an IDPT framework that serves as a general component and allows for the utilization of wearable data to support Ecological Momentary Assessment (EMA) and (2) a demonstrative component that provides an example of how wearable data may be utilized in interventions to support adaptation. We have created an artifact, comprised of these two components, according to the design science research methodology. Through semi-structured interviews with domain experts of electrical engineering and psychology our artifact has been evaluated. As a result of this evaluation, we have learned that our artifact can serve as a basis for future research.Masteroppgave i Programutvikling samarbeid med HVLPROG399MAMN-PRO

Feature Space Augmentation: Improving Prediction Accuracy of Classical Problems in Cognitive Science and Computer Vison

The prediction accuracy in many classical problems across multiple domains has seen a rise since computational tools such as multi-layer neural nets and complex machine learning algorithms have become widely accessible to the research community. In this research, we take a step back and examine the feature space in two problems from very different domains. We show that novel augmentation to the feature space yields higher performance. Emotion Recognition in Adults from a Control Group: The objective is to quantify the emotional state of an individual at any time using data collected by wearable sensors. We define emotional state as a mixture of amusement, anger, disgust, fear, sadness, anxiety and neutral and their respective levels at any time. The generated model predicts an individual’s dominant state and generates an emotional spectrum, 1x7 vector indicating levels of each emotional state and anxiety. We present an iterative learning framework that alters the feature space uniquely to an individual’s emotion perception, and predicts the emotional state using the individual specific feature space. Hybrid Feature Space for Image Classification: The objective is to improve the accuracy of existing image recognition by leveraging text features from the images. As humans, we perceive objects using colors, dimensions, geometry and any textual information we can gather. Current image recognition algorithms rely exclusively on the first 3 and do not use the textual information. This study develops and tests an approach that trains a classifier on a hybrid text based feature space that has comparable accuracy to the state of the art CNN’s while being significantly inexpensive computationally. Moreover, when combined with CNN’S the approach yields a statistically significant boost in accuracy. Both models are validated using cross validation and holdout validation, and are evaluated against the state of the art

Urban Emotions and Cycling Experience – Enriching Traffic Planning for Cyclists with Human Sensor Data

Even though much research has been conducted on the safety of cycling infrastructures, most previous approaches only make use of traditional and proven methods based upon datasets such as accident statistics, road infrastructure data, or questionnaires. Apart from typical surveys, which are known to face numerous limitations from a psychological and sociological viewpoints, the question of how perceived safety can best be assessed is still widely unexplored. Thus, this paper presents an approach for bio-physiological sensing to identify places in urban environments which are perceived as unsafe by cyclists. Specifically, a number of physiological parameters like ECG, skin conductance, skin temperature and heart rate variability are analysed to identify moments of stress. Together with data gathered through a People as Sensors app, these stress levels can be mapped to specific emotions. This method was tested in a pilot study in Cambridge, MA (USA), which is presented in this paper. Our findings show that our method can identify places with emotional peaks, particularly fear and anger. Although our results can be qualitatively interpreted and used in urban planning, more research is necessary to quantitatively and automatically generate recommendations from the measurements for urban planners

Psychophysiology in the digital age

The research I performed for my thesis revolved around the question how affect-physiology dynamics can be best measured in daily life. In my thesis I focused on three aspects of this question: 1) Do wearable wristband devices have sufficient validity to capture ANS activity? 2) To what extent is the laboratory design suitable to measure affect-ANS dynamics? 3) Are the affect-ANS dynamics subject to individual differences, both in the laboratory and in daily life? In chapter 2, I validated a shortened version of the Sing-a-Song Stress (SSST) test, the SSSTshort. The purpose of this test is to create social-evaluative stress in participants through a simple and brief design that does not require the involvement of multiple confederates. The results indicated that the SSSTshort was effective in inducing ANS and affective reactivity. This makes the SSSTshort a cost-effective alternative to the well-known Trier-Social-Stress task (TSST), which can be easily incorporated into large-scale studies to expand the range of stress types that can be studied in laboratory designs. In chapter 3, I validated a new wrist worn technology for measuring electrodermal activity (EDA). As expected, the overall EDA levels measured on the wrist were lower than those measured on the palm, likely due to the lower density of sweat glands on the wrist. The analysis demonstrated that the frequency measure of non-specific skin conductance response (ns.SCR) was superior to the commonly used measure of skin conductance level (SCL) for both the palm and wrist. The wrist-based ns.SCR measure was sensitive to the experimental manipulations and showed similar correspondence to the pre-ejection period (PEP) as palm-based ns.SCR. Moreover, wrist-based ns.SCR demonstrated similar predictive validity for affective state as PEP. However, the predictive validity of both wrist-based ns.SCR and PEP was lower compared to palm-based ns.SCR. These findings suggest that wrist-based ns.SCR EDA parameter has a promising future for use in psychophysiological research. In Chapter 4 of my thesis, I conducted the first study to directly compare the relationship between affect and ANS activity in a laboratory setting to that in daily life. To elicit stress in the laboratory, four different stress paradigms were employed, while stressful events in daily life were left to chance. In both settings, a valence and arousal scale was constructed from a nine-item affect questionnaire, and ANS activity was collected using the same devices. Data was collected from a single population, and the affect-ANS dynamics were analyzed using the same methodology for both laboratory and daily life settings. The results showed a remarkable similarity between the laboratory and daily life affect-ANS relationships. In Chapter 5 of my thesis, I investigated the influence of individual differences in physical activity and aerobic fitness on ANS and affective stress reactivity. Previous research has yielded inconsistent results due to heterogeneity issues in the population studied, stressor type, and the way fitness was measured. My study made a unique contribution to this field by measuring physical activity in three ways: 1) as objective aerobic fitness, 2) leisure time exercise behavior, and 3) total moderate-to-vigorous exercise (including both exercise and all other regular physical activity behaviors). In addition, we measured the physiological and affective stress response in both a laboratory and daily life setting. The total amount of physical activity showed more relationships with stress reactivity compared to exercise behavior alone, suggesting that future research should include a total physical activity variable. Our results did not support the cross-stressor adaptation hypotheses, suggesting that if exercise has a stress-reducing effect, it is unlikely to be mediated by altered ANS regulation due to repeated exposure to physical stress

Using Deep Convolutional Neural Network for Emotion Detection on a Physiological Signals Dataset (AMIGOS)

Recommender systems have been based on context and content, and now the technological challenge of making personalized recommendations based on the user emotional state arises through physiological signals that are obtained from devices or sensors. This paper applies the deep learning approach using a deep convolutional neural network on a dataset of physiological signals (electrocardiogram and galvanic skin response), in this case, the AMIGOS dataset. The detection of emotions is done by correlating these physiological signals with the data of arousal and valence of this dataset, to classify the affective state of a person. In addition, an application for emotion recognition based on classic machine learning algorithms is proposed to extract the features of physiological signals in the domain of time, frequency, and non-linear. This application uses a convolutional neural network for the automatic feature extraction of the physiological signals, and through fully connected network layers, the emotion prediction is made. The experimental results on the AMIGOS dataset show that the method proposed in this paper achieves a better precision of the classification of the emotional states, in comparison with the originally obtained by the authors of this dataset.This research project is financed by theGovernment of Colombia, Colciencias and the Governorateof Boyac