Understanding Ambulatory and Wearable Data for Health and Wellness

In our research, we aim (1) to recognize human internal states and behaviors (stress level, mood and sleep behaviors etc), (2) to reveal which features in which data can work as predictors and (3) to use them for intervention. We collect multi-modal (physiological, behavioral, environmental, and social) ambulatory data using wearable sensors and mobile phones, combining with standardized questionnaires and data measured in the laboratory. In this paper, we introduce our approach and some of our projects

Recognition of Sleep Dependent Memory Consolidation with Multi-modal Sensor Data

This paper presents the possibility of recognizing sleep dependent memory consolidation using multi-modal sensor data. We collected visual discrimination task (VDT) performance before and after sleep at laboratory, hospital and home for N=24 participants while recording EEG (electroencepharogram), EDA (electrodermal activity) and ACC (accelerometer) or actigraphy data during sleep. We extracted features and applied machine learning techniques (discriminant analysis, support vector machine and k-nearest neighbor) from the sleep data to classify whether the participants showed improvement in the memory task. Our results showed 60–70% accuracy in a binary classification of task performance using EDA or EDA+ACC features, which provided an improvement over the more traditional use of sleep stages (the percentages of slow wave sleep (SWS) in the 1st quarter and rapid eye movement (REM) in the 4th quarter of the night) to predict VDT improvement

Effects of an unexpected and expected event on older adults’ autonomic arousal and eye fixations during autonomous driving

© Copyright © 2020 Stephenson, Eimontaite, Caleb-Solly, Morgan, Khatun, Davis and Alford. Driving cessation for some older adults can exacerbate physical, cognitive, and mental health challenges due to loss of independence and social isolation. Fully autonomous vehicles may offer an alternative transport solution, increasing social contact and encouraging independence. However, there are gaps in understanding the impact of older adults’ passive role on safe human–vehicle interaction, and on their well-being. 37 older adults (mean age ± SD = 68.35 ± 8.49 years) participated in an experiment where they experienced fully autonomous journeys consisting of a distinct stop (an unexpected event versus an expected event). The autonomous behavior of the vehicle was achieved using the Wizard of Oz approach. Subjective ratings of trust and reliability, and driver state monitoring including visual attention strategies (fixation duration and count) and physiological arousal (skin conductance and heart rate), were captured during the journeys. Results revealed that subjective trust and reliability ratings were high after journeys for both types of events. During an unexpected stop, overt visual attention was allocated toward the event, whereas during an expected stop, visual attention was directed toward the human–machine interface (HMI) and distributed across the central and peripheral driving environment. Elevated skin conductance level reflecting increased arousal persisted only after the unexpected event. These results suggest that safety-critical events occurring during passive fully automated driving may narrow visual attention and elevate arousal mechanisms. To improve in-vehicle user experience for older adults, a driver state monitoring system could examine such psychophysiological indices to evaluate functional state and well-being. This information could then be used to make informed decisions on vehicle behavior and offer reassurance during elevated arousal during unexpected events

Psychophysiology in games

Psychophysiology is the study of the relationship between psychology and its physiological manifestations. That relationship is of particular importance for both game design and ultimately gameplaying. Players’ psychophysiology offers a gateway towards a better understanding of playing behavior and experience. That knowledge can, in turn, be beneficial for the player as it allows designers to make better games for them; either explicitly by altering the game during play or implicitly during the game design process. This chapter argues for the importance of physiology for the investigation of player affect in games, reviews the current state of the art in sensor technology and outlines the key phases for the application of psychophysiology in games.The work is supported, in part, by the EU-funded FP7 ICT iLearnRWproject (project no: 318803).peer-reviewe

Heterogeneity of psychophysiological stress responses in fibromyalgia syndrome patients

Dysregulated psychophysiological responses have been observed in patients with fibromyalgia syndrome (FMS), although the results are inconsistent. Surface electromyographic (EMG), systolic and diastolic blood pressure, heart rate (HR), and skin conductance levels (SCLs) were continuously recorded at baseline, and during a series of stress and relaxation tasks in 90 FMS patients and 30 age and sex matched healthy controls (HCs). The patient sample demonstrated lower baseline EMG levels compared to the HCs on all tasks. In contrast, the patients displayed elevated HR and SCL (sympathetic vasomotor and sudomotor indices, respectively) during both stress tasks. A cluster analysis identified four psychophysiological response patterns: 63.3% of HCs showed increased muscle tension and stable cardiovascular responses; 34.8% of FMS patients showed a pattern of increased sympathetic vasomotor reactivity with stable sudomotor and reduced muscular response; 12.2% of FMS patients showed a pattern of increased sympathetic sudomotor reactivity connected with increased sympathetic vasomotor response and reduced muscular response; and, in contrast, 46.7% of FMS patients showed a pattern of parasympathetic vasomotor reactivity and reduced sudomotor as well as muscular response. The identification of low baseline muscle tension in FMS is discrepant with other chronic pain syndromes and suggests that unique psychophysiological features may be associated with FMS. The different psychophysiological response patterns within the patient sample support the heterogeneity of FMS

Physiological synchrony in brain and body as a measure of attentional engagement

Attentional engagement – the emotional, cognitive and behavioral connection with information to which the attention is focused – is important in all settings where humans process information. Measures of attentional engagement could be helpful to, for instance, support teachers in online classrooms, or individuals working together in teams. This thesis aims to use physiological synchrony, the similarity in neurophysiological responses across individuals, as an implicit measure of attentional engagement. The research is divided into two parts: the first investigates how different attentional modulations affect physiological synchrony in brains and bodies, the second explores the feasibility of using physiological synchrony as a tool to monitor attention in real-life settings.In Part I, the effect of different manipulations of attention on physiological synchrony in brain and body is explored. We find that physiological synchrony does not only reflect attentional engagement when measured in the electroencephalogram (EEG), but also when measured in electrodermal activity (EDA) or heart rate. Moreover, we find that physiological synchrony can reflect both sensory and top-down variations in attention, where top-down focus of attention is best reflected by synchrony in EEG, and where emotionally salient events attracting attention are best reflected by EDA and heart rate. Part II transitions into the practical applications of physiological synchrony in real-life contexts. Wearables are employed to measure physiological synchrony in EDA and heart rate, demonstrating comparable accuracy to high-end lab-grade equipment. The research also incorporates machine learning techniques, showing that physiological synchrony can be combined with novel unsupervised learning algorithms. Finally, measurements in classrooms reveal that physiological synchrony can be successfully monitored in real-life settings.While the findings are promising, the thesis acknowledges limitations in terms of sufficient data that are required for robust monitoring of attentional engagement and in terms of limited variance in attention explained by physiological synchrony. To advance the field, future work should focus on the applied, methodological and ethical questions that remain unanswered

Exploring digital biomarkers of illness activity in mood episodes:Hypotheses generating and model development study

Background: Depressive and manic episodes within bipolar disorder (BD) and major depressive disorder (MDD) involve altered mood, sleep, and activity, alongside physiological alterations wearables can capture. Objective: Firstly, we explored whether physiological wearable data could predict (aim 1) the severity of an acute affective episode at the intra-individual level and (aim 2) the polarity of an acute affective episode and euthymia among different individuals. Secondarily, we explored which physiological data were related to prior predictions, generalization across patients, and associations between affective symptoms and physiological data.Methods: We conducted a prospective exploratory observational study including patients with BD and MDD on acute affective episodes (manic, depressed, and mixed) whose physiological data were recorded using a research-grade wearable (Empatica E4) across 3 consecutive time points (acute, response, and remission of episode). Euthymic patients and healthy controls were recorded during a single session (approximately 48 h). Manic and depressive symptoms were assessed using standardized psychometric scales. Physiological wearable data included the following channels: acceleration (ACC), skin temperature, blood volume pulse, heart rate (HR), and electrodermal activity (EDA). Invalid physiological data were removed using a rule-based filter, and channels were time aligned at 1-second time units and segmented at window lengths of 32 seconds, as best-performing parameters. We developed deep learning predictive models, assessed the channels’ individual contribution using permutation feature importance analysis, and computed physiological data to psychometric scales’ items normalized mutual information (NMI). We present a novel, fully automated method for the preprocessing and analysis of physiological data from a research-grade wearable device, including a viable supervised learning pipeline for time-series analyses.Results: Overall, 35 sessions (1512 hours) from 12 patients (manic, depressed, mixed, and euthymic) and 7 healthy controls (mean age 39.7, SD 12.6 years; 6/19, 32% female) were analyzed. The severity of mood episodes was predicted with moderate (62%-85%) accuracies (aim 1), and their polarity with moderate (70%) accuracy (aim 2). The most relevant features for the former tasks were ACC, EDA, and HR. There was a fair agreement in feature importance across classification tasks (Kendall W=0.383). Generalization of the former models on unseen patients was of overall low accuracy, except for the intra-individual models. ACC was associated with “increased motor activity” (NMI>0.55), “insomnia” (NMI=0.6), and “motor inhibition” (NMI=0.75). EDA was associated with “aggressive behavior” (NMI=1.0) and “psychic anxiety” (NMI=0.52).Conclusions: Physiological data from wearables show potential to identify mood episodes and specific symptoms of mania and depression quantitatively, both in BD and MDD. Motor activity and stress-related physiological data (EDA and HR) stand out as potential digital biomarkers for predicting mania and depression, respectively. These findings represent a promising pathway toward personalized psychiatry, in which physiological wearable data could allow the early identification and intervention of mood episodes

Psychophysiology in the digital age

The research I performed for my thesis revolved around the question how affect-physiology dynamics can be best measured in daily life. In my thesis I focused on three aspects of this question: 1) Do wearable wristband devices have sufficient validity to capture ANS activity? 2) To what extent is the laboratory design suitable to measure affect-ANS dynamics? 3) Are the affect-ANS dynamics subject to individual differences, both in the laboratory and in daily life? In chapter 2, I validated a shortened version of the Sing-a-Song Stress (SSST) test, the SSSTshort. The purpose of this test is to create social-evaluative stress in participants through a simple and brief design that does not require the involvement of multiple confederates. The results indicated that the SSSTshort was effective in inducing ANS and affective reactivity. This makes the SSSTshort a cost-effective alternative to the well-known Trier-Social-Stress task (TSST), which can be easily incorporated into large-scale studies to expand the range of stress types that can be studied in laboratory designs. In chapter 3, I validated a new wrist worn technology for measuring electrodermal activity (EDA). As expected, the overall EDA levels measured on the wrist were lower than those measured on the palm, likely due to the lower density of sweat glands on the wrist. The analysis demonstrated that the frequency measure of non-specific skin conductance response (ns.SCR) was superior to the commonly used measure of skin conductance level (SCL) for both the palm and wrist. The wrist-based ns.SCR measure was sensitive to the experimental manipulations and showed similar correspondence to the pre-ejection period (PEP) as palm-based ns.SCR. Moreover, wrist-based ns.SCR demonstrated similar predictive validity for affective state as PEP. However, the predictive validity of both wrist-based ns.SCR and PEP was lower compared to palm-based ns.SCR. These findings suggest that wrist-based ns.SCR EDA parameter has a promising future for use in psychophysiological research. In Chapter 4 of my thesis, I conducted the first study to directly compare the relationship between affect and ANS activity in a laboratory setting to that in daily life. To elicit stress in the laboratory, four different stress paradigms were employed, while stressful events in daily life were left to chance. In both settings, a valence and arousal scale was constructed from a nine-item affect questionnaire, and ANS activity was collected using the same devices. Data was collected from a single population, and the affect-ANS dynamics were analyzed using the same methodology for both laboratory and daily life settings. The results showed a remarkable similarity between the laboratory and daily life affect-ANS relationships. In Chapter 5 of my thesis, I investigated the influence of individual differences in physical activity and aerobic fitness on ANS and affective stress reactivity. Previous research has yielded inconsistent results due to heterogeneity issues in the population studied, stressor type, and the way fitness was measured. My study made a unique contribution to this field by measuring physical activity in three ways: 1) as objective aerobic fitness, 2) leisure time exercise behavior, and 3) total moderate-to-vigorous exercise (including both exercise and all other regular physical activity behaviors). In addition, we measured the physiological and affective stress response in both a laboratory and daily life setting. The total amount of physical activity showed more relationships with stress reactivity compared to exercise behavior alone, suggesting that future research should include a total physical activity variable. Our results did not support the cross-stressor adaptation hypotheses, suggesting that if exercise has a stress-reducing effect, it is unlikely to be mediated by altered ANS regulation due to repeated exposure to physical stress

Stressful first impressions in job interviews

Stress can impact many aspects of our lives, such as the way we interact and work with others, or the first impressions that we make. In the past, stress has been most commonly assessed through self-reported questionnaires; however, advancements in wearable technology have enabled the measurement of physiological symptoms of stress in an unobtrusive manner. Using a dataset of job interviews, we investigate whether first impressions of stress (from annotations) are equivalent to physiological measurements of the electrodermal activity (EDA). We examine the use of automatically extracted nonverbal cues stemming from both the visual and audio modalities, as well EDA stress measurements for the inference of stress impressions obtained from manual annotations. Stress impressions were found to be significantly negatively correlated with hireability ratings i.e individuals who were perceived to be more stressed were more likely to obtained lower hireability scores. The analysis revealed a significant relationship between audio and visual features but low predictability and no significant effects were found for the EDA features. While some nonverbal cues were more clearly related to stress, the physiological cues were less reliable and warrant further investigation into the use of wearable sensors for stress detection