Computational and Causal Approaches on Social Media and Multimodal Sensing Data: Examining Wellbeing in Situated Contexts

A core aspect of our lives is often embedded in the communities we are situated in. The interconnectedness of our interactions and experiences intertwines our situated context with our wellbeing. A better understanding of wellbeing will help us devise proactive and tailored support strategies. However, existing methodologies to assess wellbeing suffer from limitations of scale and timeliness. These limitations are surmountable by social and ubiquitous technologies. Given its ubiquity and wide use, social media can be considered a “passive sensor” that can act as a complementary source of unobtrusive, real-time, and naturalistic data to infer wellbeing. This dissertation leverages social media in concert with multimodal sensing data, which facilitate analyzing dense and longitudinal behavior at scale. This work adopts machine learning, natural language, and causal inference analysis to infer wellbeing of individuals and collectives, particularly in situated communities, such as college campuses and workplaces. Before incorporating sensing modalities in practice, we need to account for confounds. One such confound that might impact behavior change is the phenomenon of “observer effect” --- that individuals may deviate from their typical or otherwise normal behavior because of the awareness of being “monitored”. I study this problem by leveraging the potential of longitudinal and historical behavioral data through social media. Focused on a multimodal sensing study, I conduct a causal study to measure observer effect in social media behavior, and explain the observations through existing theory in psychology and social science. The findings provide recommendations to correcting biases due to observer effect in social media sensing for human behavior and wellbeing. The novelties and contributions of this dissertation are four-fold. First, I use social media data that uniquely captures the behavior of situated communities. Second, I adopt theory-driven computational and causal methods to make conclusive research claims on wellbeing dynamics. Third, I address major challenges with methods to combine social media with multimodal sensing data for a comprehensive understanding of human behavior. Fourth, I draw interpretations and explanations of online-data-driven offline inferences. This dissertation situates the findings in an interdisciplinary context, including psychology and social science, and bears implications from theoretical, practical, design, methodological, and ethical perspectives catering to various stakeholders, including researchers, practitioners, and policymakers.Ph.D

Eating Behavior In-The-Wild and Its Relationship to Mental Well-Being

The motivation for eating is beyond survival. Eating serves as means for socializing, exploring cultures, etc. Computing researchers have developed various eating detection technologies that can leverage passive sensors available on smart devices to automatically infer when and, to some extent, what an individual is eating. However, despite their significance in eating literature, crucial contextual information such as meal company, type of food, location of meals, the motivation of eating episodes, the timing of meals, etc., are difficult to detect through passive means. More importantly, the applications of currently developed automated eating detection systems are limited. My dissertation addresses several of these challenges by combining the strengths of passive sensing technologies and EMAs (Ecological Momentary Assessment). EMAs are a widely adopted tool used across a variety of disciplines that can gather in-situ information about individual experiences. In my dissertation, I demonstrate the relationship between various eating contexts and the mental well-being of college students and information workers through naturalistic studies. The contributions of my dissertation are four-fold. First, I develop a real-time meal detection system that can detect meal-level episodes and trigger EMAs to gather contextual data about one’s eating episode. Second, I deploy this system in a college student population to understand their eating behavior during day-to-day life and investigate the relationship of these eating behaviors with various mental well-being outcomes. Third, based on the limitations of passive sensing systems to detect short and sporadic chewing episodes present in snacking, I develop a snacking detection system and operationalize the definition of snacking in this thesis. Finally, I investigate the causal relationship between stress levels experienced by remote information workers during their workdays and its effect on lunchtime. This dissertation situates the findings in an interdisciplinary context, including ubiquitous computing, psychology, and nutrition.Ph.D

Inferring Mood-While-Eating with Smartphone Sensing and Community-Based Model Personalization

The interplay between mood and eating has been the subject of extensive research within the fields of nutrition and behavioral science, indicating a strong connection between the two. Further, phone sensor data have been used to characterize both eating behavior and mood, independently, in the context of mobile food diaries and mobile health applications. However, limitations within the current body of literature include: i) the lack of investigation around the generalization of mood inference models trained with passive sensor data from a range of everyday life situations, to specific contexts such as eating, ii) no prior studies that use sensor data to study the intersection of mood and eating, and iii) the inadequate examination of model personalization techniques within limited label settings, as we commonly experience in mood inference. In this study, we sought to examine everyday eating behavior and mood using two datasets of college students in Mexico (N_mex = 84, 1843 mood-while-eating reports) and eight countries (N_mul = 678, 329K mood reports incl. 24K mood-while-eating reports), containing both passive smartphone sensing and self-report data. Our results indicate that generic mood inference models decline in performance in certain contexts, such as when eating. Additionally, we found that population-level (non-personalized) and hybrid (partially personalized) modeling techniques were inadequate for the commonly used three-class mood inference task (positive, neutral, negative). Furthermore, we found that user-level modeling was challenging for the majority of participants due to a lack of sufficient labels and data from the negative class. To address these limitations, we employed a novel community-based approach for personalization by building models with data from a set of similar users to a target user

Psychological research in the digital age

The smartphone has become an important personal companion in our daily lives. Each time we use the device, we generate data that provides information about ourselves. This data, in turn, is valuable to science because it objectively reflects our everyday behavior and experiences. In this way, smartphones enable research that is closer to everyday life than traditional laboratory experiments and questionnaire-based methods. While data collected with smartphones are increasingly being used in the field of personality psychology, new digital technologies can also be leveraged to collect and analyze large-scale unobtrusively sensed data in other areas of psychological research. This dissertation, therefore, explores the insights that smartphone sensing reveals for psychological research using two examples, situation and affect research, making a twofold research contribution. First, in two empirical studies, different data types of smartphone-sensed data, such as GPS or phone data, were combined with experience-sampled self-report, and classical questionnaire data to gain valuable insights into individual behavior, thinking, and feeling in everyday life. Second, predictive modeling techniques were applied to analyze the large, high-dimensional data sets collected by smartphones. To gain a deeper understanding of the smartphone data, interpretable variables were extracted from the raw sensing data, and the predictive performance of various machine learning algorithms was compared. In summary, the empirical findings suggest that smartphone data can effectively capture certain situational and behavioral indicators of psychological phenomena in everyday life. However, in certain research areas such as affect research, smartphone data should only complement, but not completely replace, traditional questionnaire-based data as well as other data sources such as neurophysiological indicators. The dissertation also concludes that the use of smartphone sensor data introduces new difficulties and challenges for psychological research and that traditional methods and perspectives are reaching their limits. The complexity of data collection, processing, and analysis requires established guidelines for study design, interdisciplinary collaboration, and theory-driven research that integrates explanatory and predictive approaches. Accordingly, further research is needed on how machine learning models and other big data methods in psychology can be reconciled with traditional theoretical approaches. Only in this way can we move closer to the ultimate goal of psychology to better understand, explain, and predict human behavior and experiences and their interplay with everyday situations

Developing Transferable Deep Models for Mobile Health

Human behavior is one of the key facets of health. A major portion of healthcare spending in the US is attributed to chronic diseases, which are linked to behavioral risk factors such as smoking, drinking, unhealthy eating. Mobile devices that are integrated into people's everyday lives make it possible for us to get a closer look into behavior. Two of the most commonly used sensing modalities include Ecological Momentary Assessments (EMAs): surveys about mental states, environment, and other factors, and wearable sensors that are used to capture high frequency contextual and physiological signals. One of the main visions of mobile health (mHealth) is sensor-based behavior modification. Contextual data collected from participants is typically used to train a risk prediction model for adverse events such as smoking, which can then be used to inform intervention design. However, there are several choices in an mHealth study such as the demographics of the participants in the study, the type of sensors used, the questions included in the EMA. This results in two technical challenges to using machine learning models effectively across mHealth studies. The first is the problem of domain shift where the data distribution varies across studies. This would result in models trained on one study to have sub-optimal performance on a different study. Domain shift is common in wearable sensor data since there are several sources of variability such as sensor design, the placement of the sensor on the body, demographics of the users, etc. The second challenge is that of covariate-space shift where the input-space changes across datasets. This is common across EMA datasets since questions can vary based on the study. This thesis studies the problem of covariate-space shift and domain shift in mHealth data. First, I study the problem of domain shift caused by differences in the sensor type and placement in ECG and PPG signals. I propose a self-supervised learning based domain adaptation method that captures the physiological structure of these signals to improve transfer performance of predictive models. Second, I present a method to find a common input representation irrespective of the fine-grained questions in EMA datasets to overcome the problem of covariate-space shift. The next challenge to the deployment of ML models in health is explainability. I explore the problem of bridging the gap between explainability methods and domain experts and present a method to generate plausible, relevant, and convincing explanations.Ph.D

Quantifying Quality of Life

Describes technological methods and tools for objective and quantitative assessment of QoL Appraises technology-enabled methods for incorporating QoL measurements in medicine Highlights the success factors for adoption and scaling of technology-enabled methods This open access book presents the rise of technology-enabled methods and tools for objective, quantitative assessment of Quality of Life (QoL), while following the WHOQOL model. It is an in-depth resource describing and examining state-of-the-art, minimally obtrusive, ubiquitous technologies. Highlighting the required factors for adoption and scaling of technology-enabled methods and tools for QoL assessment, it also describes how these technologies can be leveraged for behavior change, disease prevention, health management and long-term QoL enhancement in populations at large. Quantifying Quality of Life: Incorporating Daily Life into Medicine fills a gap in the field of QoL by providing assessment methods, techniques and tools. These assessments differ from the current methods that are now mostly infrequent, subjective, qualitative, memory-based, context-poor and sparse. Therefore, it is an ideal resource for physicians, physicians in training, software and hardware developers, computer scientists, data scientists, behavioural scientists, entrepreneurs, healthcare leaders and administrators who are seeking an up-to-date resource on this subject

Amygdala Neurofeedback Training in Borderline Personality Disorder: Capturing Improvements in Emotion Regulation

The way we regulate emotions is a powerful determinant of behavior and directly impacts affect and physiology. Many mental disorders, such as borderline personality disorder, are in large part disorders of emotion dysregulation. Because of its important role in mental health, research has endeavored to understand the mechanisms and biological underpinnings of emotion regulation and to create trainings and specific clinical programs that aim to augment the ability to regulate emotions. The assessment of psychophysiological responses represents an important complementary method to quantify emotion regulation in both studies on healthy individuals and studies assessing clinical emotion regulation trainings. However, psychophysiological effects have been inconsistent across literature, which impedes informed decisions about suitable psychophysiological variables of emotion regulation experiments and clinical trainings. A new technique assumed to improve emotion regulation is amygdala neurofeedback training. Because patients with borderline personality disorder show hyperreactivity of the amygdala likely underlying the severe emotion regulation problems they suffer from, amygdala neurofeedback training may be a candidate training to improve emotion regulation in these patients. Until now, it has been unclear which aspects of psychopathology and emotion regulation may change with neurofeedback-aided amygdala downregulation in borderline personality disorder, which is urgently needed to determine a primary outcome measure for future randomized controlled trails. To fill these gaps, the present doctoral thesis identified the effects of psychophysiological responses of emotion regulation as well as important moderators and identified primary outcome measures of emotion dysregulation after neurofeedback training in patients with borderline personality disorder. In total, three studies were conducted. In Study I, a total of 1353 studies on psychophysiological responses of emotion regulation were screened through a systematic search of articles and meta-analyses were used to evaluate effect sizes of instructed downregulation strategies on common autonomic and electromyographic measures. Following this, Study II systematically tested effects of the startle probe timing on startle responses during emotion regulation in 47 healthy individuals. Study II aimed at optimizing emotion regulation assessment with the emotion-modulated startle that was then used in Study III. In Study III, a four-session amygdala neurofeedback training was tested in 24 female patients with borderline personality disorder. Before and after the neurofeedback training, as well as at a 6-week follow-up assessment, measures of emotion dysregulation and borderline personality disorder psychopathology were tested at diverse levels of analysis. Results from Study I demonstrate that effects of emotion regulation on autonomic measures, even if significant, were small and heterogeneous across studies, while electromyographic measures were more homogeneous and revealed medium effect sizes. Important study characteristics such as the study design, control instruction and trial duration moderated some autonomic effects of suppression and reappraisal. Study II demonstrated a significant inhibition of the startle response with emotion downregulation. Startle probes delivered at >7 seconds into the regulation phase were useful to quantify reappraisal effects, although earlier probes did not yield significantly smaller effects. Finally, Study III demonstrated that the inhibition of the startle with emotion downregulation increased after the training, suggesting improved emotion regulation abilities. In addition, we could demonstrate that general BPD psychopathology as well as affective instability and negative affect in daily life improved after training. However, after correction for multiple comparisons, observed effect sizes did not surpass the significance level and some effects (e.g., startle) faded to the 6-week follow-up assessment. In sum, the present thesis provides the groundwork for future randomized controlled trials of amygdala neurofeedback training and enables future laboratory and clinical studies to gain more stable effects in psychophysiological measurements of emotion regulation. In particular, the findings implicate that with regard to emotion regulation research, autonomic measures appear to be highly variable and thus should be selected carefully. In addition, we need more comparable psychophysiological set-ups in the empirical study of emotion regulation. The emotion-modulated startle not only proved to be a robust measure to quantify emotion regulation effects in general, but also appeared to be suitable to track improvements in emotion regulation in the context of a neurofeedback training targeting emotion dysregulation. With respect to emotion regulation outcome measures for future amygdala neurofeedback studies, further improvement of the specific paradigms is needed. In addition, the neurofeedback training itself should be optimized in terms of e.g. training time and booster sessions. Future placebo-controlled trials must then confirm that the treatment is effective in improving emotion regulation in those with borderline personality disorder

Quantifying Quality of Life

Describes technological methods and tools for objective and quantitative assessment of QoL Appraises technology-enabled methods for incorporating QoL measurements in medicine Highlights the success factors for adoption and scaling of technology-enabled methods This open access book presents the rise of technology-enabled methods and tools for objective, quantitative assessment of Quality of Life (QoL), while following the WHOQOL model. It is an in-depth resource describing and examining state-of-the-art, minimally obtrusive, ubiquitous technologies. Highlighting the required factors for adoption and scaling of technology-enabled methods and tools for QoL assessment, it also describes how these technologies can be leveraged for behavior change, disease prevention, health management and long-term QoL enhancement in populations at large. Quantifying Quality of Life: Incorporating Daily Life into Medicine fills a gap in the field of QoL by providing assessment methods, techniques and tools. These assessments differ from the current methods that are now mostly infrequent, subjective, qualitative, memory-based, context-poor and sparse. Therefore, it is an ideal resource for physicians, physicians in training, software and hardware developers, computer scientists, data scientists, behavioural scientists, entrepreneurs, healthcare leaders and administrators who are seeking an up-to-date resource on this subject

Characterization of Time-variant and Time-invariant Assessment of Suicidality on Reddit using C-SSRS

Suicide is the 10th leading cause of death in the U.S (1999-2019). However, predicting when someone will attempt suicide has been nearly impossible. In the modern world, many individuals suffering from mental illness seek emotional support and advice on well-known and easily-accessible social media platforms such as Reddit. While prior artificial intelligence research has demonstrated the ability to extract valuable information from social media on suicidal thoughts and behaviors, these efforts have not considered both severity and temporality of risk. The insights made possible by access to such data have enormous clinical potential - most dramatically envisioned as a trigger to employ timely and targeted interventions (i.e., voluntary and involuntary psychiatric hospitalization) to save lives. In this work, we address this knowledge gap by developing deep learning algorithms to assess suicide risk in terms of severity and temporality from Reddit data based on the Columbia Suicide Severity Rating Scale (C-SSRS). In particular, we employ two deep learning approaches: time-variant and time-invariant modeling, for user-level suicide risk assessment, and evaluate their performance against a clinician-adjudicated gold standard Reddit corpus annotated based on the C-SSRS. Our results suggest that the time-variant approach outperforms the time-invariant method in the assessment of suicide-related ideations and supportive behaviors (AUC:0.78), while the time-invariant model performed better in predicting suicide-related behaviors and suicide attempt (AUC:0.64). The proposed approach can be integrated with clinical diagnostic interviews for improving suicide risk assessments.Comment: 24 Pages, 8 Tables, 6 Figures; Accepted by PLoS One ; One of the two mentioned Datasets in the manuscript has Closed Access. We will make it public after PLoS One produces the manuscrip