Evaluation of Methods for Data-Driven Tools that Empower Mental Health Professionals

It is estimated that nearly one in five adults in the United States live with mental illness, and for individuals who struggle with mental health, the experience can be excruciating. The rise of mobile devices presents a unique opportunity to improve mental health outcomes, in part through empowering mental health professionals. Because many individuals always have their smartphones with them, smartphones may be able to enable health professionals to identify when an individual is in need and provide immediate care, rather than the current model of delaying care until a scheduled appointment. To this end, I investigate the feasibility of two new data-driven tools: one to identify when care is needed and the second to help train counselors to intervene when care is needed. The first tool I consider seeks to use a smartphone to sense an individual's well-being. Such a tool could be used to inform health professionals of patients' states, evaluate the efficacy of therapies, and deliver just-in-time interventions. To evaluate the potential accuracy of such a tool, I collect students' passive smartphone data and self-reported well-being measures, and then consider predicting well-being on a daily basis and detecting significant changes over a period of time. As this approach seems unreliable for most individuals, I further explore for which individuals such an approach may be reliable and develop a framework for evaluating longitudinal sensing quality. I find that while correlations between smartphone-sensed measures and reported wellbeing scores exist, these relationships are often too weak to reliably predict wellbeing. The second tool I explore seeks to help suicide prevention counselors practice intervening over chat for individuals in crisis. For this, I collect and leverage synthetic conversation transcripts and show how to evaluate a baseline system for counselors to practice crisis de-escalation strategies in a no-risk environment. While text retrieval and generation methods can return responses that make sense in limited context most of the time, i.e., in greater that 50% of examples, generated responses are shorter than retrieving full messages, implying that generation may potentially be a less engaging approach. Overall, I find that significant consideration of context is needed to provide meaningful evaluation of methods for the tools envisioned. While popular algorithms and methods may hold potential to develop the tools discussed, rigorous evaluation and further work is needed to ensure reliability within the application context

Evaluation of Methods for Data-Driven Tools that Empower Mental Health Professionals

It is estimated that nearly one in five adults in the United States live with mental illness, and for individuals who struggle with mental health, the experience can be excruciating. The rise of mobile devices presents a unique opportunity to improve mental health outcomes, in part through empowering mental health professionals. Because many individuals always have their smartphones with them, smartphones may be able to enable health professionals to identify when an individual is in need and provide immediate care, rather than the current model of delaying care until a scheduled appointment. To this end, I investigate the feasibility of two new data-driven tools: one to identify when care is needed and the second to help train counselors to intervene when care is needed. The first tool I consider seeks to use a smartphone to sense an individual's well-being. Such a tool could be used to inform health professionals of patients' states, evaluate the efficacy of therapies, and deliver just-in-time interventions. To evaluate the potential accuracy of such a tool, I collect students' passive smartphone data and self-reported well-being measures, and then consider predicting well-being on a daily basis and detecting significant changes over a period of time. As this approach seems unreliable for most individuals, I further explore for which individuals such an approach may be reliable and develop a framework for evaluating longitudinal sensing quality. I find that while correlations between smartphone-sensed measures and reported wellbeing scores exist, these relationships are often too weak to reliably predict wellbeing. The second tool I explore seeks to help suicide prevention counselors practice intervening over chat for individuals in crisis. For this, I collect and leverage synthetic conversation transcripts and show how to evaluate a baseline system for counselors to practice crisis de-escalation strategies in a no-risk environment. While text retrieval and generation methods can return responses that make sense in limited context most of the time, i.e., in greater that 50% of examples, generated responses are shorter than retrieving full messages, implying that generation may potentially be a less engaging approach. Overall, I find that significant consideration of context is needed to provide meaningful evaluation of methods for the tools envisioned. While popular algorithms and methods may hold potential to develop the tools discussed, rigorous evaluation and further work is needed to ensure reliability within the application context

Ad hoc efforts for advancing data science education.

With increasing demand for training in data science, extracurricular or "ad hoc" education efforts have emerged to help individuals acquire relevant skills and expertise. Although extracurricular efforts already exist for many computationally intensive disciplines, their support of data science education has significantly helped in coping with the speed of innovation in data science practice and formal curricula. While the proliferation of ad hoc efforts is an indication of their popularity, less has been documented about the needs that they are designed to meet, the limitations that they face, and practical suggestions for holding successful efforts. To holistically understand the role of different ad hoc formats for data science, we surveyed organizers of ad hoc data science education efforts to understand how organizers perceived the events to have gone-including areas of strength and areas requiring growth. We also gathered recommendations from these past events for future organizers. Our results suggest that the perceived benefits of ad hoc efforts go beyond developing technical skills and may provide continued benefit in conjunction with formal curricula, which warrants further investigation. As increasing numbers of researchers from computational fields with a history of complex data become involved with ad hoc efforts to share their skills, the lessons learned that we extract from the surveys will provide concrete suggestions for the practitioner-leaders interested in creating, improving, and sustaining future efforts

Understanding Communities via Hashtag Engagement: A Clustering Based Approach

We develop insight into community use of hashtags on social media and find that hashtags with behavior indicative of real world communities are more engaging. To do this, we study the relationship of hashtag usage with user engagement on Twitter. Hashtag engagement is useful as a surrogate measure of how active community members are. We develop a framework for describing hashtag temporal usage, show the existence of 4 broad classes of hashtags, and show that the engagement of a hashtag varies significantly between classes. Periodically used hashtags, such as for TV shows and weekly community chats, are the most engaging, while hashtags relating to events are the least engaging. Looking at how community dynamics vary within this framework reveals that a hashtag being used more frequently is not positively correlated with it being more engaging. We then explore the periodically used hashtags and find negative correlations with diversity of the user base, which implies concentrated communities are the most engaging. We conclude by studying a set of community conversation-oriented hashtags and find these hashtags to be more engaging than other hashtags, regardless of dynamic type. Our findings support the hypothesis that hashtags with stronger community behavior are more engaging

Meaningless comparisons lead to false optimism in medical machine learning.

A new trend in medicine is the use of algorithms to analyze big datasets, e.g. using everything your phone measures about you for diagnostics or monitoring. However, these algorithms are commonly compared against weak baselines, which may contribute to excessive optimism. To assess how well an algorithm works, scientists typically ask how well its output correlates with medically assigned scores. Here we perform a meta-analysis to quantify how the literature evaluates their algorithms for monitoring mental wellbeing. We find that the bulk of the literature (∼77%) uses meaningless comparisons that ignore patient baseline state. For example, having an algorithm that uses phone data to diagnose mood disorders would be useful. However, it is possible to explain over 80% of the variance of some mood measures in the population by simply guessing that each patient has their own average mood-the patient-specific baseline. Thus, an algorithm that just predicts that our mood is like it usually is can explain the majority of variance, but is, obviously, entirely useless. Comparing to the wrong (population) baseline has a massive effect on the perceived quality of algorithms and produces baseless optimism in the field. To solve this problem we propose "user lift" that reduces these systematic errors in the evaluation of personalized medical monitoring

Automated Text Messaging as an Adjunct to Cognitive Behavioral Therapy for Depression: A Clinical Trial

Background: Cognitive Behavioral Therapy (CBT) for depression is efficacious, but effectiveness is limited when implemented in low-income settings due to engagement difficulties including nonadherence with skill-building homework and early discontinuation of treatment. Automated messaging can be used in clinical settings to increase dosage of depression treatment and encourage sustained engagement with psychotherapy.Objectives: The aim of this study was to test whether a text messaging adjunct (mood monitoring text messages, treatment-related text messages, and a clinician dashboard to display patient data) increases engagement and improves clinical outcomes in a group CBT treatment for depression. Specifically, we aim to assess whether the text messaging adjunct led to an increase in group therapy sessions attended, an increase in duration of therapy attended, and reductions in Patient Health Questionnaire-9 item (PHQ-9) symptoms compared with the control condition of standard group CBT in a sample of low-income Spanish speaking Latino patients.Methods: Patients in an outpatient behavioral health clinic were assigned to standard group CBT for depression (control condition; n=40) or the same treatment with the addition of a text messaging adjunct (n=45). The adjunct consisted of a daily mood monitoring message, a daily message reiterating the theme of that week’s content, and medication and appointment reminders. Mood data and qualitative responses were sent to a Web-based platform (HealthySMS) for review by the therapist and displayed in session as a tool for teaching CBT skills.Results: Intent-to-treat analyses on therapy attendance during 16 sessions of weekly therapy found that patients assigned to the text messaging adjunct stayed in therapy significantly longer (median of 13.5 weeks before dropping out) than patients assigned to the control condition (median of 3 weeks before dropping out; Wilcoxon-Mann-Whitney z=−2.21, P=.03). Patients assigned to the text messaging adjunct also generally attended more sessions (median=6 sessions) during this period than patients assigned to the control condition (median =2.5 sessions), but the effect was not significant (Wilcoxon-Mann-Whitney z=−1.65, P=.10). Both patients assigned to the text messaging adjunct (B=−.29, 95% CI −0.38 to −0.19, z=−5.80, P<.001) and patients assigned to the control conditions (B=−.20, 95% CI −0.32 to −0.07, z=−3.12, P=.002) experienced significant decreases in depressive symptom severity over the course of treatment; however, the conditions did not significantly differ in their degree of symptom reduction.Conclusions: This study provides support for automated text messaging as a tool to sustain engagement in CBT for depression over time. There were no differences in depression outcomes between conditions, but this may be influenced by low follow-up rates of patients who dropped out of treatment

Automated Text Messaging as an Adjunct to Cognitive Behavioral Therapy for Depression: A Clinical Trial

Background: Cognitive Behavioral Therapy (CBT) for depression is efficacious, but effectiveness is limited when implemented in low-income settings due to engagement difficulties including nonadherence with skill-building homework and early discontinuation of treatment. Automated messaging can be used in clinical settings to increase dosage of depression treatment and encourage sustained engagement with psychotherapy.Objectives: The aim of this study was to test whether a text messaging adjunct (mood monitoring text messages, treatment-related text messages, and a clinician dashboard to display patient data) increases engagement and improves clinical outcomes in a group CBT treatment for depression. Specifically, we aim to assess whether the text messaging adjunct led to an increase in group therapy sessions attended, an increase in duration of therapy attended, and reductions in Patient Health Questionnaire-9 item (PHQ-9) symptoms compared with the control condition of standard group CBT in a sample of low-income Spanish speaking Latino patients.Methods: Patients in an outpatient behavioral health clinic were assigned to standard group CBT for depression (control condition; n=40) or the same treatment with the addition of a text messaging adjunct (n=45). The adjunct consisted of a daily mood monitoring message, a daily message reiterating the theme of that week’s content, and medication and appointment reminders. Mood data and qualitative responses were sent to a Web-based platform (HealthySMS) for review by the therapist and displayed in session as a tool for teaching CBT skills.Results: Intent-to-treat analyses on therapy attendance during 16 sessions of weekly therapy found that patients assigned to the text messaging adjunct stayed in therapy significantly longer (median of 13.5 weeks before dropping out) than patients assigned to the control condition (median of 3 weeks before dropping out; Wilcoxon-Mann-Whitney z=−2.21, P=.03). Patients assigned to the text messaging adjunct also generally attended more sessions (median=6 sessions) during this period than patients assigned to the control condition (median =2.5 sessions), but the effect was not significant (Wilcoxon-Mann-Whitney z=−1.65, P=.10). Both patients assigned to the text messaging adjunct (B=−.29, 95% CI −0.38 to −0.19, z=−5.80, P<.001) and patients assigned to the control conditions (B=−.20, 95% CI −0.32 to −0.07, z=−3.12, P=.002) experienced significant decreases in depressive symptom severity over the course of treatment; however, the conditions did not significantly differ in their degree of symptom reduction.Conclusions: This study provides support for automated text messaging as a tool to sustain engagement in CBT for depression over time. There were no differences in depression outcomes between conditions, but this may be influenced by low follow-up rates of patients who dropped out of treatment