Implementation outcome scales for digital mental health (iOSDMH): Scale development and cross-sectional study

Background: Digital mental health interventions are being used more than ever for the prevention and treatment of psychological problems. Optimizing the implementation aspects of digital mental health is essential to deliver the program to populations in need, but there is a lack of validated implementation outcome measures for digital mental health interventions. Objective: The primary aim of this study is to develop implementation outcome scales of digital mental health for different levels of stakeholders involved in the implementation process: Users, providers, and managers or policy makers. The secondary aim is to validate the developed scale for users. Methods: We developed English and Japanese versions of the implementation outcome scales for digital mental health (iOSDMH) based on the literature review and panel discussions with experts in implementation research and web-based psychotherapy. The study developed acceptability, appropriateness, feasibility, satisfaction, and harm as the outcome measures for users, providers, and managers or policy makers. We conducted evidence-based interventions via the internet using UTSMeD, a website for mental health information (N=200). Exploratory factor analysis (EFA) was conducted to assess the structural validity of the iOSDMH for users. Satisfaction, which consisted of a single item, was not included in the EFA. Results: The iOSDMH was developed for users, providers, and managers or policy makers. The iOSDMH contains 19 items for users, 11 items for providers, and 14 items for managers or policy makers. Cronbach α coefficients indicated intermediate internal consistency for acceptability (α=.665) but high consistency for appropriateness (α=.776), feasibility (α=.832), and harm (α=.777) of the iOSDMH for users. EFA revealed 3-factor structures, indicating acceptability and appropriateness as close concepts. Despite the similarity between these 2 concepts, we inferred that acceptability and appropriateness should be used as different factors, following previous studies. Conclusions: We developed iOSDMH for users, providers, and managers. Psychometric assessment of the scales for users demonstrated acceptable reliability and validity. Evaluating the components of digital mental health implementation is a major step forward in implementation science

Usefulness of Implementation Outcome Scales for Digital Mental Health (iOSDMH): Experiences from Six Randomized Controlled Trials

Objectives: Measuring implementation outcomes for digital mental health interventions is essential for examining the effective delivery of these interventions. The “Implementation Outcome Scale of Digital Mental Health” (iOSDMH) has been validated and used in several trials. This study aimed to compare the iOSDMH for participants in six randomized controlled trials (RCTs) involving web-based interventions and to discuss the implications of the iOSDMH for improving the interventions. Additionally, this study examined the associations between iOSDMH scores and program completion rate (adherence). Methods: Variations in total scores and subscales of the iOSDMH were compared in six RCTs of digital mental health interventions conducted in Japan. The web-based intervention programs were based on cognitive behavioral therapy (2 programs), behavioral activation (1 program), acceptance and commitment (1 program), a combination of mindfulness, behavioral activation, and physical activity (1 program), and government guidelines for suicide prevention (1 program). Participants were full-time employees (2 programs), perinatal women (2 programs), working mothers with children (1 program), and students (1 program). The total score and subscale scores were tested using analysis of variance for between-group differences. Results: Total score and subscale scores of the iOSDMH among six trials showed a significant group difference, reflecting users’ perceptions of how each program was implemented, including aspects such as acceptability, appropriateness, feasibility, overall satisfaction, and harm. Subscale scores showed positive associations with completion rate, especially in terms of acceptability and satisfaction (R-squared = 0.93 and 0.89, respectively). Conclusions: The iOSDMH may be a useful tool for evaluating participants’ perceptions of features implemented in web-based interventions, which could contribute to improvements and further development of the intervention