Evaluating Digital Health Interventions: Key Questions and Approaches

Digital health interventions have enormous potential as scalable tools to improve health and healthcare delivery by improving effectiveness, efficiency, accessibility, safety, and personalization. Achieving these improvements requires a cumulative knowledge base to inform development and deployment of digital health interventions. However, evaluations of digital health interventions present special challenges. This paper aims to examine these challenges and outline an evaluation strategy in terms of the research questions needed to appraise such interventions. As they are at the intersection of biomedical, behavioral, computing, and engineering research, methods drawn from all of these disciplines are required. Relevant research questions include defining the problem and the likely benefit of the digital health intervention, which in turn requires establishing the likely reach and uptake of the intervention, the causal model describing how the intervention will achieve its intended benefit, key components, and how they interact with one another, and estimating overall benefit in terms of effectiveness, cost effectiveness, and harms. Although RCTs are important for evaluation of effectiveness and cost effectiveness, they are best undertaken only when: (1) the intervention and its delivery package are stable; (2) these can be implemented with high fidelity; and (3) there is a reasonable likelihood that the overall benefits will be clinically meaningful (improved outcomes or equivalent outcomes at lower cost). Broadening the portfolio of research questions and evaluation methods will help with developing the necessary knowledge base to inform decisions on policy, practice, and research

Advancing Models and Theories for Digital Behavior Change Interventions

To be suitable for informing digital behavior change interventions, theories and models of behavior change need to capture individual variation and changes over time. The aim of this paper is to provide recommendations for development of models and theories that are informed by, and can inform, digital behavior change interventions based on discussions by international experts, including behavioral, computer, and health scientists and engineers. The proposed framework stipulates the use of a state-space representation to define when, where, for whom, and in what state for that person, an intervention will produce a targeted effect. The "state" is that of the individual based on multiple variables that define the "space" when a mechanism of action may produce the effect. A state-space representation can be used to help guide theorizing and identify crossdisciplinary methodologic strategies for improving measurement, experimental design, and analysis that can feasibly match the complexity of real-world behavior change via digital behavior change interventions

Behavior change interventions: the potential of ontologies for advancing science and practice

A central goal of behavioral medicine is the creation of evidence-based interventions for promoting behavior change. Scientific knowledge about behavior change could be more effectively accumulated using "ontologies." In information science, an ontology is a systematic method for articulating a "controlled vocabulary" of agreed-upon terms and their inter-relationships. It involves three core elements: (1) a controlled vocabulary specifying and defining existing classes; (2) specification of the inter-relationships between classes; and (3) codification in a computer-readable format to enable knowledge generation, organization, reuse, integration, and analysis. This paper introduces ontologies, provides a review of current efforts to create ontologies related to behavior change interventions and suggests future work. This paper was written by behavioral medicine and information science experts and was developed in partnership between the Society of Behavioral Medicine's Technology Special Interest Group (SIG) and the Theories and Techniques of Behavior Change Interventions SIG. In recent years significant progress has been made in the foundational work needed to develop ontologies of behavior change. Ontologies of behavior change could facilitate a transformation of behavioral science from a field in which data from different experiments are siloed into one in which data across experiments could be compared and/or integrated. This could facilitate new approaches to hypothesis generation and knowledge discovery in behavioral science

Reliability and validity of three questionnaires measuring context-specific sedentary behaviour and associated correlates in adolescents, adults and older adults

BACKGROUND: Reliable and valid measures of total sedentary time, context-specific sedentary behaviour (SB) and its potential correlates are useful for the development of future interventions. The purpose was to examine test-retest reliability and criterion validity of three newly developed questionnaires on total sedentary time, context-specific SB and its potential correlates in adolescents, adults and older adults. METHODS: Reliability and validity was tested in six different samples of Flemish (Belgium) residents. For the reliability study, 20 adolescents, 22 adults and 20 older adults filled out the age-specific SB questionnaire twice. Test-retest reliability was analysed using Kappa coefficients, Intraclass Correlation Coefficients and/or percentage agreement, separately for the three age groups. For the validity study, data were retrieved from 62 adolescents, 33 adults and 33 older adults, with activPAL as criterion measure. Spearman correlations and Bland-Altman plots (or non-parametric approach) were used to analyse criterion validity, separately for the three age groups and for weekday, weekend day and average day. RESULTS: The test-retest reliability for self-reported total sedentary time indicated following values: ICC = 0.37-0.67 in adolescents; ICC = 0.73-0.77 in adults; ICC = 0.68-0.80 in older adults. Item-specific reliability results (e.g. context-specific SB and its potential correlates) showed good-to-excellent reliability in 67.94%, 68.90% and 66.38% of the items in adolescents, adults and older adults respectively. All items belonging to sedentary-related equipment and simultaneous SB showed good reliability. The sections of the questionnaire with lowest reliability were: context-specific SB (adolescents), potential correlates of computer use (adults) and potential correlates of motorized transport (older adults). Spearman correlations between self-reported total sedentary time and the activPAL were different for each age group: rho = 0.02-0.42 (adolescents), rho = 0.06-0.52 (adults), rho = 0.38-0.50 (older adults). Participants over-reported total sedentary time (except for weekend day in older adults) compared to the activPAL, for weekday, weekend day and average day respectively by +57.05%, +46.29%, +53.34% in adolescents; +40.40%, +19.15%, +32.89% in adults; +10.10%, -6.24%, +4.11% in older adults. CONCLUSIONS: The questionnaires showed acceptable test-retest reliability and criterion validity. However, over-reporting of total SB was noticeable in adolescents and adults. Nevertheless, these questionnaires will be useful in getting context-specific information on SB

Development of a questionnaire to assess sedentary time in older persons -- a comparative study using accelerometry

Background: There is currently no validated questionnaire available to assess total sedentary time in older adults. Most studies only used TV viewing time as an indicator of sedentary time. The first aim of our study was to investigate the self-reported time spent by older persons on a set of sedentary activities, and to compare this with objective sedentary time measured by accelerometry. The second aim was to determine what set of self-reported sedentary activities should be used to validly rank people's total sedentary time. Finally we tested the reliability of our newly developed questionnaire using the best performing set of sedentary activities. Methods. The study sample included 83 men and women aged 65-92 y, a random sample of Longitudinal Aging Study Amsterdam participants, who completed a questionnaire including ten sedentary activities and wore an Actigraph GT3X accelerometer for 8 days. Spearman correlation coefficients were calculated to examine the association between self-reported time and objective sedentary time. The test-retest reliability was calculated using the intraclass correlation coefficient (ICC). Results: Mean total self-reported sedentary time was 10.4 (SD 3.5) h/d and was not significantly different from mean total objective sedentary time (10.2 (1.2) h/d, p = 0.63). Total self-reported sedentary time on an average day (sum of ten activities) correlated moderately (Spearman's r = 0.35, p < 0.01) with total objective sedentary time. The correlation improved when using the sum of six activities (r = 0.46, p < 0.01), and was much higher than when using TV watching only (r = 0.22, p = 0.05). The test-retest reliability of the sum of six sedentary activities was 0.71 (95% CI 0.57-0.81). Conclusions: A questionnaire including six sedentary activities was moderately associated with accelerometry-derived sedentary time and can be used to reliably rank sedentary time in older persons. © 2013 Visser and Koster; licensee BioMed Central Ltd

Validity and reliability of subjective methods to assess sedentary behaviour in adults: a systematic review and meta-analysis.

BACKGROUND: Subjective measures of sedentary behaviour (SB) (i.e. questionnaires and diaries/logs) are widely implemented, and can be useful for capturing type and context of SBs. However, little is known about comparative validity and reliability. The aim of this systematic review and meta-analysis was to: 1) identify subjective methods to assess overall, domain- and behaviour-specific SB, and 2) examine the validity and reliability of these methods. METHODS: The databases MEDLINE, EMBASE and SPORTDiscus were searched up to March 2020. Inclusion criteria were: 1) assessment of SB, 2) evaluation of subjective measurement tools, 3) being performed in healthy adults, 4) manuscript written in English, and 5) paper was peer-reviewed. Data of validity and/or reliability measurements was extracted from included studies and a meta-analysis using random effects was performed to assess the pooled correlation coefficients of the validity. RESULTS: The systematic search resulted in 2423 hits. After excluding duplicates and screening on title and abstract, 82 studies were included with 75 self-reported measurement tools. There was wide variability in the measurement properties and quality of the studies. The criterion validity varied between poor-to-excellent (correlation coefficient [R] range - 0.01- 0.90) with logs/diaries (R = 0.63 [95%CI 0.48-0.78]) showing higher criterion validity compared to questionnaires (R = 0.35 [95%CI 0.32-0.39]). Furthermore, correlation coefficients of single- and multiple-item questionnaires were comparable (1-item R = 0.34; 2-to-9-items R = 0.35; ≥10-items R = 0.37). The reliability of SB measures was moderate-to-good, with the quality of these studies being mostly fair-to-good. CONCLUSION: Logs and diaries are recommended to validly and reliably assess self-reported SB. However, due to time and resources constraints, 1-item questionnaires may be preferred to subjectively assess SB in large-scale observations when showing similar validity and reliability compared to longer questionnaires. REGISTRATION NUMBER: CRD42018105994

Understanding the Importance of Context:A Qualitative Study of a Location-Based Exergame to Enhance School Childrens Physical Activity

Many public health interventions are less effective than expected in 'real life settings', yet little work is undertaken to understand the reasons why. The effectiveness of complex public health interventions can often be traced back to a robust programme theory (how and why an intervention brings about a change in outcome(s)) and assumptions that are made about the context in which it is implemented. Understanding whether effectiveness (or lack thereof) is due to the intervention or the context is hugely helpful in decisions about whether to a) modify the intervention; b) modify the context; c) stop providing the intervention. Exergames-also known as Active Video Games or AVGS-are video games which use the player's bodily movements as input and have potential to increase physical activity in children. However, the results of a recent pilot randomised controlled trial (RCT) of a location-based exergame (FitQuest) in a school setting were inconclusive; no significant effect was detected for any of the outcome measures. The aim of this study was to explore whether the programme theory for FitQuest was correct with respect to how and why it would change children's perceptions of physical activity (PA) and exercise self-efficacy in the school setting. A further aim was to investigate the features of the school setting (context) that may impact on FitQuest's implementation and effectiveness. Qualitative data (gathered during the RCT) were gathered from interviews with teachers and children, and observation of sessions using FitQuest. Thematic analysis indicated that whilst children enjoyed playing the game, engaged with goal setting within the game context and undertook low to vigorous physical activity, there were significant contextual factors that prevented it from being played as often as intended. These included environmental factors (e.g. size of the playground), school factors (cancellations due to other activities), school technology policy (rules relating to mobile phone usage) and teacher factors (engagement with the intervention). A revised logic model for the FitQuest intervention indicates how both the design of exergame technology (intervention) and features of the school environment (context) could be improved to increase chances of effectiveness in the future