Causes and Consequences of the Risk of Generalizability Biases in Health Behavioral Interventions

Preliminary testing of health behavior interventions (e.g., pilot, feasibility studies) are used to evaluate intervention viability prior to additional testing and resource investment (e.g., larger trials). This initial testing provides valuable information, but promising estimates of effectiveness produced during early testing are rarely reproduced in larger studies, stalling the development of effective, scalable health interventions. In the obesity intervention literature, external validity biases, features of the intervention that are not (or cannot) transfer to the larger study, are associated with these diminished effects. These study features, such who delivers the intervention, the characteristics of the population receiving the intervention or the duration and intensity of the intervention are nearly ubiquitous features of behavioral interventions leading us to believe that they exist, and negatively impact interventions beyond the field of obesity research. Establishing the construct validity of external validity bias across multiple disciplines of health behavior could inform health behavior intervention design, delivery, and evaluation. Equipped with the necessary guidance to identify external validity in early-stage studies, researchers and funders could proactively make informed decisions regarding the design and evaluation of preliminary studies, choosing to avoid these pitfalls and stabilize intervention effects across studies. However, identification of the biases alone is unlikely to reduce their inclusion in intervention development. Just as intervention features are ubiquitous across fields, so too is the research enterprise in which they are developed. Early-stage studies are considered the primary source of data to support larger-scale grant applications and must present especially compelling results to be considered competitive. Delivery of an intervention by a highly skilled principal investigator is likely to increase the effectiveness of an intervention but is also a response to the natural constraints of preliminary-study budgets. Identifying these system-level drivers of bias inclusion is necessary to mitigate their use. The purpose of this dissertation is to identify the prevalence and impact of external validity biases in multiple fields of health behavior research and explore the contextual factors surrounding the development of interventions that may lead to the inclusion of these biases. By providing a foundational understanding of the prevalence of external validity biases, as well as the complex contextual factors that may drive their inclusion in preliminary studies, these findings can inform efforts to amend the research enterprise in favor of scalable interventions development that can positively impact population health

Identifying Effective Intervention Strategies to Reduce Children’s Screen Time: A Systematic Review and Meta-Analysis

Background Excessive screen time (≥ 2 h per day) is associated with childhood overweight and obesity, physical inactivity, increased sedentary time, unfavorable dietary behaviors, and disrupted sleep. Previous reviews suggest intervening on screen time is associated with reductions in screen time and improvements in other obesogenic behaviors. However, it is unclear what study characteristics and behavior change techniques are potential mechanisms underlying the effectiveness of behavioral interventions. The purpose of this meta-analysis was to identify the behavior change techniques and study characteristics associated with effectiveness in behavioral interventions to reduce children’s (0–18 years) screen time. Methods A literature search of four databases (Ebscohost, Web of Science, EMBASE, and PubMed) was executed between January and February 2020 and updated during July 2021. Behavioral interventions targeting reductions in children’s (0–18 years) screen time were included. Information on study characteristics (e.g., sample size, duration) and behavior change techniques (e.g., information, goal-setting) were extracted. Data on randomization, allocation concealment, and blinding was extracted and used to assess risk of bias. Meta-regressions were used to explore whether intervention effectiveness was associated with the presence of behavior change techniques and study characteristics. Results The search identified 15,529 articles, of which 10,714 were screened for relevancy and 680 were retained for full-text screening. Of these, 204 studies provided quantitative data in the meta-analysis. The overall summary of random effects showed a small, beneficial impact of screen time interventions compared to controls (SDM = 0.116, 95CI 0.08 to 0.15). Inclusion of the Goals, Feedback, and Planning behavioral techniques were associated with a positive impact on intervention effectiveness (SDM = 0.145, 95CI 0.11 to 0.18). Interventions with smaller sample sizes (n \u3c 95) delivered over short durations (\u3c 52 weeks) were associated with larger effects compared to studies with larger sample sizes delivered over longer durations. In the presence of the Goals, Feedback, and Planning behavioral techniques, intervention effectiveness diminished as sample size increased. Conclusions Both intervention content and context are important to consider when designing interventions to reduce children’s screen time. As interventions are scaled, determining the active ingredients to optimize interventions along the translational continuum will be crucial to maximize reductions in children’s screen time

Estimating Physical Activity and Sleep using the Combination of Movement and Heart Rate: A Systematic Review and Meta-Analysis

International Journal of Exercise Science 16(7): 1514-1539, 2023. The purpose of this meta-analysis was to quantify the difference in physical activity and sleep estimates assessed via 1) movement, 2) heart rate (HR), or 3) the combination of movement and HR (MOVE+HR) compared to criterion indicators of the outcomes. Searches in four electronic databases were executed September 21-24 of 2021. Weighted mean was calculated from standardized group-level estimates of mean percent error (MPE) and mean absolute percent error (MAPE) of the proxy signal compared to the criterion measurement method for physical activity, HR, or sleep. Standardized mean difference (SMD) effect sizes between the proxy and criterion estimates were calculated for each study across all outcomes, and meta-regression analyses were conducted. Two-One-Sided-Tests method were conducted to meta-analytically evaluate the equivalence of the proxy and criterion. Thirty-nine studies (physical activity k = 29 and sleep k = 10) were identified for data extraction. Sample size weighted means for MPE were -38.0%, 7.8%, -1.4%, and -0.6% for physical activity movement only, HR only, MOVE+HR, and sleep MOVE+HR, respectively. Sample size weighted means for MAPE were 41.4%, 32.6%, 13.3%, and 10.8% for physical activity movement only, HR only, MOVE+HR, and sleep MOVE+HR, respectively. Few estimates were statistically equivalent at a SMD of 0.8. Estimates of physical activity from MOVE+HR were not statistically significantly different from estimates based on movement or HR only. For sleep, included studies based their estimates solely on the combination of MOVE+HR, so it was impossible to determine if the combination produced significantly different estimates than either method alone

Impact of risk of generalizability biases in adult obesity interventions: A meta‐epidemiological review and meta‐analysis

Funder: Sue and Bob O'DonnellSummary: Biases introduced in early‐stage studies can lead to inflated early discoveries. The risk of generalizability biases (RGBs) identifies key features of feasibility studies that, when present, lead to reduced impact in a larger trial. This meta‐study examined the influence of RGBs in adult obesity interventions. Behavioral interventions with a published feasibility study and a larger scale trial of the same intervention (e.g., pairs) were identified. Each pair was coded for the presence of RGBs. Quantitative outcomes were extracted. Multilevel meta‐regression models were used to examine the impact of RGBs on the difference in the effect size (ES, standardized mean difference) from pilot to larger scale trial. A total of 114 pairs, representing 230 studies, were identified. Overall, 75% of the pairs had at least one RGB present. The four most prevalent RGBs were duration (33%), delivery agent (30%), implementation support (23%), and target audience (22%) bias. The largest reductions in the ES were observed in pairs where an RGB was present in the pilot and removed in the larger scale trial (average reduction ES −0.41, range −1.06 to 0.01), compared with pairs without an RGB (average reduction ES −0.15, range −0.18 to −0.14). Eliminating RGBs during early‐stage testing may result in improved evidence

Feasibility indicators in obesity-related behavioral intervention preliminary studies: A historical scoping review

The aims of this study were to: 1) conduct a historical scoping review of the reporting of feasibility indicators in behavioral pilot/feasibility studies related to obesity published through 2020, and 2) describe trends in the amount and type of feasibility indicators reported in studies published across three time periods: 1982-2006, 2011-2013, and 2018-2020

Additional file 1 of Fund behavioral science like the frameworks we endorse: the case for increased funding of preliminary studies by the National Institutes of Health

Additional file 1. Methods for qualitative quotes

The mysterious case of the disappearing pilot study: a review of publication bias in preliminary behavioral interventions presented at health behavior conferences

Abstract Background The number of preliminary studies conducted and published has increased in recent years. However, there are likely many preliminary studies that go unpublished because preliminary studies are typically small and may not be perceived as methodologically rigorous. The extent of publication bias within preliminary studies is unknown but can prove useful to determine whether preliminary studies appearing in peer-reviewed journals are fundamentally different than those that are unpublished. The purpose of this study was to identify characteristics associated with publication in a sample of abstracts of preliminary studies of behavioral interventions presented at conferences. Methods Abstract supplements from two primary outlets for behavioral intervention research (Society of Behavioral Medicine and International Society of Behavioral Nutrition and Physical Activity) were searched to identify all abstracts reporting findings of behavioral interventions from preliminary studies. Study characteristics were extracted from the abstracts including year presented, sample size, design, and statistical significance. To determine if abstracts had a matching peer-reviewed publication, a search of authors’ curriculum vitae and research databases was conducted. Iterative logistic regression models were used to estimate odds of abstract publication. Authors with unpublished preliminary studies were surveyed to identify reasons for nonpublication. Results Across conferences, a total of 18,961 abstracts were presented. Of these, 791 were preliminary behavioral interventions, of which 49% (388) were published in a peer-reviewed journal. For models with main effects only, preliminary studies with sample sizes greater than n = 24 were more likely to be published (range of odds ratios, 1.82 to 2.01). For models including interactions among study characteristics, no significant associations were found. Authors of unpublished preliminary studies indicated small sample sizes and being underpowered to detect effects as barriers to attempting publication. Conclusions Half of preliminary studies presented at conferences go unpublished, but published preliminary studies appearing in peer-reviewed literature are not systematically different from those that remain unpublished. Without publication, it is difficult to assess the quality of information regarding the early-stage development of interventions. This inaccessibility inhibits our ability to learn from the progression of preliminary studies

INEQUITIES IN CHILDREN’S INVOLVEMENT IN STRUCTURED PROGRAMMING DURING SCHOOL AND SUMMER: AN OBSERVATIONAL COHORT TIME USE STUDY

BACKGROUND: In the US, children are at a greater risk of excessive weight gain during summer vacation compared to the school year. The Structured Days Hypothesis theorizes this is because in summer, children spend less time in structured environments that promote healthy behaviors. However, little is known about how children spend their time during the summer vs school and how this may differ by income and age. METHODS: In an observational cohort of 890 children (50% girls, 43% ≤200% Poverty, grades K-6th), parents completed a time use record (TUR) on their smartphones every evening for 14d in April/May (school) and 14d in July (summer) in 2021 and 2022. Parents reported the timing of setting-specific contexts of their child’s daily activities (e.g., afterschool program, sports, summer programs). Zero-inflated hurdle models estimated the probability of involvement in structure (0/1) and percent of wake time children spent in structured settings (non-zeros) between school and summer. Poverty status (living in poverty [LP] vs not living in poverty [NLP]) and grade were examined as moderators. RESULTS: A total of 17,029 TUR were completed. Compared to children NLP, children LP were more likely to never engage in structured activities outside of school (19% vs 38%, OR 0.19, 95CI 0.11-0.34) or during summer (23% vs 32%, 0.51, 0.28-0.92). For children NLP, involvement in structured activities during school peaked at 3rd grade and declined through 6th; while summer structure was greatest during K and declined through 6th. For children LP, involvement in structure did not vary across grades during school or summer. For children involved in structure, percent of time spent in structure varied by grade and income. During school and summer, children LP spent the greatest amount of time in structure in K and this declined thru 6th. Conversely, children NLP in 3rd and 4th spent the greatest amount of time in structure during school, whereas during summer children in K spent the greatest amount of time in structure. CONCLUSIONS: During school and summer, fewer children LP were involved in structured activities compared to children NLP. However, when they were involved, they spent a greater percentage of their day in structure compared to children NLP. Additional policies are needed to reduce the gap between children LP vs NLP and involvement in structured programs during school and summer. Grant Funding: Research reported in this publication was supported by the National Institute of Diabetes and Digestive and Kidney Diseases of the National Institutes of Health under Award Number R01DK116665. The content is solely the responsibility of the authors and does not necessarily represent the official views of the National Institutes of Health

Use of guidelines, checklists, frameworks, and recommendations in behavioral intervention preliminary studies and associations with reporting comprehensiveness: a scoping bibliometric review

Abstract Background Guidelines, checklists, frameworks, and recommendations (GCFRs) related to preliminary studies serve as essential resources to assist behavioral intervention researchers in reporting findings from preliminary studies, but their impact on preliminary study reporting comprehensiveness is unknown. The purpose of this study was to conduct a scoping bibliometric review of recently published preliminary behavioral-focused intervention studies to (1) examine the prevalence of GCFR usage and (2) determine the associations between GCFR usage and reporting feasibility-related characteristics. Methods A systematic search was conducted for preliminary studies of behavioral-focused interventions published between 2018 and 2020. Studies were limited to the top 25 journals publishing behavioral-focused interventions, text mined to identify usage of GCFRs, and categorized as either not citing GCFRs or citing ≥ 2 GCFRs (Citers). A random sample of non-Citers was text mined to identify studies which cited other preliminary studies that cited GCFRs (Indirect Citers) and those that did not (Never Citers). The presence/absence of feasibility-related characteristics was compared between Citers, Indirect Citers, and Never Citers via univariate logistic regression. Results Studies (n = 4143) were identified, and 1316 were text mined to identify GCFR usage (n = 167 Citers). A random sample of 200 studies not citing a GCFR were selected and categorized into Indirect Citers (n = 71) and Never Citers (n = 129). Compared to Never Citers, Citers had higher odds of reporting retention, acceptability, adverse events, compliance, cost, data collection feasibility, and treatment fidelity (OR range = 2.62–14.15, p < 0.005). Citers also had higher odds of mentioning feasibility in purpose statements, providing progression criteria, framing feasibility as the primary outcome, and mentioning feasibility in conclusions (OR range = 6.31–17.04, p < 0.005) and lower odds of mentioning efficacy in purpose statements, testing for efficacy, mentioning efficacy in conclusions, and suggesting future testing (ORrange = 0.13–0.54, p < 0.05). Indirect Citers had higher odds of reporting acceptability and treatment fidelity (OR range = 2.12–2.39, p < 0.05) but lower odds of testing for efficacy (OR = 0.36, p < 0.05) compared to Never Citers. Conclusion The citation of GCFRs is associated with greater reporting of feasibility-related characteristics in preliminary studies of behavioral-focused interventions. Researchers are encouraged to use and cite literature that provides guidance on design, implementation, analysis, and reporting to improve the comprehensiveness of reporting for preliminary studies

Measuring Microtemporal Processes Underlying Preschoolers’ Screen Use and Behavioral Health: Protocol for the Tots and Tech Study

BackgroundExcessive screen time is associated with poor health and behavioral outcomes in children. However, research on screen time use has been hindered by methodological limitations, including retrospective reports of usual screen time and lack of momentary etiologic processes occurring within each day. ObjectiveThis study is designed to assess the feasibility and utility of a comprehensive multibehavior protocol to measure the digital media use and screen time context among a racially and economically diverse sample of preschoolers and their families. This paper describes the recruitment, data collection, and analytical protocols for the Tots and Tech study. MethodsThe Tots and Tech study is a longitudinal, observational study of 100 dyads: caregivers and their preschool-age children (aged 3-5 years). Both caregivers and children will wear an Axivity AX3 accelerometer (Axivity Ltd) for 30 days to assess their physical activity, sedentary behavior, and sleep. Caregivers will complete ecological momentary assessments (EMAs) for 1 week to measure child behavioral problems, caregiver stress, and child screen time. ResultsThe Tots and Tech study was funded in March 2020. This study maintains rolling recruitment, with each dyad on their own assessment schedule, depending on the time of enrollment. Enrollment was scheduled to take place between September 2020 and May 2022. We aim to enroll 100 caregiver-child dyads. The Tots and Tech outcome paper is expected to be published in 2022. ConclusionsThe Tots and Tech study attempts to overcome previous methodological limitations by using objective measures of screen time, physical activity, sedentary behavior, and sleep behaviors with contextual factors measured by EMA. The results will be used to evaluate the feasibility and utility of a comprehensive multibehavior protocol using objective measures of mobile screen time and accelerometry in conjunction with EMA among caregiver-child dyads. Future observational and intervention studies will be able to use this study protocol to better measure screen time and its context. International Registered Report Identifier (IRRID)DERR1-10.2196/3624