Identification and evaluation of risk of generalizability biases in pilot versus efficacy/effectiveness trials: A systematic review and meta-analysis

Background: Preliminary evaluations of behavioral interventions, referred to as pilot studies, predate the conduct of many large-scale efficacy/effectiveness trial. The ability of a pilot study to inform an efficacy/effectiveness trial relies on careful considerations in the design, delivery, and interpretation of the pilot results to avoid exaggerated early discoveries that may lead to subsequent failed efficacy/effectiveness trials. “Risk of generalizability biases (RGB)” in pilot studies may reduce the probability of replicating results in a larger efficacy/effectiveness trial. We aimed to generate an operational list of potential RGBs and to evaluate their impact in pairs of published pilot studies and larger, more well-powered trial on the topic of childhood obesity. Methods: We conducted a systematic literature review to identify published pilot studies that had a published larger-scale trial of the same or similar intervention. Searches were updated and completed through December 31st, 2018. Eligible studies were behavioral interventions involving youth (≤18 yrs) on a topic related to childhood obesity (e.g., prevention/treatment, weight reduction, physical activity, diet, sleep, screen time/sedentary behavior). Extracted information included study characteristics and all outcomes. A list of 9 RGBs were defined and coded: intervention intensity bias, implementation support bias, delivery agent bias, target audience bias, duration bias, setting bias, measurement bias, directional conclusion bias, and outcome bias. Three reviewers independently coded for the presence of RGBs. Multi-level random effects meta-analyses were performed to investigate the association of the biases to study outcomes. Results: A total of 39 pilot and larger trial pairs were identified. The frequency of the biases varied: delivery agent bias (19/39 pairs), duration bias (15/39), implementation support bias (13/39), outcome bias (6/39), measurement bias (4/39), directional conclusion bias (3/39), target audience bias (3/39), intervention intensity bias (1/39), and setting bias (0/39). In meta-analyses, delivery agent, implementation support, duration, and measurement bias were associated with an attenuation of the effect size of − 0.325 (95CI − 0.556 to − 0.094), − 0.346 (− 0.640 to − 0.052), − 0.342 (− 0.498 to − 0.187), and − 0.360 (− 0.631 to − 0.089), respectively. Conclusions Pre-emptive avoidance of RGBs during the initial testing of an intervention may diminish the voltage drop between pilot and larger efficacy/effectiveness trials and enhance the odds of successful translation

We are all one together : peer educators\u27 views about falls prevention education for community-dwelling older adults - a qualitative study

Background: Falls are common in older people. Despite strong evidence for effective falls prevention strategies, there appears to be limited translation of these strategies from research to clinical practice. Use of peers in delivering falls prevention education messages has been proposed to improve uptake of falls prevention strategies and facilitate translation to practice. Volunteer peer educators often deliver educational presentations on falls prevention to community-dwelling older adults. However, research evaluating the effectiveness of peer-led education approaches in falls prevention has been limited and no known study has evaluated such a program from the perspective of peer educators involved in delivering the message. The purpose of this study was to explore peer educators’ perspective about their role in delivering peer-led falls prevention education for community-dwelling older adults. Methods: A two-stage qualitative inductive constant comparative design was used.In stage one (core component) focus group interviews involving a total of eleven participants were conducted. During stage two (supplementary component) semi-structured interviews with two participants were conducted. Data were analysed thematically by two researchers independently. Key themes were identified and findings were displayed in a conceptual framework. Results: Peer educators were motivated to deliver educational presentations and importantly, to reach an optimal peer connection with their audience. Key themes identified included both personal and organisational factors that impact on educators’ capacity to facilitate their peers’ engagement with the message. Personal factors that facilitated message delivery and engagement included peer-to-peer connection and perceived credibility, while barriers included a reluctance to accept the message that they were at risk of falling by some members in the audience. Organisational factors, including ongoing training for peer educators and formative feedback following presentations, were perceived as essential because they affect successful message delivery. Conclusions: Peer educators have the potential to effectively deliver falls prevention education to older adults and influence acceptance of the message as they possess the peer-to-peer connection that facilitates optimal engagement. There is a need to consider incorporating learnings from this research into a formal large scale evaluation of the effectiveness of the peer education approach in reducing falls in older adults

Public health research outputs from efficacy to dissemination: a bibliometric analysis

<p>Abstract</p> <p>Background</p> <p>More intervention research is needed, particularly 'real world' intervention replication and dissemination studies, to optimize improvements in health. This study assessed the proportion and type of published public health intervention research papers over time in physical activity and falls prevention, both important contributors to preventable morbidity and mortality.</p> <p>Methods</p> <p>A keyword search was conducted, using Medline and PsycINFO to locate publications in 1988-1989, 1998-1999, and 2008-2009 for the two topic areas. In stage 1, a random sample of 1200 publications per time period for both topics were categorized as: non-public health, non-data-based public health, or data-based public health. In stage 2 data-based public health articles were further classified as measurement, descriptive, etiological or intervention research. Finally, intervention papers were categorized as: efficacy, intervention replication or dissemination studies. Inter-rater reliability of paper classification was 88%.</p> <p>Results</p> <p>Descriptive studies were the most common data-based papers across all time periods (1988-89; 1998-1999;2008-2009) for both issues (physical activity: 47%; 54%; 65% and falls 75%; 64%; 63%), increasing significantly over time for physical activity. The proportion of intervention publications did not increase over time for physical activity comprising 23% across all time periods and fluctuated for falls across the time periods (10%; 21%; 17%). The proportion of intervention articles that were replication studies increased over the three time periods for physical activity (0%; 2%; 11%) and for falls (0%; 22%; 35%). Dissemination studies first appeared in the literature in 2008-2009, making up only 3% of physical activity and 7% of falls intervention studies.</p> <p>Conclusions</p> <p>Intervention research studies remain only a modest proportion of all published studies in physical activity and falls prevention; the majority of the intervention studies, are efficacy studies although there is growing evidence of a move towards replication and dissemination studies, which may have greater potential for improving population health.</p

Policy and practice impacts of applied research: a case study analysis of the New South Wales Health Promotion Demonstration Research Grants Scheme 2000-2006

BackgroundIntervention research provides important information regarding feasible and effective interventions for health policy makers, but few empirical studies have explored the mechanisms by which these studies influence policy and practice. This study provides an exploratory case series analysis of the policy, practice and other related impacts of the 15 research projects funded through the New South Wales Health Promotion Demonstration Research Grants Scheme during the period 2000 to 2006, and explored the factors mediating impacts.MethodsData collection included semi-structured interviews with the chief investigators (n = 17) and end-users (n = 29) of each of the 15 projects to explore if, how and under what circumstances the findings had been used, as well as bibliometric analysis and verification using documentary evidence. Data analysis involved thematic coding of interview data and triangulation with other data sources to produce case summaries of impacts for each project. Case summaries were then individually assessed against four impact criteria and discussed at a verification panel meeting where final group assessments of the impact of research projects were made and key influences of research impact identified.ResultsFunded projects had variable impacts on policy and practice. Project findings were used for agenda setting (raising awareness of issues), identifying areas and target groups for interventions, informing new policies, and supporting and justifying existing policies and programs across sectors. Reported factors influencing the use of findings were: i) nature of the intervention; ii) leadership and champions; iii) research quality; iv) effective partnerships; v) dissemination strategies used; and, vi) contextual factors.ConclusionsThe case series analysis provides new insights into how and under what circumstances intervention research is used to influence real world policy and practice. The findings highlight that intervention research projects can achieve the greatest policy and practice impacts if they address proximal needs of the policy context by engaging end-users from the inception of projects and utilizing existing policy networks and structures, and using a range of strategies to disseminate findings that go beond traditional peer review publications.<br /

Behaviour change interventions to reduce second-hand smoke (SHS) exposure at home in pregnant women - A systematic review and intervention appraisal

Abstract Background Second-hand smoke (SHS) exposure during pregnancy is associated with poor pregnancy and foetal outcomes. Theory-based behaviour change interventions (BCI) have been used successfully to change smoking related behaviours and offer the potential to reduce exposure of SHS in pregnant women. Systematic reviews conducted so far do not evaluate the generalisability and scalability of interventions. The objectives of this review were to (1) report the BCIs for reduction in home exposure to SHS for pregnant women; and (2) critically appraise intervention-reporting, generalisability, feasibility and scalability of the BCIs employed. Methods Standard methods following PRISMA guidelines were employed. Eight databases were searched from 2000 to 2015 in English. The studies included used BCIs on pregnant women to reduce their home SHS exposure by targeting husbands/partners. The Workgroup for Intervention Development and Evaluation Research (WIDER) guidelines were used to assess intervention reporting. Generalisability, feasibility and scalability were assessed against criteria described by Bonell and Milat. Results Of 3479 papers identified, six studies met the inclusion criteria. These studies found that BCIs led to increased knowledge about SHS harms, reduction or husbands quitting smoking, and increased susceptibility and change in level of actions to reduce SHS at home. Two studies reported objective exposure measures, and one reported objective health outcomes. The studies partially followed WIDER guidelines for reporting, and none met all generalisability, feasibility and scalability criteria. Conclusions There is a dearth of literature in this area and the quality of studies reviewed was moderate to low. The BCIs appear effective in reducing SHS, however, weak study methodology (self-reported exposure, lack of objective outcome assessment, short follow-up, absence of control group) preclude firm conclusion. Some components of the WIDER checklist were followed for BCI reporting, scalability and feasibility of the studies were not described. More rigorous studies using biochemical and clinical measures for exposures and health outcomes in varied study settings are required. Studies should report interventions in detail using WIDER checklist and assess them for generalisability, feasibility and scalability. Trial registration CRD40125026666