Transparent, Open, and Reproducible Prevention Science

The field of prevention science aims to understand societal problems, identify effective interventions, and translate scientific evidence into policy and practice. There is growing interest among prevention scientists in the potential for transparency, openness, and reproducibility to facilitate this mission by providing opportunities to align scientific practice with scientific ideals, accelerate scientific discovery, and broaden access to scientific knowledge. The overarching goal of this manuscript is to serve as a primer introducing and providing an overview of open science for prevention researchers. In this paper, we discuss factors motivating interest in transparency and reproducibility, research practices associated with open science, and stakeholders engaged in and impacted by open science reform efforts. In addition, we discuss how and why different types of prevention research could incorporate open science practices, as well as ways that prevention science tools and methods could be leveraged to advance the wider open science movement. To promote further discussion, we conclude with potential reservations and challenges for the field of prevention science to address as it transitions to greater transparency, openness, and reproducibility. Throughout, we identify activities that aim to strengthen the reliability and efficiency of prevention science, facilitate access to its products and outputs, and promote collaborative and inclusive participation in research activities. By embracing principles of transparency, openness, and reproducibility, prevention science can better achieve its mission to advance evidence-based solutions to promote individual and collective well-being

Clearinghouse Standards of Evidence on the Transparency, Openness, and Reproducibility of Intervention Evaluations

Clearinghouses are influential repositories of information on the effectiveness of social interventions. To identify which interventions are “evidence-based,” clearinghouses review intervention evaluations using published standards of evidence that focus primarily on internal validity and causal inferences. Open science practices can improve trust in evidence from evaluations on the effectiveness of social interventions. Including open science practices in clearinghouse standards of evidence is one of many efforts that could increase confidence in designations of interventions as “evidence-based.” In this study, we examined the policies, procedures, and practices of 10 federal evidence clearinghouses that review preventive interventions—an important and influential subset of all evidence clearinghouses. We found that seven consider at least one open science practice when evaluating interventions: replication (6 of 10 clearinghouses), public availability of results (6), investigator conflicts of interest (3), design and analysis transparency (3), study registration (2), and protocol sharing (1). We did not identify any policies, procedures, or practices related to analysis plan registration, data sharing, code sharing, material sharing, and citation standards. We provide a framework with specific recommendations to help federal and other evidence clearinghouses implement the Transparency and Openness Promotion (TOP) Guidelines. Our proposed “TOP Guidelines for Clearinghouses” includes reporting whether evaluations used open science practices, incorporating open science practices in their standards for receiving “evidence-based” designations, and verifying that evaluations used open science practices. Doing so could increase the trustworthiness of evidence used for policy making and support improvements throughout the evidence ecosystem

Expert-generated standard practice elements for evidence-based home visiting programs using a Delphi process

Background: States, territories, non-profits, and tribes are eligible to obtain federal funding to implement federally endorsed evidence-based home visiting programs. This represents a massive success in translational science, with $400 million a year allocated to these implementation efforts. This legislation also requires that 3% of this annual funding be allocated to tribal entities implementing home visiting in their communities. However, implementing stakeholders face challenges with selecting which program is best for their desired outcomes and context. Moreover, recent reviews have indicated that when implemented in practice and delivered at scale, many evidence-based home visiting programs fail to replicate the retention rates and effects achieved during clinical trials. To inform program implementers and better identify the active ingredients in home visiting programs that drive significant impacts, we aimed to develop an expert derived consensus taxonomy on the elements used in home visiting practice that are essential to priority outcome domains. Methods: We convened a panel of 16 experts representing researchers, model representatives, and program implementers using a Delphi approach. We first elicited standard practice elements (SPEs) using open-ended inquiry, then compared these elements to behavior change techniques (BCTs) given their general importance in the field of home visiting; and finally rated their importance to 10 outcome domains. Results: Our process identified 48 SPEs derived from the panel, with 83 additional BCTs added based on the literature. Six SPEs, mostly related to home visitor characteristics and skills, were rated essential across all outcome domains. Fifty-three of the 83 BCTs were rated unnecessary across all outcome domains. Conclusions: This work represents the first step in a consensus-grounded taxonomy of techniques and strategies necessary for home visiting programs and provides a framework for future hypothesis testing and replication studies.</p