Assessing the applicability of public health intervention evaluations from one setting to another: a methodological study of the usability and usefulness of assessment tools and frameworks

BACKGROUND: Public health interventions can be complicated, complex and context dependent, making the assessment of applicability challenging. Nevertheless, for them to be of use beyond the original study setting, they need to be generalisable to other settings and, crucially, research users need to be able to identify to which contexts it may be applicable. There are many tools with set criteria for assessing generalisability/applicability, yet few seem to be widely used and there is no consensus on which should be used, or when. This methodological study aimed to test these tools to assess how easy they were to use and how useful they appeared to be. METHODS: We identified tools from an existing review and an update of its search. References were screened on pre-specified criteria. Included tools were tested by using them to assess the applicability of a Swedish weight management intervention to the English context. Researcher assessments and reflections on the usability and utility of the tools were gathered using a standard pro-forma. RESULTS: Eleven tools were included. Their length, content, style and time required to complete varied. No tool was considered ideal for assessing applicability. Their limitations included unrealistic criteria (requiring unavailable information), a focus on implementation to the neglect of transferability (i.e. little focus on potential effectiveness in the new setting), overly broad criteria (associated with low reliability), and a lack of an explicit focus on how interventions worked (i.e. their mechanisms of action). CONCLUSION: Tools presenting criteria ready to be used may not be the best method for applicability assessments. They are likely to be either too long or incomplete, too focused on differences and fail to address elements that matter for the specific topic of interest. It is time to progress from developing lists of set criteria that are not widely used in the literature, to creating a new approach to applicability assessment. Focusing on mechanisms of action, rather than solely on characteristics, could be a useful approach, and one that remains underutilised in current tools. New approaches to assessing generalisability that evolve away from checklist style assessments need to be developed, tested, reported and discussed

The value of demonstration projects for new interventions: The case of human papillomavirus vaccine introduction in low- and middle-income countries.

Demonstration projects or pilots of new public health interventions aim to build learning and capacity to inform country-wide implementation. Authors examined the value of HPV vaccination demonstration projects and initial national programmes in low-income and lower-middle-income countries, including potential drawbacks and how value for national scale-up might be increased. Data from a systematic review and key informant interviews, analyzed thematically, included 55 demonstration projects and 8 national programmes implemented between 2007-2015 (89 years' experience). Initial demonstration projects quickly provided consistent lessons. Value would increase if projects were designed to inform sustainable national scale-up. Well-designed projects can test multiple delivery strategies, implementation for challenging areas and populations, and integration with national systems. Introduction of vaccines or other health interventions, particularly those involving new target groups or delivery strategies, needs flexible funding approaches to address specific questions of scalability and sustainability, including learning lessons through phased national expansion

The emergence and current performance of a health research system: lessons from Guinea Bissau

<p>Abstract</p> <p>Background</p> <p>Little is known about how health research systems (HRS) in low-income countries emerge and evolve over time, and how this process relates to their performance. Understanding how HRSs emerge is important for the development of well functioning National Health Research Systems (NHRS). The aim of this study was to assess how the HRS in Guinea Bissau has emerged and evolved over time and how the present system functions.</p> <p>Methods</p> <p>We used a qualitative case-study methodology to explore the emergence and current performance of the HRS, using the NHRS framework. We reviewed documents and carried out 39 in-depth interviews, ranging from health research to policy and practice stakeholders. Using an iterative approach, we undertook a thematic analysis of the data.</p> <p>Results</p> <p>The research practices in Guinea Bissau led to the emergence of a HRS with both local and international links and strong dependencies on international partners and donors. The post-colonial, volatile and resource-dependent context, changes in donor policies, training of local researchers and nature of the research findings influenced how the HRS evolved. Research priorities have mostly been set by 'expatriate' researchers and focused on understanding and reducing child mortality. Research funding is almost exclusively provided by foreign donors and international agencies. The training of Guinean researchers started in the mid-nineties and has since reinforced the links with the health system, broadened the research agenda and enhanced local use of research. While some studies have made an important contribution to global health, the use of research within Guinea Bissau has been constrained by the weak and donor dependent health system, volatile government, top-down policies of international agencies, and the controversial nature of some of the research findings.</p> <p>Conclusions</p> <p>In Guinea Bissau a de facto 'system' of research has emerged through research practices and co-evolving national and international research and development dynamics. If the aim of research is to contribute to local decision making, it is essential to modulate the emerged system by setting national research priorities, aligning funding, building national research capacity and linking research to decision making processes. Donors and international agencies can contribute to this process by coordinating their efforts and aligning to national priorities.</p

Adapting evidence-informed complex population health interventions for new contexts : a systematic review of guidance

Background Adapting interventions that have worked elsewhere can save resources associated with developing new interventions for each specific context. While a developing body of evidence shows benefits of adapted interventions compared with interventions transported without adaptation, there are also examples of interventions which have been extensively adapted, yet have not worked in the new context. Decisions on when, to what extent, and how to adapt interventions therefore are not straightforward, particularly when conceptualising intervention effects as contingent upon contextual interactions in complex systems. No guidance currently addresses these questions comprehensively. To inform development of an overarching guidance on adaptation of complex population health interventions, this systematic review synthesises the content of the existing guidance papers. Methods We searched for papers published between January 2000 and October 2018 in 7 bibliographic databases. We used citation tracking and contacted authors and experts to locate further papers. We double screened all the identified records. We extracted data into the following categories: descriptive information, key concepts and definitions, rationale for adaptation, aspects of adaptation, process of adaptation, evaluating and reporting adapted interventions. Data extraction was conducted independently by two reviewers, and retrieved data were synthesised thematically within pre-specified and emergent categories. Results We retrieved 6694 unique records. Thirty-eight papers were included in the review representing 35 sources of guidance. Most papers were developed in the USA in the context of implementing evidence-informed interventions among different population groups within the country, such as minority populations. We found much agreement on how the papers defined key concepts, aims, and procedures of adaptation, including involvement of key stakeholders, but also identified gaps in scope, conceptualisation, and operationalisation in several categories. Conclusions Our review found limitations that should be addressed in future guidance on adaptation. Specifically, future guidance needs to be reflective of adaptations in the context of transferring interventions across countries, including macro- (e.g. national-) level interventions, better theorise the role of intervention mechanisms and contextual interactions in the replicability of effects and accordingly conceptualise key concepts, such as fidelity to intervention functions, and finally, suggest evidence-informed strategies for adaptation re-evaluation and reporting

The applicability and transferability of public health research from one setting to another: a survey of maternal health researchers.

BACKGROUND: Little is known about the process of assessing whether research conducted in one setting is applicable (i.e. implementable) and transferable (i.e. as effective) to another, despite its importance for health policy and practice. Applicability/transferability differs from external validity; the former focuses on potential utility in another specific setting, whilst the latter is more general. This study explored perceptions of applicability/transferability among maternal health researchers. METHODS: Published maternal public health researchers in low- or middle-income countries were invited to complete an online questionnaire. They were shown four summaries of maternal public health intervention evaluations and asked which they felt were the most and least applicable/transferable to their own setting and why. RESULTS: 283 valid questionnaires were received (41% response rate). Applicability/transferability decisions frequently depended on the pertinence of the problem addressed by the intervention or the intervention's characteristics. Less common were comparison of the respondents' setting with the study setting, or consideration of the study's effectiveness. CONCLUSIONS: The factors affecting perceptions of applicability/transferability are broader than those associated with external validity. Improving the reporting of intervention characteristics and implementation is particularly important for applicability/transferability assessments and could increase the appropriate use of public health research in policy and practice

National decision-making on adopting new vaccines: a systematic review.

In recent years numerous new vaccines have been developed, offering potential reductions in the morbidity and mortality caused by a range of diseases. This has led to increased interest in decision-making about the adoption of new vaccines into national immunization programmes. This paper aims to systematically review the literature on national decision-making around the adoption of new vaccines. A thematic framework was developed inductively through analysis of the vaccine adoption decision-making frameworks included in the review. This thematic framework was then applied to the remaining studies included in the review. In total, 85 articles were included in the review: 39 articles describing examples of vaccine adoption decision-making, 26 presenting vaccine decision-making frameworks, 21 empirical articles of decision-making relating to vaccine adoption and 19 theoretical essays. An analysis of vaccine adoption decision-making frameworks identified nine broad categories of criteria: the importance of the health problem; vaccine characteristics; immunization programme considerations; acceptability; accessibility, equity and ethics; financial/economic issues; impact; alternative interventions and the decision-making process. The quality of the empirical studies was varied. Although some of the issues included in the frameworks were similar to those considered in the studies, there were also some notable differences. On the whole, the frameworks were more comprehensive than the studies, including a greater range of criteria. The existing literature provides a good foundation for further research into vaccine adoption decision-making. The current review, in pulling together what is already known and by identifying strengths, weaknesses and gaps in the existing evidence base, aims to encourage a more focused and rigorous approach to the topic in future. This could help to identify the most appropriate ways to develop vaccine adoption decision-making, so as to improve decisions and, ultimately, health outcomes

Human papillomavirus (HPV) vaccine coverage achievements in low and middle-income countries 2007–2016

10.1016/j.pvr.2017.09.001Papillomavirus Research472-7