RAMESES II reporting standards for realist evaluations

Realist evaluation is increasingly used in health services and other fields of research and evaluation. No previous standards exist for reporting realist evaluations. This standard was developed as part of the RAMESES II project. The project's aim is to produce initial reporting standards for realist evaluations.We purposively recruited a maximum variety sample of an international group of experts in realist evaluation to our online Delphi panel. Panel members came from a variety of disciplines, sectors and policy fields. We prepared the briefing materials for our Delphi panel by summarising the most recent literature on realist evaluations to identify how and why rigour had been demonstrated and where gaps in expertise and rigour were evident. We also drew on our collective experience as realist evaluators, in training and supporting realist evaluations, and on the RAMESES email list to help us develop the briefing materials. Through discussion within the project team, we developed a list of issues related to quality that needed to be addressed when carrying out realist evaluations. These were then shared with the panel members and their feedback was sought. Once the panel members had provided their feedback on our briefing materials, we constructed a set of items for potential inclusion in the reporting standards and circulated these online to panel members. Panel members were asked to rank each potential item twice on a 7-point Likert scale, once for relevance and once for validity. They were also encouraged to provide free text comments.We recruited 35 panel members from 27 organisations across six countries from nine different disciplines. Within three rounds our Delphi panel was able to reach consensus on 20 items that should be included in the reporting standards for realist evaluations. The overall response rates for all items for rounds 1, 2 and 3 were 94 %, 76 % and 80 %, respectively.These reporting standards for realist evaluations have been developed by drawing on a range of sources. We hope that these standards will lead to greater consistency and rigour of reporting and make realist evaluation reports more accessible, usable and helpful to different stakeholders

RAMESES publication standards: realist syntheses

PMCID: PMC3558331This is an open access article distributed under the terms of the Creative Commons Attribution License (http://creativecommons.org/licenses/by/2.0), which permits unrestricted use, distribution, and reproduction in any medium, provided the original work is properly cited

Effect of stigma reduction intervention strategies on HIV test uptake in low- and middle-income countries: a realist review protocol

Background Several stigma reduction intervention strategies have been developed and tested for effectiveness in terms of increasing human immunodeficiency virus (HIV) test uptake. These strategies have been more effective in some contexts and less effective in others. Individual factors, such as lack of knowledge and fear of disclosure, and social-contextual factors, such as poverty and illiteracy, might influence the effect of stigma reduction intervention strategies on HIV test uptake in low- and middle-income countries. So far, it is not clearly known how the stigma reduction intervention strategies interact with these contextual factors to increase HIV test uptake. Therefore, we will conduct a review that will synthesize existing studies on stigma reduction intervention strategies to increase HIV test uptake to better understand the mechanisms underlying this process in low- and middle-income countries. Methods A realist review will be conducted to unpack context-mechanism-outcome configurations of the effect of stigma reduction intervention strategies on HIV test uptake. Based on a scoping review, we developed a preliminary theoretical framework outlining a potential mechanism of how the intervention strategies influence HIV test uptake. Our realist synthesis will be used to refine the preliminary theoretical framework to better reflect mechanisms that are supported by existing evidence. Journal articles and grey literature will be searched following a purposeful sampling strategy. Data will be extracted and tested against the preliminary theoretical framework. Data synthesis and analysis will be performed in five steps: organizing extracted data into evidence tables, theming, formulating chains of inference from the identified themes, linking the chains of inference and developing generative mechanisms, and refining the framework. Discussion This will be the first realist review that offers both a quantitative and a qualitative exploration of the available evidence to develop and propose a theoretical framework that explains why and how HIV stigma reduction intervention strategies influence HIV test uptake in low- and middle-income countries. Our theoretical framework is meant to provide guidance to program managers on identifying the most effective stigma reduction intervention strategies to increase HIV test uptake. We also include advice on how to effectively implement these strategies to reduce the rate of HIV transmission.status: publishe

Enterprise Education Competitions: A Theoretically Flawed Intervention?

The demand for including enterprise in the education system, at all levels and for all pupils is now a global phenomenon. Within this context, the use of competitions and competitive learning activities is presented as a popular and effective vehicle for learning. The purpose of this chapter is to illustrate how a realist method of enquiry – which utilises theory as the unit of analysis – can shed new light on the assumed and unintended outcomes of enterprise education competitions. The case developed here is that there are inherent flaws in assuming that competitions will ‘work’ in the ways set out in policy and guidance. Some of the most prevalent stated outcomes – that competitions will motivate and reward young people, that they will enable the development of entrepreneurial skills, and that learners will be inspired by their peers – are challenged by theory from psychology and education. The issue at stake is that the expansion of enterprise education policy into primary and secondary education increases the likelihood that more learners will be sheep dipped in competitions, and competitive activities, without a clear recognition of the potential unintended effects. In this chapter, we employ a realist-informed approach to critically evaluate the theoretical basis that underpins the use of competitions and competitive learning activities in school-based enterprise education. We believe that our findings and subsequent recommendations will provide those who promote and practice the use of competitions with a richer, more sophisticated picture of the potential flaws within such activities.Peer reviewedFinal Published versio

Framework, principles and recommendations for utilising participatory methodologies in the co-creation and evaluation of public health interventions

Background: Due to the chronic disease burden on society, there is a need for preventive public health interventions to stimulate society towards a healthier lifestyle. To deal with the complex variability between individual lifestyles and settings, collaborating with end-users to develop interventions tailored to their unique circumstances has been suggested as a potential way to improve effectiveness and adherence. Co-creation of public health interventions using participatory methodologies has shown promise but lacks a framework to make this process systematic. The aim of this paper was to identify and set key principles and recommendations for systematically applying participatory methodologies to co-create and evaluate public health interventions. Methods: These principles and recommendations were derived using an iterative reflection process, combining key learning from published literature in addition to critical reflection on three case studies conducted by research groups in three European institutions, all of whom have expertise in co-creating public health interventions using different participatory methodologies. Results: Key principles and recommendations for using participatory methodologies in public health intervention co-creation are presented for the stages of: Planning (framing the aim of the study and identifying the appropriate sampling strategy); Conducting (defining the procedure, in addition to manifesting ownership); Evaluating (the process and the effectiveness) and Reporting (providing guidelines to report the findings). Three scaling models are proposed to demonstrate how to scale locally developed interventions to a population level. Conclusions: These recommendations aim to facilitate public health intervention co-creation and evaluation utilising participatory methodologies by ensuring the process is systematic and reproducible

How stakeholder participation can contribute to systematic reviews of complex interventions

Although patient and public involvement in research is a requirement for research funding in many countries, the knowledge base for how to effectively involve people—and evidence of the effectiveness of involvement—is weak. This article describes how methods used in participatory health research were used to involve patients, clients, providers and community health workers across all stages of a realist review. Sustained involvement enabled better identification of the components of the complex intervention of community-based peer support. It also challenged assumptions of how peer support is constructed, leading the review team to question whether the process of designing and implementing interventions has more influence on effectiveness than previously recognised in empirical studies. We conclude with a discussion on when sustained involvement should be used, and the challenges of incorporating it into the traditional researcher-led approach to systematic reviews

Towards a methodology for cluster searching to provide conceptual and contextual "richness" for systematic reviews of complex interventions: case study (CLUSTER)

Background Systematic review methodologies can be harnessed to help researchers to understand and explain how complex interventions may work. Typically, when reviewing complex interventions, a review team will seek to understand the theories that underpin an intervention and the specific context for that intervention. A single published report from a research project does not typically contain this required level of detail. A review team may find it more useful to examine a “study cluster”; a group of related papers that explore and explain various features of a single project and thus supply necessary detail relating to theory and/or context. We sought to conduct a preliminary investigation, from a single case study review, of techniques required to identify a cluster of related research reports, to document the yield from such methods, and to outline a systematic methodology for cluster searching. Methods In a systematic review of community engagement we identified a relevant project – the Gay Men’s Task Force. From a single “key pearl citation” we conducted a series of related searches to find contextually or theoretically proximate documents. We followed up Citations, traced Lead authors, identified Unpublished materials, searched Google Scholar, tracked Theories, undertook ancestry searching for Early examples and followed up Related projects (embodied in the CLUSTER mnemonic). Results Our structured, formalised procedure for cluster searching identified useful reports that are not typically identified from topic-based searches on bibliographic databases. Items previously rejected by an initial sift were subsequently found to inform our understanding of underpinning theory (for example Diffusion of Innovations Theory), context or both. Relevant material included book chapters, a Web-based process evaluation, and peer reviewed reports of projects sharing a common ancestry. We used these reports to understand the context for the intervention and to explore explanations for its relative lack of success. Additional data helped us to challenge simplistic assumptions on the homogeneity of the target population. Conclusions A single case study suggests the potential utility of cluster searching, particularly for reviews that depend on an understanding of context, e.g. realist synthesis. The methodology is transparent, explicit and reproducible. There is no reason to believe that cluster searching is not generalizable to other review topics. Further research should examine the contribution of the methodology beyond improved yield, to the final synthesis and interpretation, possibly by utilizing qualitative sensitivity analysis