RAMESES publication standards: meta-narrative reviews

PMCID: PMC3558334This is an open access article distributed under the terms of the Creative Commons Attribution License (http://creativecommons.org/licenses/by/2.0), which permits unrestricted use, distribution, and reproduction in any medium, provided the original work is properly cited

Realist Evaluation : an overview

This report summarises the discussions and presentations of the Expert Seminar ‘Realist Evaluation’ with Gill Westhorp, which took place in Wageningen on March 29, 2011. The Expert Seminar was organised by the Wageningen UR Centre for Development Innovation in collaboration with Learning by Design and Context, international cooperation

RAMESES II reporting standards for realist evaluations

Realist evaluation is increasingly used in health services and other fields of research and evaluation. No previous standards exist for reporting realist evaluations. This standard was developed as part of the RAMESES II project. The project's aim is to produce initial reporting standards for realist evaluations.We purposively recruited a maximum variety sample of an international group of experts in realist evaluation to our online Delphi panel. Panel members came from a variety of disciplines, sectors and policy fields. We prepared the briefing materials for our Delphi panel by summarising the most recent literature on realist evaluations to identify how and why rigour had been demonstrated and where gaps in expertise and rigour were evident. We also drew on our collective experience as realist evaluators, in training and supporting realist evaluations, and on the RAMESES email list to help us develop the briefing materials. Through discussion within the project team, we developed a list of issues related to quality that needed to be addressed when carrying out realist evaluations. These were then shared with the panel members and their feedback was sought. Once the panel members had provided their feedback on our briefing materials, we constructed a set of items for potential inclusion in the reporting standards and circulated these online to panel members. Panel members were asked to rank each potential item twice on a 7-point Likert scale, once for relevance and once for validity. They were also encouraged to provide free text comments.We recruited 35 panel members from 27 organisations across six countries from nine different disciplines. Within three rounds our Delphi panel was able to reach consensus on 20 items that should be included in the reporting standards for realist evaluations. The overall response rates for all items for rounds 1, 2 and 3 were 94 %, 76 % and 80 %, respectively.These reporting standards for realist evaluations have been developed by drawing on a range of sources. We hope that these standards will lead to greater consistency and rigour of reporting and make realist evaluation reports more accessible, usable and helpful to different stakeholders

Dealing with complexity in a realist synthesis: community accountability and empowerment initiatives

Realist evaluation (Pawson and Tilley, 1997) and realist synthesis (Pawson, 2006) were developed to assist policy makers and program staff to find pathways through complexity - specifically, to develop and test usable theory about complex and varied interventions applied across multiple contexts. One of the central ideas is that understanding how and why something works, or does not, in particular contexts can assist with practical decision-making: whether to use a particular kind of program in a particular situation; how to adapt a program to a particular context; and so on. But if complication and complexity are the norm in social programs (Rogers, 2012, Westhorp, 2012, Westhorp, 2013), how exactly does realist synthesis address this? Can it in fact deal with the complication and complexity of real programs and real decisions? And how might practitioners and policy makers actually use the results? This chapter explores these questions through the lens of a recently completed realist review of community accountability and empowerment initiatives. It begins with an overview of the review in terms of its scope, processes and findings. It then analyses the review in terms of Pawson's VICTORE framework (discussed earlier in Cchapter 1) to demonstrate the complexity of the review topic, how complexity was reflected in the findings, and how the methodology of realist synthesis helped us to manage and deal with complexity. We then discuss how the findings from a realist synthesis can assist in dealing with the complexities of policy and program management in the real world

RAMESES publication standards: realist syntheses

PMCID: PMC3558331This is an open access article distributed under the terms of the Creative Commons Attribution License (http://creativecommons.org/licenses/by/2.0), which permits unrestricted use, distribution, and reproduction in any medium, provided the original work is properly cited

What works and why in the identification and referral of adults with comorbid obesity in primary care: a realist review

Primary care practitioners (PCPs) are well placed to identify individuals with obesity and weight‐related comorbidities and to refer them to weight management services (WMS), but this does not often happen in practice. In this realist review, we searched six databases for intervention studies targeted at PCPs to improve the identification and referral of adults with comorbid obesity. Realist analysis was used to identify context‐mechanism‐outcome (CMO) configurations across 30 included papers (reporting on 27 studies). Most studies used multiple intervention strategies, categorised into: (a) training, (b) tools to improve identification, (c) tools to improve ease of referral, (d) audit/feedback, (e) working in networks/quality circles, and (f) other. The realist synthesis identified 12 mechanisms through which interventions work to improve identification and referral, including increasing knowledge about obesity and awareness of and confidence in WMS among practitioners, improved communication and trust between practitioners and WMS, and higher priority given to weight management among primary care teams. The theory of “candidacy” (a person's eligibility for medical attention and intervention) provided a robust explanatory framework but required refinement: (a) to take account of the different services (primary care and weight management) that patients must navigate to access support; and (b) to acknowledge the importance of wider contextual factors

Using the realist interview approach to maintain theoretical awareness in realist studies.

Realist evaluation submits that theories and models of how, why, for whom and under what circumstances programs work could be formulated by conceptualizing the relational links between the context within which programs are implemented, the generative mechanisms the programs trigger, and the outcomes of interest. Qualitative and quantitative data collection and analysis allow for the description of the relevant context, the generative mechanisms, and the emergent outcomes of programs and provide explanatory power to link these elements. The ‘realist interviewing technique’, whereby interviewees comment on a suggested ‘program theory’ to provide refinement, is proposed as a distinctive approach for conducting interviews in a realist-informed inquiry

Policymakers\u27 experience of a capacity-building intervention designed to increase their use of research: A realist process evaluation

Background: An intervention’s success depends on how participants interact with it in local settings. Process evaluation examines these interactions, indicating why an intervention was or was not effective, and how it (and similar interventions) can be improved for better contextual fit. This is particularly important for innovative trials like Supporting Policy In health with Research: an Intervention Trial (SPIRIT), where causal mechanisms are poorly understood. SPIRIT was testing a multi-component intervention designed to increase the capacity of health policymakers to use research. Methods: Our mixed-methods process evaluation sought to explain variation in observed process effects across the six agencies that participated in SPIRIT. Data collection included observations of intervention workshops (n = 59), purposively sampled interviews (n = 76) and participant feedback forms (n = 553). Using a realist approach, data was coded for context-mechanism-process effect configurations (retroductive analysis) by two authors. Results: Intervention workshops were very well received. There was greater variation of views regarding other aspects of SPIRIT such as data collection, communication and the intervention’s overall value. We identified nine inter-related mechanisms that were crucial for engaging participants in these policy settings: (1) Accepting the premise (agreeing with the study’s assumptions); (2) Self-determination (participative choice); (3) The Value Proposition (seeing potential gain); (4) ‘Getting good stuff’ (identifying useful ideas, resources or connections); (5) Self-efficacy (believing ‘we can do this!’); (6) Respect (feeling that SPIRIT understands and values one’s work); (7) Confidence (believing in the study’s integrity and validity); (8) Persuasive leadership (authentic and compelling advocacy from leaders); and (9) Strategic insider facilitation (local translation and mediation). These findings were used to develop tentative explanatory propositions and to revise the programme theory. Conclusion: This paper describes how SPIRIT functioned in six policy agencies, including why strategies that worked well in one site were less effective in others. Findings indicate a complex interaction between participants’ perception of the intervention, shifting contextual factors, and the form that the intervention took in each site. Our propositions provide transferable lessons about contextualised areas of strength and weakness that may be useful in the development and implementation of similar studies