RAMESES publication standards: meta-narrative reviews

PMCID: PMC3558334This is an open access article distributed under the terms of the Creative Commons Attribution License (http://creativecommons.org/licenses/by/2.0), which permits unrestricted use, distribution, and reproduction in any medium, provided the original work is properly cited

Realist Evaluation : an overview

This report summarises the discussions and presentations of the Expert Seminar ‘Realist Evaluation’ with Gill Westhorp, which took place in Wageningen on March 29, 2011. The Expert Seminar was organised by the Wageningen UR Centre for Development Innovation in collaboration with Learning by Design and Context, international cooperation

RAMESES II reporting standards for realist evaluations

Realist evaluation is increasingly used in health services and other fields of research and evaluation. No previous standards exist for reporting realist evaluations. This standard was developed as part of the RAMESES II project. The project's aim is to produce initial reporting standards for realist evaluations.We purposively recruited a maximum variety sample of an international group of experts in realist evaluation to our online Delphi panel. Panel members came from a variety of disciplines, sectors and policy fields. We prepared the briefing materials for our Delphi panel by summarising the most recent literature on realist evaluations to identify how and why rigour had been demonstrated and where gaps in expertise and rigour were evident. We also drew on our collective experience as realist evaluators, in training and supporting realist evaluations, and on the RAMESES email list to help us develop the briefing materials. Through discussion within the project team, we developed a list of issues related to quality that needed to be addressed when carrying out realist evaluations. These were then shared with the panel members and their feedback was sought. Once the panel members had provided their feedback on our briefing materials, we constructed a set of items for potential inclusion in the reporting standards and circulated these online to panel members. Panel members were asked to rank each potential item twice on a 7-point Likert scale, once for relevance and once for validity. They were also encouraged to provide free text comments.We recruited 35 panel members from 27 organisations across six countries from nine different disciplines. Within three rounds our Delphi panel was able to reach consensus on 20 items that should be included in the reporting standards for realist evaluations. The overall response rates for all items for rounds 1, 2 and 3 were 94 %, 76 % and 80 %, respectively.These reporting standards for realist evaluations have been developed by drawing on a range of sources. We hope that these standards will lead to greater consistency and rigour of reporting and make realist evaluation reports more accessible, usable and helpful to different stakeholders

Protocol - realist and meta-narrative evidence synthesis: Evolving Standards (RAMESES)

<p>Abstract</p> <p>Background</p> <p>There is growing interest in theory-driven, qualitative and mixed-method approaches to systematic review as an alternative to (or to extend and supplement) conventional Cochrane-style reviews. These approaches offer the potential to expand the knowledge base in policy-relevant areas - for example by explaining the success, failure or mixed fortunes of complex interventions. However, the quality of such reviews can be difficult to assess. This study aims to produce methodological guidance, publication standards and training resources for those seeking to use the realist and/or meta-narrative approach to systematic review.</p> <p>Methods/design</p> <p>We will: [a] collate and summarise existing literature on the principles of good practice in realist and meta-narrative systematic review; [b] consider the extent to which these principles have been followed by published and in-progress reviews, thereby identifying how rigour may be lost and how existing methods could be improved; [c] using an online Delphi method with an interdisciplinary panel of experts from academia and policy, produce a draft set of methodological steps and publication standards; [d] produce training materials with learning outcomes linked to these steps; [e] pilot these standards and training materials prospectively on real reviews-in-progress, capturing methodological and other challenges as they arise; [f] synthesise expert input, evidence review and real-time problem analysis into more definitive guidance and standards; [g] disseminate outputs to audiences in academia and policy. The outputs of the study will be threefold:</p> <p>1. Quality standards and methodological guidance for realist and meta-narrative reviews for use by researchers, research sponsors, students and supervisors</p> <p>2. A 'RAMESES' (Realist and Meta-review Evidence Synthesis: Evolving Standards) statement (comparable to CONSORT or PRISMA) of publication standards for such reviews, published in an open-access academic journal.</p> <p>3. A training module for researchers, including learning outcomes, outline course materials and assessment criteria.</p> <p>Discussion</p> <p>Realist and meta-narrative review are relatively new approaches to systematic review whose overall place in the secondary research toolkit is not yet fully established. As with all secondary research methods, guidance on quality assurance and uniform reporting is an important step towards improving quality and consistency of studies.</p

Dealing with complexity in a realist synthesis: community accountability and empowerment initiatives

Realist evaluation (Pawson and Tilley, 1997) and realist synthesis (Pawson, 2006) were developed to assist policy makers and program staff to find pathways through complexity - specifically, to develop and test usable theory about complex and varied interventions applied across multiple contexts. One of the central ideas is that understanding how and why something works, or does not, in particular contexts can assist with practical decision-making: whether to use a particular kind of program in a particular situation; how to adapt a program to a particular context; and so on. But if complication and complexity are the norm in social programs (Rogers, 2012, Westhorp, 2012, Westhorp, 2013), how exactly does realist synthesis address this? Can it in fact deal with the complication and complexity of real programs and real decisions? And how might practitioners and policy makers actually use the results? This chapter explores these questions through the lens of a recently completed realist review of community accountability and empowerment initiatives. It begins with an overview of the review in terms of its scope, processes and findings. It then analyses the review in terms of Pawson&#039;s VICTORE framework (discussed earlier in Cchapter 1) to demonstrate the complexity of the review topic, how complexity was reflected in the findings, and how the methodology of realist synthesis helped us to manage and deal with complexity. We then discuss how the findings from a realist synthesis can assist in dealing with the complexities of policy and program management in the real world

A Potent Fuel?: Faith Identity And Development Impact In World Vision Community Programming

This paper explores the role of faith identity on impact in development by looking at the programming of a major international faith‐based development organisation (FBDO). It argues that faith identity rests not only in the internal projected identity of the FBDO but also on perceptions of that identity in the community, highlighting the role context plays in the formulation of faith identity and its impact. Secondly, the paper argues that FBDOs possess not only a faith identity but also a development one and that it is the interplay between both that allows for the creation of engagement and trust in interventions

RAMESES publication standards: realist syntheses

PMCID: PMC3558331This is an open access article distributed under the terms of the Creative Commons Attribution License (http://creativecommons.org/licenses/by/2.0), which permits unrestricted use, distribution, and reproduction in any medium, provided the original work is properly cited