Realist Evaluation : an overview

This report summarises the discussions and presentations of the Expert Seminar ‘Realist Evaluation’ with Gill Westhorp, which took place in Wageningen on March 29, 2011. The Expert Seminar was organised by the Wageningen UR Centre for Development Innovation in collaboration with Learning by Design and Context, international cooperation

Exploring the possibility of explicitly combining realist and system thinking: an exemplar

Realist evaluation has gained momentum in the last decade as key mixed methods to understand and engage with socially complex systems. As this interest grows, there is also increasing interest in its possible combination with other complexity oriented approaches. We suggest that the epistemology of Soft System Methodology (SSM), with its focus on constructing models to account for the phenomenon being examined, complements that of realist evaluation, which provides theory driven explanations of what lies beneath complex social programmes. In SSM, models are built and compared to the real world using stakeholders’ descriptions of complex programmes based on structured thinking. Realist evaluation places emphasis on stakeholder engagement to enhance understanding of how complex programmes work. In this presentation, we argue that SSM can support the development of theory driven explanations of complex phenomena. Exemplar: SSM maps were used within a realist evaluation of an Integrated Care Pathway (ICP) for patients with palliative care needs in North East England. In this study, the maps were co-created with service developers and implementers over a series of focus groups in order to visualise and detail the pathway and its varied resources. Chains of causality were hypothesised and tested through qualitative and quantitative data collection. The project provided theory based explanations of how, why and in which circumstances the care pathway was most effective. We conclude that using SSM with realist evaluation can provide; a) a systematic approach to understanding complexity within programmes; b) a framework for stakeholder involvement; c) explicit identification of contextual intricacies. The combination of SSM and realist evaluation presents an innovation in mixed methods research designs, which can serve a key role in advancing understanding of complex social issues. We hope this presentation will stimulate debate and open up the possibilities of explicitly combining realist and system thinking

Realist Evaluation for Practice

The University Archives has determined that this item is of continuing value to OSU's history.Keynote Speaker: Dr. Mansoor A.F. Kazi, Reader and Director of Centre for Evaluation Studies, The University of Huddersfield, Queensgate, Huddersfield, HD13DH, England - "Realist Evaluation for Practice"The Ohio State University College of Social Wor

A realist evaluation of student use of a virtual reality smartphone application in undergraduate legal education

This paper provides a realist evaluation of the Open Justice virtual reality (VR) smartphone app, designed to develop presentation skills for students on an undergraduate legal education module. This work addresses two research questions: what proportion of students engaged with the Open Justice app, and what worked for whom in what circumstances and why? Questionnaire and interviews were conducted and analysed using a realist evaluation approach. This approach is particularly pertinent to the evaluation of how emerging educational technologies are used, as it can point to the potential affordances of a technological intervention, in addition to assessing its current use. In keeping with the realist evaluation approach, data analysis was completed using the context–mechanism–outcome framework. The findings suggest that more than half of students did not engage with the app during their studies, but this was not surprising as it was a pilot project into using VR with our students and such innovations can be difficult to manage in distance education contexts. Those who did engage with it recognised the potential of immersive VR to contribute to legal skills development, but they found the application of the technology, in this context, to be of limited use. By utilising a realist evaluation framework, this study contributes to the emerging field of programme theory‐based evaluations of educational technology, which might form the basis of further research

Let’s Go! 2022-2023 Educational Programs: A Realist Evaluation

Introduction: Let’s Go! is a community engagement initiative working to improve health behaviors that impact obesity. Let’s Go! provides educational programming for all medical learners and attending staff throughout Maine. A realist evaluation is a theory-led approach to evaluation that seeks to understand what works for whom, in what circumstances, and in what respects an intervention is more likely to succeed. This project used the realist evaluation framework to evaluate Let’s Go! educational programming in 2022-2023.https://knowledgeconnection.mainehealth.org/lambrew-retreat-2023/1030/thumbnail.jp

ReseArch with Patient and Public invOlvement: a RealisT evaluation - the RAPPORT study

Background Patient and public involvement (PPI) is a prerequisite for many funding bodies and NHS research ethics approval. PPI in research is defined as research carried out with or by the public rather than to, about or for them. While the benefits of PPI have been widely discussed, there is a lack of evidence on the impact and outcomes of PPI in research. Objectives To determine the types of PPI in funded research, describe key processes, analyse the contextual and temporal dynamics of PPI and explore the experience of PPI in research for all those involved. Mechanisms contributing to the routine incorporation of PPI in the research process were assessed, the impact of PPI on research processes and outcomes evaluated, and barriers and enablers to effective PPI identified. Design A three-staged realist evaluation drawing on Normalisation Process Theory to understand how far PPI was embedded within health-care research in six areas: diabetes mellitus, arthritis, cystic fibrosis, dementia, public health and learning disabilities. The first two stages comprised a scoping exercise and online survey to chief investigators to assess current PPI activity. The third stage consisted of case studies tracked over 18 months through interviews and document analysis. The research was conducted in four regions of England. Participants Non-commercial studies currently running or completed within the previous 2 years eligible for adoption on the UK Clinical Research Network portfolio. A total of 129 case study participants included researchers and PPI representatives from 22 research studies, and representatives from funding bodies and PPI networks

Supporting general hospital staff to provide dementia sensitive care: A realist evaluation

© 2019 The Authors. Published by Elsevier Ltd. This is an open access article under the CC BY-NC-ND license (http://creativecommons.org/licenses/by-nc-nd/4.0).Background: There are an increasing number of interventions to improve hospital care for patients with dementia. Evidence for their impact on staff actions and patient outcomes is, however, limited and context dependent. Objective: To explain the factors that support hospital staff to provide dementia sensitive care and with what outcomes for patients with dementia. Design: A realist evaluation using a two-site case study approach. Setting: Two hospital trusts in the East of England. Site 1 had a ward for patients with dementia that would address their medical and mental health needs. Site 2 used a team of healthcare assistants, who had support from dementia specialist nurses, to work with patients with dementia across the hospital. Participants: Hospital staff who had a responsibility for inpatients with dementia (healthcare assistants, nurses, medical staff, allied healthcare professionals and support staff) (n = 36), patients with dementia (n = 28), and family carers of patients with dementia (n = 2). Methods: A three stage realist evaluation: 1) building the programme theory of what works and when; 2) testing the programme theory through empirical data (80 h non-participant observation, 42 interviews, 28 patient medical notes, 27 neuropsychiatric inventory, and documentary review); 3) synthesis and verification of findings with key stakeholders. Findings: The programme theory comprised six interconnected context-mechanism-outcome configurations: 1) knowledge and authority to respond to an unmet need; 2) role relevant training and opportunities for reflection; 3) clinical experts and senior staff promoting practices that are patient-focused; 4) engaging with opportunities to spend time with patients; 5) risk management as an opportunity for person-centred care; 6) valuing dementia care as skilled work. Effective interactions reduced patient distress and supported patient orientation. Training and allocation of staff time were of themselves insufficient to ensure dementia care was prioritised and valued as skilled work. Staff concerns about the consequences of adverse incidents and work pressures on the ward, even with support, took precedence and influenced the quality of their interactions with patients with dementia. A key finding linked to staff retention and developing capacity in the workforce to provide expert dementia care was that despite extra training and organisational endorsement, nursing staff did not regard dementia care as skilled nursing work. Conclusions: There is increased awareness and organisational commitment to dementia-friendly healthcare in general hospitals. However, in addition to training and adapting the environment to the patient, further work is needed to make explicit the specialist skills required for effective dementia care.Peer reviewe

Appendix to Harris, Mason and Ryan LAC Isle of Wight realist evaluation

Using an Agent Based Model to estimate the avoided costs of the LAC Coordination schem

A realist evaluation of the management of a well- performing regional hospital in Ghana

<p>Abstract</p> <p>Background</p> <p>Realist evaluation offers an interesting approach to evaluation of interventions in complex settings, but has been little applied in health care. We report on a realist case study of a well performing hospital in Ghana and show how such a realist evaluation design can help to overcome the limited external validity of a traditional case study.</p> <p>Methods</p> <p>We developed a realist evaluation framework for hypothesis formulation, data collection, data analysis and synthesis of the findings. Focusing on the role of human resource management in hospital performance, we formulated our hypothesis around the high commitment management concept. Mixed methods were used in data collection, including individual and group interviews, observations and document reviews.</p> <p>Results</p> <p>We found that the human resource management approach (the actual intervention) included induction of new staff, training and personal development, good communication and information sharing, and decentralised decision-making. We identified 3 additional practices: ensuring optimal physical working conditions, access to top managers and managers' involvement on the work floor. Teamwork, recognition and trust emerged as key elements of the organisational climate. Interviewees reported high levels of organisational commitment. The analysis unearthed perceived organisational support and reciprocity as underlying mechanisms that link the management practices with commitment.</p> <p>Methodologically, we found that realist evaluation can be fruitfully used to develop detailed case studies that analyse how management interventions work and in which conditions. Analysing the links between intervention, mechanism and outcome increases the explaining power, while identification of essential context elements improves the usefulness of the findings for decision-makers in other settings (external validity). We also identified a number of practical difficulties and priorities for further methodological development.</p> <p>Conclusion</p> <p>This case suggests that a well-balanced HRM bundle can stimulate organisational commitment of health workers. Such practices can be implemented even with narrow decision spaces. Realist evaluation provides an appropriate approach to increase the usefulness of case studies to managers and policymakers.</p

The uncritical realism of realist evaluation

This article is a response to Ray Pawson’s critique of critical realism, the philosophy of science elaborated by Roy Bhaskar. I argue with Pawson’s interpretation of critical realism’s positions on both natural and social science and his charges concerning its totalizing ontology, its arrogant epistemology and its naive methodology. The differences between critical realism and realist evaluation are not as significant as Pawson contends. The main differences between the two realisms lie in their approaches to the relationship between social structures and human agency, and between facts and values. I argue that evaluation scientists need to clearly distinguish structure and agency. They should also make their values explicit. The uncritical approach of realist evaluation, combined with its underplaying of the importance of agency, leaves it open to implication in the abuses of bureaucratic instrumentalism.</jats:p