Attree, M.

Clark, E.

Draper, J.

English

Open Research Online (The Open University)

Open Research OnlineThe Open University’s repository of research publicationsand other research outputsEvaluating the impact of healthcare education: thechallenges and a way forwardConference ItemHow to cite:Clark, E.; Draper, J. and Attree, M. (2007). Evaluating the impact of healthcare education: the chal-lenges and a way forward. In: RCN International Nursing Research Conference, 1-4 May 2007, Dundee,Scotland.For guidance on citations see FAQs.c© 2007 The AuthorsVersion: Version of RecordLink(s) to article on publisher’s website:http://www.rcn.org.uk/development/researchanddevelopment/rs/Annual conference archive/2007 - DundeeCopyright and Moral Rights for the articles on this site are retained by the individual authors and/or other copy-right owners. For more information on Open Research Online’s data policy on reuse of materials please consultthe policies page.oro.open.ac.uk 1 Paper 3 Evaluating the impact of healthcare education: the challenges and a way forward  Introduction (Liz) Having set the scene in the last 2 papers with respect to the complexities of designing rigorous education evaluation processes, in this paper we want to illustrate this with reference to our own ‘real-life’ experiences. First, Moira will discuss her experiences of conducting an impact evaluation project that sought to evaluate the impact of a patient safety education programme on individuals and organisations. Second, Jan and I will discuss our experiences to date on a project which aims to develop an approach to demonstrate impact on practice of CPE across a range of settings. These personal experiences have served to underline the complexities of doing this kind of work. So, finally we will draw together what we consider to be the key challenges facing evaluators in this area and identify a number of nitty gritty questions to structure our discussions for the remaining time available.  Project 1 RCA study (Moira)  Evaluating the Impact of Patient Safety Education RCA training  Study aims: were to  Assess the efficacy and impact of using blended e-learning to educate healthcare staff about Root Cause Analysis (RCA) effectiveness of the blended e-learning educational approach ways organisational & operational practices facilitated access to, and skill in using electronic resources to support root cause analysis extent to which organisational & operational systems enable and promote use of patient safety e-learning   2 Study Methods The study adopted an Impact Evaluation (Rossi et al 2004) approach, using multiple case study (Yin 1994 ) and using simultaneous mixed methods data collection. The intervention was a Root Cause Analysis blended e-learning programme (developed by the NPSA) for health and social care practitioners and managers. A quasi-experimental/ intervention study was used with pre and post-intervention assessment. Quantitative data were collected by Confidence Logs, Questionnaire and e-learning server logs; qualitative data were collected by Focus Group and Individual Interviews.  The study adopted a mixed method, with multiple foci / evaluation levels; moving up from learner’s thoughts & feelings and knowledge, to learning transfer and behaviour change in individual practice and onto organisational  impact in terms of RCA report quality and learning from feedback . Sample:  health and social care practitioners and managers individuals & organisation  Strengths of the study: Comprehensive evaluation matrix; Measurement of lower and higher educational outcomes from multiple perspective and using multi data sources .   Although we thought we had an appropriate design and study plan some limitations remained: these included :-  Sample size small;  no control group/site Tools adapted - not yet fully validated Self-report data   Low response rate for post-course Confidence Logs  Challenge of e-learning monitoring Time between course end and impact evaluation  Despite these essentially pragmatic limitations Impact Evaluation was a useful evaluation framework.      3 Project 2: RCNI (Jan and Liz) So despite designing research approaches, such as the multi-method approaches described by Moira, in an attempt to overcome the challenges with pure research approaches we identified in the first paper, still we encounter difficulties. We were therefore interested in trying to develop and approach to impact-on-practice evaluation that attempts to take account of: the influence of the practice milieu; the challenges associated with designing robust methods; the views of all the different stakeholders; the importance of scaleability The project we are involved in, supported by Higher Education Innovation Fund 3 funding from Hefce, is at the preliminary stages.  The remit of the project is to  ‘work with healthcare employers and other key stakeholders to develop, pilot and disseminate a tool for employers and CPE providers to assess the impact of CPE on practice that is robust yet sufficiently flexible to apply to a range of modes of delivery and professional contexts.’  The approach is one of collaborative engagement with key stakeholders, working with them to establish what are the key challenges and what each stakeholder would like to see in an evaluative tool.  As a prelude to more systematic data gathering using focus groups and interviews, we have had a number of in-depth conversations about the benefits of CPE with 3 groups of stakeholders – employers, patients and post-registration nursing students. Employers introduced the following issues: If it is to be useful any process that is developed must be manageable in a pressurised working environment, so it must be  4 easy to understand, administer and analyse. An approach must not be programme specific but be able to capture the essence of things that actually make a difference to patient care. The work must be dynamic and able to respond to rapidly changing healthcare environments.  Patients had one key message – they wanted reassurance that all health professionals have the necessary knowledge and skills to provide good care. And students recognised the value of helping them to articulate the longer-term benefits of their CPE. They were also keen to be able to demonstrate direct benefits for patient outcomes and thought that their employers would also want this. However, since the start of the project we have seen a shift both in the political landscape and also in our own methodological landscapes. The political imperative to demonstrate the value of CPE is now even more crucial, given the financial difficulties facing the NHS. So, the investment seen in the evidence-based practice movement to demonstrate positive clinical and economic benefits of ‘interventions’ needs to be urgently made with respect to CPE. And how have our own ideas shifted over time? Initially we envisaged being able to develop a ‘tool’ in partnership with stakeholders that could be used by educators, clinicians, organisations, to explicitly demonstrate the direct impacts of CPE on practice. During the preliminary phase of the project, reading, debating at conference and informal discussions with stakeholders, we are now more aware of the complexities of the task and have, perhaps, moved position. What we have articulated in the previous papers, is that the whole issue of evaluating education and its practice impact is complex. We believe that such work needs to take account of these complexities and not focus merely on one of the key issues. They all have to be  5 considered in the round. Take a cake as a metaphor. When making a cake, although it is important to understand the individual contributions the egg, the sugar, the flour and the butter make to the whole, it is how they are all brought together that makes a successful cake. So, the total or whole is greater than the sum of the parts. So, in designing evaluation approaches that demonstrate impact on practice, all the ingredients need to be explored. This has led us to alter slightly our perspective and language. We are no longer thinking about a ‘tool’ as this seems to imply a focussed, mechanistic approach. Rather we are pursuing an evaluation approach that has the capacity to take account of all the ingredients in our ‘impact on practice cake’!  In taking such a stance, it may be that in addition to developing an evaluation approach, we may also make recommendations about the key features of the individual, the education programme, the practice context and the organisation that work together to create maximum benefits of CPE. Summary of the key challenges (All) Across the three papers of this Symposium we have identified the key challenges of evaluating the impact of CPE on healthcare practice. So, in summary, what are these challenges? Outcomes We need first to define what is meant by impact on practice in order to define meaningful impact on practice outcome measures. With reference to Hakkennes and Green (2006), are we talking about patient, learner/practitioner or organisational outcomes? Kirkpatrick’s evaluation hierarchy may also provide a useful framework. Most studies address the level 1 and 2 educational outcomes of learners’ views and knowledge acquisition. There is an  6 urgent need to focus on the higher levels 3 and 4 of knowledge application and impact on performance and to do so over time. And perhaps the Logic Model has a potential contribution to make. Stakeholders For us, evaluation of impact on practice needs to take account of more than just the view of students. Employers, commissioners, educators and patients and carers should also be involved. There is therefore an imperative to represent a range of perspectives. Methodology We have identified a range of challenges at the level of methodology. There are no standardised ‘off-the-shelf’ evaluation designs that provide valid, reliable and sensitive measures. The majority of existing measures relate to small, single programme evaluations and are self-devised, unvalidated tools of unproven reliability. We need to develop valid and reliable tools to describe and evaluate educational processes i.e. what works and how, under what conditions/contexts. And because of the complexity of the messy real world of practice where confounding variables are hard to control and resources scarce, no amount of experimental or quasi-experimental research is ever going to expose a causal relationship between CPE and practice outcomes. So, methodological design has to be appropriate for the aim of the evaluation and fit the context, conditions and circumstances and yet also be scientifically credible. Method Current studies have tended to use blunt data collection methods and have relied on retrospective and self-report data on student perceptions. There needs also to be an emphasis on gaining multiple perspectives rather than just the view of the learner, that  7 capture data over time, not just in the immediate aftermath of CPE. Perhaps there is a role for multiple methods including observation.  Scale In the light of our preliminary work with key stakeholders, commissioners of education and employers of healthcare professionals seem to require an approach that could be applied to potentially large numbers of learners in the workforce. So, any approach needs not only to be rigorous but also, as importantly, it needs to be scaleable.  Summary and key questions In this Symposium we have made the case for the importance of demonstrating the returns on the investments made in CPE. We have described the limitations of research approaches and have argued that the practice context, definitions of impact on practice and different stakeholder perspectives make the design of robust evaluation approaches a challenging one.  In the words of Hutchinson (1999, p. 1267) there is a ‘fundamental difficulty in addressing the questions that everyone wants answered: what works in what context, with which groups and at what cost? Unfortunately, there may not be simple answers to these questions.’  It would seem that the key question that we are left with is   8 Is it feasible to develop an approach to evaluating the impact of CPE on practice that is manageable in a pressurised work environment? If so, what methods offer potential and how broad/narrow does the evaluation focus need to be? If not, what should we do?    

Evaluating the impact of healthcare education: the challenges and a way forward

Abstract

Similar works

Full text

Available Versions

Open Research Online (The Open University)