Attree, M.

Clark, E.

Draper, J.

English

Open Research Online (The Open University)

Open Research OnlineThe Open University’s repository of research publicationsand other research outputsEvaluating the impact of healthcare education: Ap-proaches and challengesConference ItemHow to cite:Clark, E.; Draper, J. and Attree, M. (2007). Evaluating the impact of healthcare education: Approaches andchallenges. In: RCN International Nursing Research Conference, 1-4 May 2007, Dundee, Scotland.For guidance on citations see FAQs.c© 2007 The AuthorsVersion: Version of RecordLink(s) to article on publisher’s website:http://www.rcn.org.uk/development/researchanddevelopment/rs/Annual conference archive/2007 - DundeeCopyright and Moral Rights for the articles on this site are retained by the individual authors and/or other copy-right owners. For more information on Open Research Online’s data policy on reuse of materials please consultthe policies page.oro.open.ac.uk 1 Paper 2 Evaluating the impact of healthcare education on practice: an outcomes-driven approach (Jan) 20 mins  The limitations of using a pure research approach From the first paper, we can see that there is a developing body of research evidence about the value of CPE on practice. Moira also made reference to some of the limitations or challenges associated with educational research in this area, at paradigmatic, methodological and method levels.   We know, for example that research has tended to focus on general issues such as learner satisfaction and that patient outcomes and impact on practice are infrequently assessed. Other methodological limitations include a reliance on self-perception reporting, an emphasis in evaluating small scale individual programmes of study, small sample sizes, operational difficulties associated with RCTs, and that there are few longitudinal studies.   So, one of the key issues we would wish to identify is that whilst the research evidence base is important, it does present us with some practical and methodological challenges. In the words of Greenhalgh et al (2003, p. 145) ‘the linear and formulaic link between evidence and practice implicit in evidence based medicine (has proven) inadequate for the complexities of educational research.’  Over the course of the next 2 papers we will return to this issue and engage you in the debates in order to try and establish a way forward for the future.  In addition to identifying these methodological challenges, we suggest that there are also other ‘context’ issues which influence attempts to evaluate the impact that CPE may have on practice. We  2 believe these issues, which are about the real and messy world in which CPE takes place, to be of great importance when considering developing evaluation approaches and cannot be isolated from the research/evaluation process. This is what Moira was referring to regarding the what works, how and under what circumstances. This paper, therefore, now goes on to discuss some of these issues including what we mean by ‘impact on practice’ and the importance of the practice environment in supporting both learning and the application of learning to practice.  What is meant by impact on practice? One of the things that is apparent is the lack of clarity about what we all mean by impact on practice. There is an assumption amongst people working in this area that there is a shared understanding of the term ‘impact on practice’. However, there is evidence in the literature that many authors either do not address the issue at all or use terms such as clinical effectiveness and clinical outcomes interchangeably. So in developing a robust evidence base in this area we need to be clear about what we mean by impact on practice.  Should the ultimate goal of CPE be about improving patient care? And if so, what then do we mean by patient care outcomes? Does this mean direct outcomes such as lower levels of reported pain, shorter length of stay, fewer infections, faster rehabilitation, or fewer complaints etc. In addition to the impact on direct patient care or experience, can ‘impact on practice’ also legitimately refer to impact on the organisation more generally or even on the learner more generally?  If impact on practice can mean all these things then, when working in this area of education evaluation, we have to be very explicit about exactly what we mean by impact on practice.    3 What are the views of different stakeholders? Different stakeholders may therefore have different understandings of what this means to them. For example, students may be looking at impact from the perspective of a direct effect on patient care or patient experience, or on their ability to influence changes in practice, or on their ability to learn, or on their professional development more generally. Managers might be looking at it from a perspective of whether it enables the practitioner to do the job more effectively or more efficiently, or whether it contributes to a different skill mix within the team. Patients might look at it from the perspective of does it make a difference to my experience? Does it make my care any better? Educationalists might look at it from the perspective of the effects on educational practice, whether this type of provision works or whether it is an effective use of resources. Organisations might look at it from the perspective of whether it means the organisation is doing things more efficiently, more coherently and of course, whether it promotes better value for money.  So all of these different stakeholders may have very different views on what they mean by ‘impact on practice’. In summary, impact might mean:  • Impact on patient outcomes • Impact on practice development • Impact on the learning process  • Impact on the practice of educators • Impact on the practice of clinicians and ways of working • Impact on the organisation  So when we are embarking on doing work on impact on practice we need to be mindful of these different views, and be very clear from  4 which perspective we are attempting evaluation. One of our interests (JD and LC in particular) concerns the development of an approach to help key stakeholders evaluate the impact of CPE on practice. We tentatively suggest at this stage, that if such an approach is to be multi-dimensional, then it needs to have the capacity to take account of all of these potential different stakeholder perspectives.  Hakkennes and Green (2006) describe 5 different types of impact on practice outcome measures at the 3 levels of patient, practitioner and organisation:  Patient At the patient level there are 2 categories:  a). actual change for example changes in health status such as levels of pain b). surrogate measures of the above such as patient compliance, length of stay.  Practitioner At the practitioner level there are also 2 categories:  a). actual change in health practice such as compliance with guidelines, prescribing rates etc  b). surrogate measures of the above such as knowledge and attitudes  Organisation At the organisational level there are measures of changes in health systems such as policy, costs, complaints, patient satisfaction and star ratings.   5 This may provide a helpful organising framework to inform the development of an evaluation approach. A different way to cut the cake is offered by Kirkpatrick and Kirkpatrick (2005) which may also offer some insight into how we might develop evaluation strategies which are multidimensional. They suggest 4 levels of learning behaviour of which evaluation approaches need to take account:  Level 1 Reaction. How do learners react to the programme?  Level 2 Learning. To what extent has learning occurred? (understanding concepts/principles/techniques; developing/improving skills; changing attitudes)  Level 3 Behaviour. ‘To what extent has on-the-job behaviour changed as a result of the programme?’  Level 4 Results. To what extent have results occurred because of the training?’  The Logic Model (ref) provides a further perspective particularly with respect to considering impact on practice over time. The model proposes short term, medium term and longer term impacts such as knowledge, skills and attitude acquisition; medium term outcomes such as the impact on clinical practice and team working; and longer-term outcomes such as improving patient outcomes and quality of care and service improvement.  They suggest that enabling students to make the transfer between levels 2 and 3 (and then on to 4) is the greater challenge. Designing evaluation approaches mirrors this increasing complexity.    6 What is the influence of local context and the environment? Another issue which any evaluation approach also needs to consider is the education and practice contexts in which CPE takes place. Sometimes research approaches can fail to take account of the ‘real’ world and its complexity or messiness. An example of this from the generic nursing research world is the RCTs done on the efficacy of hip protectors to reduce hip fractures in older people. The RCT evidence strongly indicates that hip protectors do indeed reduce the risk of fracture but a key factor not given sufficient consideration in the trials is that of poor adherence (compliance) – older people hate wearing them! So there seems little point designing research studies that fail to take account of the real world in which people live.  So the real world issues are really important. With respect to educational evaluation we all know that the practice environment is complex (Ellis and Nolan 2005) and can either enable or disable learning. Ellis and Nolan (2005) for example found that a supportive practice environment was essential to the success of CPE. There is evidence that student support (Hardwick and Jordan 2002, Ellis and Nolan 2005) and what West et al (2006) describe as ‘key allies’ are important to the success of CPE. According to Ellis and Nolan (2005) robust processes for selecting students onto programmes are also important and Ferguson (1994) talks of the importance of ensuring students and managers talk about selection. And Ellis and Nolan (2005) found that motivated and enthusiastic students were more likely to derive the greatest benefit. So, is there something about the characteristics of the individual learner that is important here? And after CPE, whilst Jordan (2000) calls for a follow up of the application of knowledge to practice, the evidence appears to indicate little systematic evaluation by managers of the CPE their  7 staff engage in and its effects (Ellis and Nolan 2005, Sharples et al 2003).  In considering the impact of learning on practice therefore, the interplay between the programme of study itself and the practice milieu is very important (Sloan and Watson 2001). The practice setting, whether it is supportive and fosters learning, is very important. Indifference of managers and colleagues is frequently cited in the literature as a barrier to change (Ellis and Nolan 2005 for example). So it would appear that the extent to which the organisation is a ‘learning organisation’ influences the student’s ability to change practice.  An important contribution to the field is the work by Lorraine Ellis. She identifies the key issues that appear to ‘influence the outcomes of CPE over time’. She identified 4 phases reflecting students’ experiences of CPE overtime and the factors that enhance or limit the impact of CPE: ‘going in’, ‘coming out’, ‘reaping the benefits’ and ‘carrying it on’. Her work emphasises the contribution an enabling practice milieu makes to the success of CPE. In addition, a concept analysis of work-based learning undertaken by colleagues at the RCN Institute (Hardy, Manley and Titchen) also identifies important enabling factors that promote learning in the workplace. These include issues such as a supportive infrastructure and a learning philosophy that promotes a learner centred approach, where educational outcomes are related to the needs of the organisation and the individual, and where there is a genuine learning culture in place that nurtures creativity and reflexivity.  In conclusion, we have articulated in this paper the contextual issues that need to be considered and addressed when attempting to evaluate the impact of CPE on healthcare practice. Consequently  8 we argue that ‘doing’ educational evaluation is far more complex than merely demonstrating attribution. The practice context exerts such a strong influence on students’ ability to learn and apply that learning to practice that its role cannot be ignored when planning impact-on-practice evaluation approaches.   In the next paper, we build on our acknowledgement of the complexities of designing education evaluation approaches – in particular the influence of the practice milieu and the limitations of a pure research approach to demonstrate impact on practice – by illustrating our own experiences of empirical work in this area.   

Evaluating the impact of healthcare education: Approaches and challenges

Abstract

Similar works

Full text

Available Versions

Open Research Online (The Open University)