Russ, Elaine

Smith, Jake

Stacey, Mark

Online Research @ Cardiff

ow to:Evaluation is vital for medical educators, enabling the continual evaluation and adjustment of learning programmes, assuring they continue to meet the expectations and needs of learners.  However, evidence suggests that frequently used evaluation forms are unable to adequately measure learning experiences and there is an argument that a move towards delayed evaluation will provide more helpful results for trainers. This How To guide is based on Will Thalheimer’s (2016) book “Performance Focused smile sheets”, which looks at how to optimise the evaluation data we receive from learners for any learning event.The problem with current evaluation formsIn order to get useful feedback from learners, we should be aiming to increase the validity, reliability and effectiveness of the feedback questions we use, while also minimising bias and focusing the attention of respondents on the content of the learning itself (Pullman, 2015). Most current evaluation forms employ a variation on the Likert scale, this presents options on a scale e.g., between “strongly disagree’ and ‘strongly agree”.  However, unfocused questions mean this scale can often fail to gather meaningful data, producing unclear guidance for action. For example, the statement “I am satisfied with today’s training” (presented with the Likert scale), is often used to assess training events but the learner has no indication as to what aspect of the training they are rating.  The training may have been excellent but the lunch/venue unsatisfactory. This would influence a learner’s satisfaction rating but has no bearing on the effectiveness of the training itself.How can we optimise the evaluation we receive?We collect evaluation data for a variety of reasons: to flag up ineffective training programmes; to gather ideas for ongoing revision of the programme; to assess pilot programmes and to help learners reflect on and reinforce what they have learned.According to Kirkpatrick (1976), effective evaluation forms should therefore enable us to understand: a) whether learners have understood what we taught them.b) whether they remember what we taught them.c) are they motivated to apply what we taught them.d) what after-training support may be required.An evaluation form that establishes all four of the above will help us refine our understanding, improve our learning designs and ensure that we are getting meaningful and actionable results. Trainers and training programmes that respond to feedback from learners that explores the above areas are more likely to improve job performance and increase retention of learning (Phillips & Phillips, 2016).The Importance of Delayed EvaluationWidespread practice is to measure learning directly after learning events.  While this seems sensible and provides the benefit of immediate feedback, doing so does not take into account a multitude of research on human learning. We are only measuring our learner’s understanding and / or satisfaction at that point in time and not the extent to which they retained and are able to recall or apply the learning (Phillips & Phillips, 2016).When we conduct evaluation at the end of a training event the learners are at their highest level of memory retrieval and we are getting biased results that make us, and our learning interventions, look better than actual implementation. Learners may feel they will always have the learned information at the top of their minds but our brain is just not set up to allow us to imagine what our future cognitive states will feel like (Willingham, 2010).Get More from Evaluation Forms Through Delayed Feedback   Jake Smith, Elaine Russ and Dr Mark StaceyMedical Education @ Cardiff, 2018Category: Assessment and evaluationExample Question: In regard to the course topics taught HOW ABLE ARE YOU to put what you’ve learned into practice on the job?A. I’m NOT AT ALL ABLE to put the concepts into practiceB. I have GENERAL AWARENESS of the concepts taught but I will need more training/practice/guidance/experience TO DO ACTUAL JOB   TASKS using the concepts taughtC. I am ABLE TO WORK ON ACTUAL JOB TASKS, but I’LL NEED MORE HANDS-ON EXPERIENCE to be fully competent in using the    concepts taughtD. I am ABLE TO PERFORM ACTUAL JOB TASKS at a FULLY COMPETENT LEVEL using the concepts taughtE. I am ABLE TO PERFORM ACTUAL JOB TASKS at an EXPERT LEVEL using the concepts taughtFor further questions examples please visit: http://doo.vote/f89378dSUMMARYGiven that current evaluation methods are uncorrelated with learning results (Thalheimer, 2016) and the often employed Likert-type scale can fail to provide meaningful data, there is a case for re-evaluating how we approach training evaluation.  w	Effective evaluation forms need to ascertain whether   a learner has understood, remembered and applied   the training. w	Pre-training questions can allow us to tailor our    training evaluation to the right level. SUMMARYw	Evaluation data gathered directly after learning will   only provide a snapshot of learners when they are   at their highest level of memory retrieval and it is   sensible to introduce a degree of delayed evaluation   into our learning programmes. w Delayed feedback (2-4 weeks after learning is optimal   according to Thalheimer) can measure whether    learners have applied the concepts taught, how work   place situations impact on application and can cement   learning (Harry, 2010).Centre for Medical Education (C4ME)School of Medicine, Cardiff University, 9th Floor, Neuadd Meirionydd, Heath Park, Cardiff  CF14 4YS Tel: +44 (0)29 2068 7451        E-mail: medicaleducation@cardiff.ac.ukISBN: 978-1-907019-86-9Series EditorDr Michal Tombs – Senior Lecturer in Medical Education, Academic Section of Medical Education, C4ME, School of Medicine, Cardiff University.Jake Smith, Cardiff Unit for Research and Evaluation in Medical and Dental Education (CUREMeDE)Elaine Russ, Cardiff Unit for Research and Evaluation in Medical and Dental Education (CUREMeDE)Dr Mark Stacey, Consultant Anaesthetist and Associate Dean HEIW, Cardiff and Vale NHS trustWorking Memory in Learning and Teaching Whiteboard Video -  https://www.youtube.com/watch?v=zPNwWK7T39k&t=11s Interested in learning more about this and other educational topics? Why not professionalise your role with an academic qualification at PGCert, Dip or MSc in Medical Education via e-learning or attendance courses.Contact: medicaleducation@cardiff.ac.ukwww.cardiff.ac.uk/medicaleducationwww.cardiff.ac.uk/meded-elearningReferencesPulliam, P., 2015. Real World Training Evaluation. Assn For Talent Development.Kirkpatrick, D. L. (1976). Evaluation of training. In R. L. Craig (Ed.), Training and Development Handbook. New York: McGraw-Hill.Phillips, J.J. and Phillips, P.P., 2016. Handbook of training evaluation and measurement methods. Routledge.Willingham, D. (2010). Why don’t students like school? San Francisco, CA: Jossey-Bass.Thalheimer,W. (2016) Performance-focussed smile sheets: A radical rethinking of a dangerous art form. Work-Learning Press.Harry J.M. (2010). Improving training impact through effective follow up: techniques and their application, Journal of ManagementDevelopment, Vol. 29 Issue: 6, pp.520-534, https:// doi.org/10.1108/02621711011046495Further informationDelayed evaluation refers to the practice of waiting for a period of time after the learning event before seeking feedback.  It can enhance evaluation in numerous ways by ensuring that learning is more representative of on-the-job cognitions, that learners will have a greater insight about the effectiveness of the training to support their remembering and about their ability to apply what they have learned on the job.Thalheimer suggests that the optimal time for delayed feedback is 2-4 weeks post training. This means learners are back on their job and are able to say whether they have attempted to apply the new concepts, whether they have encountered situations relevant to the topics covered, and how successful or not they have been in applying the learning in the workplace. See our example questions for guidance.

How to: get more from evaluation forms through delayed feedback

https://orca.cardiff.ac.uk/id/eprint/118442/1/How%20To%20Get%20More%20from%20Evaluation%20Forms%20Through%20Delayed%20Feedback.pdf

How to: get more from evaluation forms through delayed feedback

Abstract

Similar works

Full text

Available Versions

Online Research @ Cardiff