AG Barnett

C Fluit

C Heidenreich

CR Fluit

DH Dolmans

EA Hesketh

G Bandiera

G Sutkin

JL Paukert

JM Sargeant

K Baker

L Snell

P Chitsabesan

RE Stalmeijer

RM Harden

S Kilminster

SM Kilminster

TB Boerboom

TJ Beckman

Many studies report on the validation of instruments for facilitating feedback to
            clinical supervisors. There is mixed evidence whether evaluations lead to more effective
            teaching and higher ratings. We assessed changes in resident ratings after an evaluation
            and feedback session with their supervisors. Supervisors of three medical specialities
            were evaluated, using a validated instrument (EFFECT). Mean overall scores (MOS) and
            mean scale scores were calculated and compared using paired T-tests. 24 Supervisors from
            three departments were evaluated at two subsequent years. MOS increased from 4.36 to
            4.49. The MOS of two scales showed an increase &gt;0.2: ‘teaching methodology’
            (4.34–4.55), and ‘assessment’ (4.11–4.39). Supervisors with an MOS &lt;4.0 at year 1
            (n = 5) all demonstrated a strong increase in the MOS (mean overall increase 0.50, range
            0.34–0.64). Four supervisors with an MOS between 4.0 and 4.5 (n = 6) demonstrated an
            increase &gt;0.2 in their MOS (mean overall increase 0.21, range −0.15 to 53). One
            supervisor with an MOS &gt;4.5 (n = 13) demonstrated an increase &gt;0.02 in the MOS,
            two demonstrated a decrease &gt;0.2 (mean overall increase −0.06, range −0.42 to 0.42).
            EFFECT-S was associated with a positive change in residents’ ratings of their
            supervisors, predominantly in supervisors with relatively low initial scores.</jats:p

Cornelia R. M. G. Fluit

Remco Feskens

Sanneke Bolhuis

Richard Grol

Michel Wensing

Roland Laan

Crossref

ORI GIN AL ARTICLERepeated evaluations of the quality of clinical teachingby residentsCornelia R. M. G. Fluit • Remco Feskens •Sanneke Bolhuis • Richard Grol • Michel Wensing •Roland LaanPublished online: 8 May 2013 The Author(s) 2013. This article is published with open access at Springerlink.comAbstractMany studies report on the validation of instruments for facilitating feedback toclinical supervisors. There is mixed evidence whether evaluations lead to moreeffective teaching and higher ratings. We assessed changes in resident ratings after anevaluation and feedback session with their supervisors. Supervisors of three medicalspecialities were evaluated, using a validated instrument (EFFECT). Mean overallscores (MOS) and mean scale scores were calculated and compared using pairedT-tests. 24 Supervisors from three departments were evaluated at two subsequentyears. MOS increased from 4.36 to 4.49. The MOS of two scales showed an increase[0.2: ‘teaching methodology’ (4.34–4.55), and ‘assessment’ (4.11–4.39). Super-visors with an MOS\4.0 at year 1 (n = 5) all demonstrated a strong increase in theMOS (mean overall increase 0.50, range 0.34–0.64). Four supervisors with an MOSbetween 4.0 and 4.5 (n = 6) demonstrated an increase [0.2 in their MOS (meanoverall increase 0.21, range -0.15 to 53). One supervisor with an MOS[4.5 (n = 13)demonstrated an increase[0.02 in the MOS, two demonstrated a decrease[0.2 (meanoverall increase -0.06, range -0.42 to 0.42). EFFECT-S was associated with aC. R. M. G. Fluit (&)  S. Bolhuis  R. LaanAcademic Educational Institute, Radboud University Nijmegen Medical Centre,306 IWOO, PO Box 9101, 6500 HB Nijmegen, the Netherlandsemail: c.fluit@iwoo.umcn.nlR. FeskensDepartment of Methods and Statistics, Utrecht University, Utrecht, the NetherlandsR. Grol  M. WensingScientific Institute for Quality of Healthcare, Radboud University Nijmegen Medical Centre,306 IWOO, PO Box 9101, 6500 HB Nijmegen, the NetherlandsR. LaanDepartment of Rheumatology, Radboud University Nijmegen Medical Centre, 306 IWOO,PO Box 9101, 6500 HB Nijmegen, the Netherlands123Perspect Med Educ (2013) 2:87–94DOI 10.1007/s40037-013-0060-5positive change in residents’ ratings of their supervisors, predominantly in supervisorswith relatively low initial scores.Keywords Clinical teacher  Evaluation  Assessment  Instrument Postgraduate medical educationIntroductionPhysicians play a crucial role in teaching residents in clinical practice [1]. They canbe supported in doing so effectively by evaluating and providing feedback on theirclinical teaching performance [2]. The most prevalent evaluation methodology is thecompletion of a standardized teacher-rating form by learners [1–3]. There are manyinstruments for providing feedback to clinical teachers, often based on roles that aredefined in the literature about good clinical teaching [1, 4–12]. Several authors haveexpressed concerns about many of these instruments as they do not cover allimportant aspects of clinical teaching, lack a clear theoretical framework and/or haveinsufficient validity evidence [13–16]. This can make it difficult to establishdirections in which efforts for improving teaching should be headed and toaccomplish real improvement [14, 16].Studies, focusing on the effects of evaluating clinical teachers, demonstrate anincrease in clinical teaching scores after written feedback, whereas feedback in theform of simple numerical rating scores has not improved teaching scores [17].Reading an analysis of clinical teaching or being aware of weaknesses does notautomatically invite teachers to improve their teaching practice [18]. To furtherenhance this reflection and decision-making, discussion of the feedback with afacilitator (e.g., a peer, head or resident) seems useful [18, 19].In 2009, our hospital started to evaluate our clinical teachers in postgraduatemedical education with the EFFECT (Evaluation and Feedback For Effective ClinicalTeaching) questionnaire. Different sources of validity evidence have been collectedand a carefully designed system for using EFFECT in practice was developed [3, 20].As evaluating clinical teachers is time consuming for both clinical teachers andresidents, we wanted to know whether changes appear in clinical teachers’ ratings afteran evaluation and feedback session using EFFECT. We sought answers to thefollowing questions: (1) did resident ratings of clinical teachers improve after ourfeedback strategy, and (2) was the degree of improvement related to the initial rating?MethodsSetting and study populationA prospective pre-post test study was conducted in three medical specialities in amajor university medical centre in the Netherlands in 2009–2011. The study wasdesigned as a pilot to inform the design of a larger evaluation. Residents were invitedby e-mail to evaluate their supervisors by filling in a questionnaire and were free to88 C. R. M. G. Fluit et al.123choose which and how many supervisors they wanted to evaluate. Supervisors filledin a self-evaluation form. Evaluations took place in two subsequent years. Theevaluation results were discussed during a face-to-face meeting between thesupervisor and two residents or the head of department [3].The EFFECT questionnaireThe EFFECT questionnaire is a validated questionnaire based on literature onclinical teaching in the workplace and incorporates the CanMEDS competencies.EFFECT contains 58 items grouped into eleven subscales: (1) role modelling clinicalskills, (2) role modelling science, (3) role modelling CanMEDS competencies, (4)role modelling reflection, (5) assigning work relevant for learning, (6) planning, (7)feedback process, (8) feedback content, (9) teaching methodology, (10) assessment,and (11) personal support. Items can be scored on a five-point Likert scale (1 = verypoor, 2 = poor, 3 = intermediate, 4 = satisfactory, 5 = good). Details of theinstrument, the validation process, and the evaluation system are described elsewhere[3].The system for evaluation and feedback for effective clinical teaching(EFFECT-S)EFFECT-S starts with an introduction meeting with staff and residents at thedepartment to inform (and involve) them about the formative purpose of theevaluation procedure in their department and to make tailor-made appointments. Atailor-made EFFECT-S includes a careful planning; a discipline-specificquestionnaire; agreement on who fills in the questionnaires (residents on avoluntary basis, anonymous ratings), how the feedback procedure is organised,and who has access to the results. The evaluation itself consists of (a) an internet-based self-evaluation questionnaire for supervisors and a questionnaire to becompleted by residents, (b) a feedback report, including the mean scores per item anddomain, a group score (the mean scores of all staff of the particular department) andthe written comments, and (c) a face-to-face meeting (dialogue) between thesupervisor and two residents (representing their group) and guided by a moderator(an experienced educationalist) from outside the department. In one department,these meetings were done by the programme director [20].AnalysesMean overall scores (MOS, mean score of all items of EFFECT) and mean scores onthe separate subscales of EFFECT at year 1 and year 2 were calculated. Weperformed paired T-tests to test differences between the two consecutivemeasurements. To assess relevance of changes we translated the scores into effectsizes. As this was the first follow-up evaluation of EFFECT, we did not performstatistical power analysis.Residents evaluate their clinical teachers 89123EthicsThe ethics committee waived approval. Participation was voluntary for departments.Confidentiality was ensured. Supervisors and residents of the participatingdepartments were informed about the EFFECT questionnaire by the head of thedepartment and/or during the initial meeting. The EFFECT questionnaire was filledin anonymously by residents.ResultsTwenty-four supervisors from three departments were evaluated by three or moreresidents in two subsequent years. A total of 237 questionnaires were obtained.Paired T-tests showed a significant increase in the teaching methodology scale (from4.34 to 4.55) (p \ 0.05). Relevant improvement on EFFECT was defined as at least amoderate effect size of 0.4, which translated into an absolute change of at least 0.2 onthe EFFECT MOS and subscales. Relevant improvements were found on theteaching methodology (from 4.34 to 4.55) and the assessment scale (from 4.11 to4.39).Table 1 shows that the five supervisors with an MOS \4.0 in year 1 alldemonstrated a relevant increase in the MOS after 1 year (mean increase 0.50, range0.34–0.64); the number of subscales within this group demonstrating a relevantincrease varied from five to ten.Four supervisors with an MOS between 4.0 and 4.5 (n = 6) demonstrated arelevant increase in their MOS of[0.2 (mean increase 0.21, range -0.15–0.53). Ofthe 13 supervisors with a high MOS ([4.5), one demonstrated a further increase inthe MOS of[0.2, whereas two supervisors showed a decrease in their MOS of[0.2(mean increase -0.06, range -0.42–0.42).Table 1 Improvement of mean overall score for low and high scoring supervisorsSupervisorswithNumber ofsupervisorsMean overallimprovementMeannumber ofsubscalesimprovedMeannumber ofsubscalesdeclinedNumber ofsupervisorswith increasedscore[0.20Number ofsupervisorswith decreasedscore[0.20Meanoverallscore\4.0n = 5 ?0.50 7.4 – 5 (100 %) –Meanoverallscore4.0–4.5n = 6 ?0.21 5 0.8 3 (50 %) –Meanoverallscore[4.5n = 13 -0.06 1.5 2.8 1 (8 %) 2Only supervisors with at least three evaluations at year 1 and year 2 were included90 C. R. M. G. Fluit et al.123Overall scores of the 24 supervisors are shown in Fig. 1, starting with the lowestscoring supervisor at year 1. The two supervisors with the lowest MOS (3.38 and3.40) at year 1 improved their MOS the most, although their scores at year 2remained below 4.0 (3.80 and 3.77, respectively). The other three supervisors with anMOS \ 4.0 at year 1 succeeded in increasing their MOS to over 4.0.DiscussionEFFECT-S was associated with relevant improvements in residents’ ratings of theirsupervisors on the EFFECT questionnaire. Our results indicate that those who neededto improve most do so after an evaluation and feedback session. This is in line withother research where they found that baseline compliance was the only factor thathelped to explain variation in absolute effectiveness [21]. The greatest increase wasfound in the teaching methodology and assessment scales. The behavioursencompassed by these scales are reported to be highly beneficial for teachingeffectiveness in clinical practice [22]. On the other hand, supervisors who scoredhigh at year 1 showed little further improvement. This may be due to a ceiling effectin the measure or in what can be achieved realistically. In addition, residents mayhave became more critical over time. The result was that the contrast between thebest and worst supervisors was reduced.The strong asset of this study is that effects were measured with a validatedinstrument and did not rely on teachers self-perceptions of their improvements or on33,544,5517 11 18 1 21 20 7 4 24 22 9 5 14 8 3 23 19 2 13 15 12 16 10 6Mean Overal Score (MOS) on EFFECTCase NumberYear 1Year 2Fig. 1 Mean overall scores on EFFECT per supervisor (n = 24) on two subsequent years (supervisorswere evaluated by at least three residents on both measurements)Residents evaluate their clinical teachers 91123non-validated instruments [2, 17]. Our study was not designed to attribute changes tothe feedback, as a control group was lacking. One serious threat of this study is thatthe changes we measured over time could be due to regression to the mean (RTM)[23]. However, supervisors were evaluated by at least three residents at bothmeasurements, and it is very unlikely that the residents at both measurements are thesame. Furthermore, residents were not informed about previous results. Thereforeour results are probably less affected by measurement errors, but this cannot be ruledout.Further research is needed with larger samples to confirm these results and toinvestigate how this feedback aids supervisors in improving their teaching within thedifferent EFFECT scales and what additional measurements could help them.Qualitative research could explore why some supervisors demonstrate a decrease intheir teaching qualities. To further optimize improvements, we need a betterunderstanding of the effective ingredients of EFFECT-S, and the various contextualfactors influencing supervisors’ behaviour.Essentials• The EFFECT questionnaire covers seven domains of clinical teaching: rolemodelling, task allocation, planning, feedback, teaching methodology, assess-ment, and personal support. Behaviours are linked to the CanMEDS roles.• EFFECT-S is a carefully designed system in which residents provide formativefeedback in a dialogue with their clinical supervisor after filling in the EFFECTquestionnaire.• After participating in EFFECT-S, more than one-third of the clinical teachersshow relevant improvements on the EFFECT questionnaire.• Clinical teachers with initially low scores improve most after an evaluation andfeedback session.• Strongest improvements are in the teaching methodology and assessmentdomain.Open Access This article is distributed under the terms of the Creative Commons Attribution Licensewhich permits any use, distribution, and reproduction in any medium, provided the original author(s) andthe source are credited.References1. Snell L, Tallett S, Haist S, et al. A review of the evaluation of clinical teaching: new perspectives andchallenges. Med Educ. 2000;34(10):862–70.2. Fluit CR, Bolhuis S, Grol R, Laan R, Wensing M. Assessing the quality of clinical teachers : asystematic review of content and quality of questionnaires for assessing clinical teachers. J Gen InternMed. 2010;25(12):137–45.3. Fluit C, Bolhuis S, Grol R, et al. Evaluation and feedback for effective clinical teaching inpostgraduate medical education: validation of an assessment instrument incorporating the CanMEDSroles. Med Teach. 2012;34(11):893–901.92 C. R. M. G. Fluit et al.1234. Bandiera G, Lee S, Tiberius R. Creating effective learning in today’s emergency departments: howaccomplished teachers get it done. Ann Emerg Med. 2005;45(3):253–61.5. Chitsabesan P, Corbett S, Walker L, Spencer J, Barton JR. Describing clinical teachers’ characteristicsand behaviours using critical incidents and repertory grids. Med Educ. 2006;40(7):645–53.6. Harden RM, Crosby J. AMEE Guide No 20: the good teacher is more than a lecturer—the twelve rolesof the teacher. Med Teach. 2000;22(4):334–47.7. Heidenreich C, Lye P, Simpson D, Lourich M. The search for effective and efficient ambulatoryteaching methods through the literature. Pediatrics. 2000;105(1 Pt 3):231–7.8. Hesketh EA. A framework for developing excellence as a clinical educator. Med Educ.2001;35(6):555–64.9. Kilminster S, Cottrell D, Grant J, Jolly B. AMEE Guide No. 27: effective educational and clinicalsupervision. Med Teach. 2007;29(1):2–19.10. Kilminster SM, Jolly BC. Effective supervision in clinical practice settings: a literature review. MedEduc. 2000;34(10):827–40.11. Paukert JL, Richards BF. How medical students and residents describe the roles and characteristics oftheir influential clinical teachers. Acad Med. 2000;75(8):843–5.12. Sutkin G, Wagner E, Harris I, Schiffer R. What makes a good clinical teacher in medicine? A reviewof the literature. Acad Med. 2008;83(5):452–66.13. Stalmeijer RE, Dolmans DH, Wolfhagen IH, Muijtjens AM, Scherpbier AJ. The development of aninstrument for evaluating clinical teachers: involving stakeholders to determine content validity. MedTeach. 2008;30(8):e272–7.14. Stalmeijer RE, Dolmans DH, Wolfhagen IH, Muijtjens AM, Scherpbier AJ. The Maastricht ClinicalTeaching Questionnaire (MCTQ) as a valid and reliable instrument for the evaluation of clinicalteachers. Acad Med. 2010;85(11):1732–8.15. Beckman TJ. How reliable are assessments of clinical teaching? A review of the publishedinstruments. J Gen Intern Med. 2004;19(9):971–7.16. Dolmans DH, Wolfhagen HA, Gerver WJ, De Grave W, Scherpbier AJ. Providing physicians withfeedback on how they supervise students during patient contacts. Med Teach. 2004;26(5):409–14.17. Baker K. Clinical teaching improves with resident evaluation and feedback. Anesthesiology.2010;113(3):693–703.18. Boerboom TBB. Evaluating the Veterinary Clinical Teacher. Thesis. Enschede: Ipskamp Drukkers;2011.19. Sargeant JM, Mann KV, van der Vleuten C, Metsemakers JF. Reflection: a link between receiving andusing assessment feedback. Adv Health Sci Educ Theory Pract. 2009;14(3):399–410.20. Fluit CR, Bolhuis S, Klaassen T, et al. Residents provide feedback to their clinical teachers: reflectionthrough dialogue. Med Teach. 2013;1–8, Early Online.21. Jamtvedt G, Young JM, Kristoffersen DT, Thomson O’Brien MA, Oxman AD. Audit and feedback:effects on professional practice and health care outcomes. Cochrane Database Syst Rev. 2003;3:CD000259.22. Boerboom TB, Dolmans DH, Jaarsma AD, Muijtjens AM, Van Beukelen P, Scherpbier AJ. Exploringthe validity and reliability of a questionnaire for evaluating veterinary clinical teachers’ supervisoryskills during clinical rotations. Med Teach. 2011;33(2):e84–91.23. Barnett AG, van der Pols JC, Dobson AJ. Regression to the mean: what it is and how to deal with it. IntJ Epidemiol. 2005;34(1):215–20.Author BiographiesCornelia R. M. G. Fluit is an educationalist and researcher at the Academic Educational Institute,Radboud University Nijmegen Medical Centre.Remco Feskens is a senior research scientist at the Department of Methods and Statistics, UtrechtUniversity.Sanneke Bolhuis is an educationalist, researcher and an emeritus professor at the Teacher College in theNetherlands.Residents evaluate their clinical teachers 93123Richard Grol is a professor of Quality of Care at the Scientific Institute for Quality of Healthcare,Radboud University Nijmegen Medical Centre.Michel Wensing is a professor of Implementation Science at the Scientific Institute for Quality ofHealthcare, Radboud University Nijmegen Medical Centre.Roland Laan is a rheumatologist, professor in Medical Education and Scientific Director of the MedicalSchool, Radboud University Nijmegen Medical Centre.94 C. R. M. G. Fluit et al.123

Perspectives on Medical Education

Repeated evaluations of the quality of clinical teaching by residents

https://link.springer.com/content/pdf/10.1007%2Fs40037-013-0060-5.pdf

Repeated evaluations of the quality of clinical teaching by residents

Abstract

Similar works

Full text

Available Versions

Crossref