This current study had two goals: 1. To examine the changes in various performance measuresas residents progress through psychotherapy training2. To explore correlations among various measures of psychotherapycompetence Conclusions:  Resident self-assessment does not appear to reflect competenceas assessed by supervisors, or knowledge as assessed by objectivetests. Global supervisor evaluations do not appear to reliably measurecompetence. Standardized supervisor assessment instruments are needed. Further work is needed to develop reliable, valid measures ofcompetency

Benjamin, Sheldon

Chauhan, Mohit

Matthews-Bellinger, Julia

Zanetti, Mary L.

English

eScholarship@UMMS

University of Massachusetts Medical School eScholarship@UMMS Psychiatry Publications and Presentations Psychiatry 2010-3 Correlation of Measures of Psychotherapy Competency in Psychiatry Residents Julia Matthews-Bellinger University of Massachusetts Medical School Et al. Let us know how access to this document benefits you. Follow this and additional works at: https://escholarship.umassmed.edu/psych_pp  Part of the Psychiatry Commons Repository Citation Matthews-Bellinger J, Benjamin S, Chauhan M, Zanetti ML. (2010). Correlation of Measures of Psychotherapy Competency in Psychiatry Residents. Psychiatry Publications and Presentations. Retrieved from https://escholarship.umassmed.edu/psych_pp/479 This material is brought to you by eScholarship@UMMS. It has been accepted for inclusion in Psychiatry Publications and Presentations by an authorized administrator of eScholarship@UMMS. For more information, please contact Lisa.Palmer@umassmed.edu. Correlation of Measures of Psychotherapy  Competency in Psychiatry ResidentsJulia Matthews, PhD, MD; Sheldon Benjamin, MD; Mohit Chauhan, MDa; Mary Zanetti, EdDUMass Medical School, UMass Memorial Healthcare,  Worcester, MA       aAustin Medical Center, Mayo Health System,  Austin, MNREFERENCES: ACGME Program Requirements for Graduate Medical Education in Psy-1. chiatry (Effective: July 1, 2007)Psychotherapy Competency Assessment Tools: User-Friendly Summa-2. ry of Available Assessment Tools and Annotations Vers. 2-14-05. http://www.aadprt.org/training/assesment_tools/Mohl P, Lomax J, Tasman A, et al, (1990) Psychotherapy training for the 3. psychiatrist of the future. Am J Psychiatry 145:7-13Holloway E, Neufeldt S, (1995) Supervision: its contribution to treatment 4. efficacy. J Consult Clin Psychol 63:207-213Mullen, LS, Rieder RO, Glick, RA, Luber, B, Rosen, PJ, (2004)Testing 5. Psychodynamic Psychotherapy Skills among Psychiatric Residents: the Psychodynamic Psychotherapy Competency Test. Am J Psychiatry 161:1658-1664Larson, LM, Suzuki, LA, Gillespie, KN, Potenza, MT, Bechtel, MA, and 6. Toulouse, AL, (1992) Development and validation of the Counseling Self-Estimate Inventory. J Counseling Psychology 39:105-120Giordano FL, Briones DF. (2003) Assessing Residents’ Competence in 7. Psychotherapy Acad Psychiatry 27:145–147INTRODUCTION: The ACGME Residency Review Commit-tee in Psychiatry has stipulated that general psychiatry train-ees develop “competency in applying supportive, psychody-namic, and cognitive-behavioral psychotherapies to both brief and long-term individual practice [1].”  Residency programs are required to demonstrate and faculty are required to verify that trainees have attained the required competencies.  Yet no generally accepted reliable and valid measure of psycho-therapy competency has yet been developed.  Several measures are widely used to assess resident compe-tence in psychotherapy [2,7].Evaluation by individual psychotherapy supervisors [3,4] • Rating of resident’s conduct of psychotherapy in live, re-• corded, or transcribed performance Training portfolios • Performance on a multiple choice examination such as • The Columbia Psychodynamic Psychotherapy Compe-tence Test [5] Resident self-assessment such as the Counseling Self-• Estimate Inventory (COSE), a self-report instrument de-signed to assess confidence and self-efficacy [6]There have been few studies examining the reliability and va-lidity of available competency assessment measures, and little empirical research examining the success of training programs in developing trainee competence in psychotherapy. The current study had two goals:To examine the changes in various performance mea-1. sures as residents progress through training To explore correlations among various measures of psy-2. chotherapy competenceMETHODS: We conducted a retrospective analysis of all avail-able data for residents in our program from July 2000 through July 2009 (63 residents), including: Psychiatry Resident in Training Examination (PRITE) – Glob-1) al Psychiatry score and Psychosocial Therapies subscore Columbia Psychodynamic Psychotherapy Competency Test 2) (CPPCT) - Scores (given as percentile rank among all ex-aminees)Counseling Self Estimate Inventory (COSE) – Scores on the 3) COSE were collected for all PGY1 or PGY2 residents; how-ever, COSE assessments were not routinely collected for PGY3 and PGY4 residents in the early years of this study. Our analysis used the first 34 of 37 items on the COSE due to inadvertent omission on some forms.Supervisor ratings - Averaged global supervisor ratings of 4) resident performance (5 point scale) in the following 9 ar-eas: formulation skills; psychotherapeutic interventions; tol-erance of uncertainty; transference, countertransference, boundaries; ability to utilize different conceptual models; hu-manistic qualities; practice-based learning (self evaluation, integration of feedback into practice); communication and interpersonal skills; working with difficult patients.TABLE 1: Number of Residents with Available DataPGY-1 PGY-2 PGY-3 PGY-4PRITE 49 47 42 33CPPCT 49 48 40 21COSE 49 48 26 17Supervisor Rating NA 39 35 24Statistical analyses were performed with the Statistical Pack-age for Social Sciences, SPSS.  Analyses comparing perfor-mance in different training years were done using unpaired t tests (two tail) on pooled data for each training year. Corre-lations were done using paired data for individual residents to derive Pearson two tailed correlation coefficients.  Due to missing data, the N for correlations was often less than the total N. **Fig 1: COSE Mean Item ScoreRESULTS: Figures 1-3 show changes in COSE, PRITE, and CPPCT by training year.Fig 2: CPPCT Mean PercentileRankFig 3: PRITE Standard ScoreTable 2: Correlations Among Measures of Competence§§Pearson correlation coefficients, two-tailed, paired analysis*denotes p < 0.01 compared to PGY-1 and PGY-2*denotes p < 0.01 compared to PGY-1 * *** *CPPCT:•  CPPCT showed trend toward improved performance (average percentile rank) each training year (changes did not reach significance).Mean supervisor ratings•  consistently high & did not change significantly between PGY 2 & PGY-4 (either group means or paired analysis).  Correlated MeasuresIndividual resident’s scores on COSE throughout training• Individual resident’s scores on PRITE in PGY-2 & PGY-3• Individual resident’s scores on PRITE & CPPCT in PGY-2• • Total PGY-1 & 2 COSE score (before psychotherapy experi-• ence) with PGY-4 CPPCT scoreNon-Correlated MeasuresCOSE score with overall PRITE psychiatry score or PRITE • psychosocial subscoreCOSE score with CPPCT in same training year• PGY-2 supervisor rating with PGY-2 COSE score• PGY-3 supervisor rating with PRITE psychiatry or psychoso-• cial score or CPPCT scoreNegatively Correlated MeasuresPGY-2 & 3 COSE scores with PGY-3 psychotherapy supervi-• sor rating STUDY LIMITATIONSSmall N (6 residents/yr max) and incomplete data• Supervisor evaluations non-standardized & without measures • of validity or reliabilityPerformance measures collected at different times during train-• ing yearsRetrospective analysis of data not systematically collected for • researchDISCUSSIONResident confidence increases with first experiences conduct-• ing supervised psychodynamic psychotherapy, paralleled by significant improvement in PRITE psychosocial subscores.  The early COSE-late CPPCT score correlation may suggest • that greater psychotherapy interest or experience on entry pre-dicts greater learning.The negative correlation between supervisor rating  and COSE • scores suggests a possible supervisory reaction to overconfi-dence and/or inflation of ratings to boost confidence. Uniformly above average supervisor scores that do not change • year to year may reflect reluctance to give lower ratings, rating by PGY-expectation rather than competence, or lack of depend-ability of supervisor ratings.The relative lack of correlations may mean measures assess • different aspects of performance, lack validity, or merely reflect the small N’s in our study.CONCLUSIONS: Resident self-assessment does not appear to reflect competence as assessed by supervisors, or knowledge as assessed by ob-jective tests.  Global supervisor evaluations do not appear to reliably measure competence.  Standardized supervisor assessment instruments are needed. Further work is needed to develop reliable, valid measures of competency.Changes in Performance Measures during ResidencyPRITE:•  Global Psychiatry scores changed significantly from PGY-2 - PGY-4. Psychosocial Therapy subscore did not change significantly from PGY-1 to PGY-2, but increased sig-nificantly in PGY-3 & PGY-4.COSE•  (resident confidence): COSE score unchanged (both total score & all individual COSE items) between PGY-1 and PGY-2.  COSE increased dramatically by early PGY-3 after one year of classes and conducting supervised psychother-apy with 2 patients (24 of 34 COSE items showed significant change at p < 0.05 level or greater).  COSE increased further in PGY-4 year (COSE N for PGY-4 too small for meaningful analysis).  COSE item score range = 1-7.! "#$%&!'()*!+,-!.!"#$%&!'()*!+,-!/!"#$%&!'()*!+,-!0!1%23!+14"*!+5678!+,-!/!1%23!+14"*!+565#7!+,-!/!1%23!+14"*!+5678!+,-!0!1%23!+14"*!+565#7!+,-!0!'++'"!9%$:!1%23!+,-!.!'++'"!9%$:!1%23!!+,-!/!'++'"!9%$:!1%23!+,-!0!'++'"!9%$:!1%23!+,-!;!)<=:!*>%&:!+,-!/!)<=:!*>%&:!+,-!0!"#$%&!'()*!+,-!.!.!!9!?!;@!:ABCDD!=!?:BBB!9!?!0E!:;;AD!=!?:B;/!9!?!/.!F:.0;!=!?:;CE!9!?!00!:B0.!=!?:AE0!9!?!00!F:B/@!=!?:AA.!9!?!/@!:B0E!=!?:ACC!9!?!/@!:.;.!=!?:;BB!9!?!0A!:B.0!=!?:@;/!9!?!0;!:B0;!=!?:AEA!9!?!/G!:C@;D!=!?:B0/!9!?!.0!:B.;!=!?:@;;!9!?!/@!F:/@.!=!?:.C@!9!?!/C!"#$%&!'()*!+,-!/!!! .!!;A!:;E;D!:B/E!/0!F:B.@!:@BC!;.!F:BBE!:@G/!;.!:..A!:;@/!0E!:/E@!:..0!0E!:...!:C;B!00!:B@@!:C0;!;/!:/BG!:/;.!0;!:E.EDD!:BBA!.G!F:.;@!:0@@!0;!F:0GBD!:B0G!0/!"#$%&!'()*!+,-!0!!! ! .!!/E!F:0.@!:./.!/C!:///!:/AE!/C!F:.C;!:;A0!/0!:/;A!:/C0!/0!F:BAC!:G//!/B!F:.;E!:;GA!/E!:/B/!:0G@!/.!:/.A!:C;E!.B!:BB/!:@@;!/0!F:;ACD!:B0B!/B!1%23!+14"*!+5678H%$I6!+,-!/!! ! ! .!!;G!:;B0DD!:BBC!;G!:E;ADD!:BBB!;.!:;@CDD!:BB.!;.!:;C;D!:B./!0B!:0C@D!:B.0!;G!F:B/G!:AEA!0@!F:.CG!:;@G!/.!:;;ADD!:BB;!0@!:BE0!:G.C!0E!1%23!+14"*!+5678#5#7H%&!+,-!/!! ! ! ! .!!;G!:;B;DD!:BB@!;.!:0EGD!:B.A!;.!:0@GD!:B0B!0B!:B0.!:A0;!;G!F:BA0!:E.C!0@!:./B!:EB;!/.!:/E@!:B@A!0@!F:.;G!:0A@!0E!1%23!+14"*!+5678H%$I6!+,-!0!! ! ! ! ! .!!;/!:C@CDD!:BBB!;/!:BA@!:EG0!/C!:.EE!:/@;!;/!F:B;G!:GGC!;B!:.C@!:;@.!/.!F:BB;!:@A;!00!F:B.B!:@CE!0E!1%23!+14"*!+5678#5#7H%&!+,-!0!! ! ! ! ! ! .!!;/!:B;@!:A.;!/C!:.;.!:0G;!;/!:.0E!:;B0!;B!:BE/!:G@B!/.!:./B!:CBA!00!F:/;B!:.CA!0E!'++'"!1%23!!+,-!.!! ! ! ! ! ! ! .!!0@!:C0EDD!:BB/!0B!:0C0!:B@@!/0!:/GA!:0A/!./!:CBADD!:BBG!/G!:..;!:E.C!//!'++'"!1%23!!+,-!/!! ! ! ! ! ! ! ! .!!;A!:/.@!:.GC!;B!F:B/0!:@//!/.!:/BE!:/B@!0@!F:B//!:A@G!0E!'++'"!1%23!!+,-!0!! ! ! ! ! ! ! ! ! .!!;B!:0@/!:BAA!/B!:0;;!:BCA!0.!:.E0!:0CB!0C!'++'"!1%23!!+,-!;!! ! ! ! ! ! ! ! ! ! .!!/.!:0@B!:.//!.G!:/EB!:/@G!.A!)<=JI>H5#I!*>%&<%$H#2!+,-!/!! ! ! ! ! ! ! ! ! ! ! .!!0@!:.@/!:0/@!/A!)<=JI>H5#I!*>%&<%$H#2!+,-!0!! ! ! ! ! ! ! ! ! ! ! ! .!!0E!!

Correlation of Measures of Psychotherapy Competency in Psychiatry Residents

https://escholarship.umassmed.edu/cgi/viewcontent.cgi?article=1474&amp;context=psych_pp

Correlation of Measures of Psychotherapy Competency in Psychiatry Residents

Abstract

Similar works

Full text

Available Versions

eScholarship@UMMS