Ratings of performance in multisource feedback: Comparing performance theories of residents and nurses

Background: Multisource feedback (MSF) is increasingly being used to assess trainee performance, with different assessor groups fulfilling a crucial role in utility of assessment data. However, in health professions education, research on assessor behaviors in MSF is limited. When assessing trainee performance in work settings, assessors use multidimensional conceptualizations of what constitutes effective performance, also called personal performance theories, to distinguish between various behaviors and sub competencies., This may not only explain assessor variability in Multi Source Feedback, but also result in differing acceptance (and use) of assessment data for developmental purposes. The purpose of this study was to explore performance theories of various assessor groups (residents and nurses) when assessing performance of residents.Methods: A constructivist, inductive qualitative research approach and semi-structured interviews following MSF were used to explore performance theories of 14 nurses and 15 residents in the department of internal medicine at Aga Khan University (AKU). Inductive thematic content analysis of interview transcripts was used to identify and compare key dimensions in residents\u27 and nurses\u27 performance theories used in evaluation of resident performance.Results: Seven major themes, reflecting key dimensions of assessors\u27 performance theories, emerged from the qualitative data, namely; communication skills, patient care, accessibility, teamwork skills, responsibility, medical knowledge and professional attitude. There were considerable overlaps, but also meaningful differences in the performance theories of residents and the nurses, especially with respect to accessibility, teamwork and medical knowledge.Conclusion: Residents\u27 and nurses\u27 performance theories for assessing resident performance overlap to some extent, yet also show meaningful differences with respect to the performance dimensions they pay attention to or consider most important. In MSF, different assessor groups may therefore hold different performance theories, depending on their role. Our results further our understanding of assessor source effects in MSF. Implications of our findings are related to implementation of MSF, design of rating scales as well as interpretation and use of MSF data for selection and performance improvement

From aggregation to interpretation:how assessors judge complex data in a competency-based portfolio

While portfolios are increasingly used to assess competence, the validity of such portfolio-based assessments has hitherto remained unconfirmed. The purpose of the present research is therefore to further our understanding of how assessors form judgments when interpreting the complex data included in a competency-based portfolio. Eighteen assessors appraised one of three competency-based mock portfolios while thinking aloud, before taking part in semi-structured interviews. A thematic analysis of the think-aloud protocols and interviews revealed that assessors reached judgments through a 3-phase cyclical cognitive process of acquiring, organizing, and integrating evidence. Upon conclusion of the first cycle, assessors reviewed the remaining portfolio evidence to look for confirming or disconfirming evidence. Assessors were inclined to stick to their initial judgments even when confronted with seemingly disconfirming evidence. Although assessors reached similar final (pass-fail) judgments of students' professional competence, they differed in their information-processing approaches and the reasoning behind their judgments. Differences sprung from assessors' divergent assessment beliefs, performance theories, and inferences about the student. Assessment beliefs refer to assessors' opinions about what kind of evidence gives the most valuable and trustworthy information about the student's competence, whereas assessors' performance theories concern their conceptualizations of what constitutes professional competence and competent performance. Even when using the same pieces of information, assessors furthermore differed with respect to inferences about the student as a person as well as a (future) professional. Our findings support the notion that assessors' reasoning in judgment and decision-making varies and is guided by their mental models of performance assessment, potentially impacting feedback and the credibility of decisions. Our findings also lend further credence to the assertion that portfolios should be judged by multiple assessors who should, moreover, thoroughly substantiate their judgments. Finally, it is suggested that portfolios be designed in such a way that they facilitate the selection of and navigation through the portfolio evidence

Exploring the influence of cultural orientations on assessment of communication behaviours during patient-practitioner interactions

Background Research has shown that patients’ and practitioners’ cultural orientations affect communication behaviors and interpretations in cross-cultural patient-practitioner interactions. Little is known about the effect of cultural orientations on assessment of communication behaviors in cross-cultural educational settings. The purpose of this study is to explore cultural orientation as a potential source of assessor idiosyncrasy or between-assessor variability in assessment of communication skills. More specifically, we explored if and how (expert) assessors’ valuing of communication behaviours aligned with their cultural orientations (power-distance, masculinity-femininity, uncertainty avoidance, and individualism-collectivism). Methods Twenty-five pharmacist-assessors watched 3 videotaped scenarios (patient-pharmacist interactions) and ranked each on a 5-point global rating scale. Videotaped scenarios demonstrated combinations of well-portrayed and borderline examples of instrumental and affective communication behaviours. We used stimulated recall and verbal protocol analysis to investigate assessors’ interpretations and evaluations of communication behaviours. Uttered assessments of communication behaviours were coded as instrumental (task-oriented) or affective (socioemotional) and either positive or negative. Cultural orientations were measured using the Individual Cultural Values Scale. Correlations between cultural orientations and global scores, and frequencies of positive, negative, and total utterances of instrumental and affective behaviours were determined. Results Correlations were found to be scenario specific. In videos with poor or good performance, no differences were found across cultural orientations. When borderline performance was demonstrated, high power-distance and masculinity were significantly associated with higher global ratings (r = .445, and .537 respectively, p < 0.05) as well as with fewer negative utterances regarding instrumental (task focused) behaviours (r = −.533 and − .529, respectively). Higher masculinity scores were furthermore associated with positive utterances of affective (socioemotional) behaviours (r = .441). Conclusions Our findings thus confirm cultural orientation as a source of assessor idiosyncrasy and meaningful variations in interpretation of communication behaviours. Interestingly, expert assessors generally agreed on scenarios of good or poor performances but borderline performance was influenced by cultural orientation. Contrary to current practices of assessor and assessment instrument standardization, findings support the use of multiple assessors for patient-practitioner interactions and development of qualitative assessment tools to capture these varying, yet valid, interpretations of performanceScopu

ADEA‐ADEE Shaping the Future of Dental Education III: Assessment in Competency‐Based Dental Education: Ways Forward

Assessment in competency-based dental education continues to be a recognized area for growth and development within dental programs around the world. At the joint American Dental Education Association (ADEA) and the Association for Dental Education in Europe (ADEE) 2019 conference, Shaping the Future of Dental Education III, the workshop on assessment was designed to continue the discussion started in 2017 at the ADEA-ADEE Shaping the Future of Dental Education II.1 The focus of the 2019 conference involved examining the potential of entrustable professional activities (EPAs) and current thinking about workplace-based assessment (WBA) within competency-based education in the 21st century. Approximately 30 years ago, George Miller wrote about the assessment of competence in medical education and challenged faculty to reach for higher levels of assessment than knowledge or skill.2 Acknowledging that no one assessment method can result in a valid assessment of competence, Miller proposed a four-level framework for assessment. The lowest level involves measuring what students know ( knows ), followed by assessment of the skill with which knowledge is applied in relevant tasks or problems ( knows how ). Next is an assessment of task performance in standardized settings ( shows how ), and finally, the highest level assesses the student\u27s performance in the unstandardized clinical workplace ( does ). The 2019 assessment workshop focused on advances in the assessment of learners in the unstandardized workplace-the highest level of Miller\u27s assessment pyramid ( does ). Research has shown that dental education has struggled to implement assessment strategies that meet this level.3 The workshop brought together individuals from around the world, with an interest in assessment in dental education, to consider how assessment in the does level, specifically EPAs and WBA, factors into competence assessment in dentistry/dental educatio

Genomic SNP array as a gold standard for prenatal diagnosis of foetal ultrasound abnormalities

Background: We have investigated whether replacing conventional karyotyping by SNP array analysis in cases of foetal ultrasound abnormalities would increase the diagnostic yield and speed of prenatal diagnosis in clinical practice. Findings/results. From May 2009 till June 2011 we performed HumanCytoSNP-12 array (HCS) (http://www.Illumina.com) analysis in 207 cases of foetal structural abnormalities. HCS allows detecting unbalanced genomic abnormalities with a resolution of about 150/200 kb. All cases were selected by a clinical geneticist after excluding the most common aneuploidies by RAD (rapid aneuploidy detection). Pre-test genetic counselling was offered in all cases. In 24/207 (11,6%) foetuses a clinically relevant genetic abnormality was detected. Only 8/24 abnormalities would have been detected if only routine karyotyping was performed. Submicroscopic abnormalities were found in 16/207 (7,7%) cases. The array results were achieved within 1-2 weeks after amniocentesis. Conclusions: Prenatal SNP array testing is faster than karyotyping and allows detecting much smaller aberrations (∼0.15 Mb) in addition to the microscopic unbalanced chromosome abnormalities detectable with karyotyping (∼ > 5 Mb). Since karyotyping would have missed 66% (16/24) of genomic abnormalities in our cohort, we propose to perform genomic high resolution array testing assisted by pre-test counselling as a primary prenatal diagnostic test in cases of foetal ultrasound abnormalities

Placental studies elucidate discrepancies between NIPT showing a structural chromosome aberration and a differently abnormal fetal karyotype

Objective: Placental cytogenetic studies may reveal the origin of discordant noninvasive prenatal testing (NIPT). We performed placental studies to elucidate discordances between NIPT showing a structural chromosome aberration and the fetus having a different chromosome aberration in three cases. Method: Diagnostic testing with genomic SNP microarray was performed in three cases with NIPT showing a duplication on 4q (case 1), a terminal deletion of 13q (case 2), and a terminal deletion of 15q (case 3). Placental studies involved SNP array analysis of cytotrophoblast and mesenchymal core of chorionic villi of four placental quadrants. Clinical follow-up was performed as well. Results: Amniotic fluid revealed a different structural chromosome aberration than predicted by NIPT: a terminal 2q deletion (case 1), a segmental uniparental isodisomy of 13q (case 2), and a terminal duplication of 15q and of 13q (case 3). Placental studies revealed the aberration detected with NIPT in the cytotrophoblast, whereas the fetal karyotype was confirmed in the placental mesenchymal core. Conclusion: Our study shows that targeted cytogenetic investigations for confirmation of NIPT showing a microscopically visible structural chromosome aberration should be avoided, since another aberration, even a submicroscopic one or one involving another chromosome, may be present in the fetus

Unexpected finding of uniparental disomy mosaicism in term placentas: Is it a common feature in trisomic placentas?

Objective: Non-invasive prenatal testing (NIPT) detects placental chromosome aberrations. When amniocentesis reveals a normal karyotype, confined placental mosaicism (CPM) may be assumed. In order to confirm this, placental cytogenetic studies were performed. Method: NIPT was conducted in the course of the Dutch TRIDENT study. Placentas of 10 cases with NIPT results indicating an autosomal trisomy and showing a normal (N = 9) or low mosaic karyotype (N = 1) in amniotic fluid (AF) were investigated. The cytotrophoblast as well as the mesenchymal core of two to four placental chorionic villi biopsies were studied with single nucleotide polymorphism (SNP) array. Clinical outcome data were collected. Results: In 10/10 cases, CPM was proven. In 3/10 cases trisomy/uniparental disomy (UPD)/biparental disomy (BPD) mosaicism was discovered. In 2/3 cases, all three cell lines were present in the placenta, whereas BPD was found in AF. In 1/3 cases trisomy 22/UPD22 was present in AF while trisomy 22/BPD22 mosaicism was found in the placenta. Five of 10 pregnancies were affected with pre-eclampsia, low birth weight, preterm delivery, and/or congenital malformations. Conclusion: The presence of trisomy/UPD/BPD mosaicism in 3/10 cases that we investigated proves that trisomic zygote rescue may involve multiple rescue events during early embryogenesis. UPD mosaicism, when present in crucial fetal tissues, may explain the abnormal phenotype in undiagnosed cases