The Predictive Validity of a Text-Based Situational Judgment Test in Undergraduate Medical and Dental School Admissions

PROBLEM: Situational judgment tests (SJTs) can be used to assess the nonacademic attributes necessary for medical and dental trainees to become successful practitioners. Evidence for SJTs' predictive validity, however, relates predominantly to selection in postgraduate settings or using video-based SJTs at the undergraduate level; it may not be directly transferable to text-based SJTs in undergraduate medical and dental school selection. This preliminary study aimed to address these gaps by assessing the validity of the UK Clinical Aptitude Test (UKCAT) text-based SJT. APPROACH: Study participants were 218 first-year medical and dental students from four UK undergraduate schools who completed the first UKCAT text-based SJT in 2013. Outcome measures were educational supervisor ratings of in-role performance in problem-based learning tutorial sessions-mean rating across the three domains measured by the SJT (integrity, perspective taking, and team involvement) and an overall judgment of performance-collected in 2015. OUTCOMES: There were significant correlations between SJT scores and both mean supervisor ratings (uncorrected r = 0.24, P < .001; corrected r = 0.34) and overall judgments (uncorrected rs = 0.16, P < .05; corrected rs = 0.20). SJT scores predicted 6% of variance in mean supervisor ratings across the three nonacademic domains. NEXT STEPS: The results provide evidence that a well-designed text-based SJT can be appropriately integrated, and add value to, the selection process for undergraduate medical and dental school. More evidence is needed regarding the longitudinal predictive validity of SJTs throughout medical and dental training pathways, with appropriate outcome criteria

A preliminary investigation to explore the cognitive resources of physicians experiencing difficulty in training

Background: Treating patients is complex, and research shows that there are differences in cognitive resources between physicians who experience difficulties, and those who do not. It is possible that differences in some cognitive resources could explain the difficulties faced by some physicians. In this study, we explore differences in cognitive resources between different groups of physicians (that is, between native (UK) physicians and International Medical Graduates (IMG); those who continue with training versus those who were subsequently removed from the training programme); and also between physicians experiencing difficulties compared with the general population. Methods: A secondary evaluation was conducted on an anonymised dataset provided by the East Midlands Professional Support Unit (PSU). One hundred and twenty one postgraduate trainee physicians took part in an Educational Psychology assessment through PSU. Referrals to the PSU were mainly on the basis of problems with exam progression and difficulties in communication skills, organisation and confidence. Cognitive resources were assessed using the Wechsler Adult Intelligence Scale (WAIS-IV). Physicians were categorised into three PSU outcomes: ‘Continued in training’, ‘Removed from training’ and ‘Active’ (currently accessing the PSU). Results: Using a one-sample Z test, we compared the referred physician sample to a UK general population sample on the WAIS-IV and found the referred sample significantly higher in Verbal Comprehension (VCI; z = 8.78) and significantly lower in Working Memory (WMI; z = −4.59). In addition, the native sample were significantly higher in Verbal Comprehension than the UK general population sample (VCI; native physicians: z = 9.95, p < .001, d = 1.25), whilst there was a lesser effect for the difference between the IMG sample and the UK general population (z = 2.13, p = .03, d = 0.29). Findings also showed a significant difference in VCI scores between those physicians who were ‘Removed from training’ and those who ‘Continued in training’. Conclusions: Our results suggest it is important to understand the cognitive resources of physicians to provide a more focussed explanation of those who experience difficulties in training. This will help to implement more targeted interventions to help physicians develop compensatory strategies

A preliminary investigation to explore the cognitive resources of physicians experiencing difficulty in training

Background Treating patients is complex, and research shows that there are differences in cognitive resources between physicians who experience difficulties, and those who do not. It is possible that differences in some cognitive resources could explain the difficulties faced by some physicians. In this study, we explore differences in cognitive resources between different groups of physicians (that is, between native (UK) physicians and International Medical Graduates (IMG); those who continue with training versus those who were subsequently removed from the training programme); and also between physicians experiencing difficulties compared with the general population. Methods A secondary evaluation was conducted on an anonymised dataset provided by the East Midlands Professional Support Unit (PSU). One hundred and twenty one postgraduate trainee physicians took part in an Educational Psychology assessment through PSU. Referrals to the PSU were mainly on the basis of problems with exam progression and difficulties in communication skills, organisation and confidence. Cognitive resources were assessed using the Wechsler Adult Intelligence Scale (WAIS-IV). Physicians were categorised into three PSU outcomes: ‘Continued in training’, ‘Removed from training’ and ‘Active’ (currently accessing the PSU). Results Using a one-sample Z test, we compared the referred physician sample to a UK general population sample on the WAIS-IV and found the referred sample significantly higher in Verbal Comprehension (VCI; z = 8.78) and significantly lower in Working Memory (WMI; z = −4.59). In addition, the native sample were significantly higher in Verbal Comprehension than the UK general population sample (VCI; native physicians: z = 9.95, p < .001, d = 1.25), whilst there was a lesser effect for the difference between the IMG sample and the UK general population (z = 2.13, p = .03, d = 0.29). Findings also showed a significant difference in VCI scores between those physicians who were ‘Removed from training’ and those who ‘Continued in training’. Conclusions Our results suggest it is important to understand the cognitive resources of physicians to provide a more focussed explanation of those who experience difficulties in training. This will help to implement more targeted interventions to help physicians develop compensatory strategies

Evaluating the complementary roles of an SJT and academic assessment for entry into clinical practice

Although there is extensive evidence confirming the predictive validity of situational judgement tests (SJTs) in medical education, there remains a shortage of evidence for their predictive validity for performance of postgraduate trainees in their first role in clinical practice. Moreover, to date few researchers have empirically examined the complementary roles of academic and non-academic selection methods in predicting in-role performance. This is an important area of enquiry as despite it being common practice to use both types of methods within a selection system, there is currently no evidence that this approach translates into increased predictive validity of the selection system as a whole, over that achieved by the use of a single selection method. In this preliminary study, the majority of the range of scores achieved by successful applicants to the UK Foundation Programme provided a unique opportunity to address both of these areas of enquiry. Sampling targeted high (>80th percentile) and low (<20th percentile) scorers on the SJT. Supervisors rated 391 trainees’ in-role performance, and incidence of remedial action was collected. SJT and academic performance scores correlated with supervisor ratings (r = .31 and .28, respectively). The relationship was stronger between the SJT and in-role performance for the low scoring group (r = .33, high scoring group r = .11), and between academic performance and in-role performance for the high scoring group (r = .29, low scoring group r = .11). Trainees with low SJT scores were almost five times more likely to receive remedial action. Results indicate that an SJT for entry into trainee physicians’ first role in clinical practice has good predictive validity of supervisor-rated performance and incidence of remedial action. In addition, an SJT and a measure of academic performance appeared to be complementary to each other. These initial findings suggest that SJTs may be more predictive at the lower end of a scoring distribution, and academic attainment more predictive at the higher end

Recruiting for values in healthcare: a preliminary review of the evidence

Displaying compassion, benevolence and respect, and preserving the dignity of patients are important for any healthcare professional to ensure the provision of high quality care and patient outcomes. This paper presents a structured search and thematic review of the research evidence relating to values-based recruitment within healthcare. Several different databases, journals and government reports were searched to retrieve studies relating to values-based recruitment published between 1998 and 2013, both in healthcare settings and other occupational contexts. There is limited published research related to values-based recruitment directly, so the available theoretical context of values is explored alongside an analysis of the impact of value congruence. The implications for the design of selection methods to measure values is explored beyond the scope of the initial literature search. Research suggests some selection methods may be appropriate for values-based recruitment, such as situational judgment tests (SJTs), structured interviews and multiple-mini interviews (MMIs). Personality tests were also identified as having the potential to compliment other methods (e.g. structured interviews), as part of a values-based recruitment agenda. Methods including personal statements, references and unstructured/‘traditional’ interviews were identified as inappropriate for values-based recruitment. Practical implications are discussed in the context of values-based recruitment in the healthcare context. Theoretical implications of our findings imply that prosocial implicit trait policies, which could be measured by selection tools such as SJTs and MMIs, may be linked to individuals’ values via the behaviours individuals consider to be effective in given situations. Further research is required to state this conclusively however, and methods for values-based recruitment represent an exciting and relatively unchartered territory for further research