The effects of language proficiency and awareness of time limit in animated vs. text-based situational judgment tests.

This project examined the influence of awareness of time limit and language proficiency in the HAM-SJT and the SST in a quasi high-stakes setting

Assessing Distinguishable Social Skills in Medical Admission - Does Construct-Driven Development Solve Validity Issues of Situational Judgment Tests?

Within this project, we aim at investigating whether construct validity issues of traditional Situational Judgment Tests (SJTs) in high-stakes medical admission can be overcome by using a construct-driven SJT development approach. Specifically, we introduce a construct-driven SJT measuring agency and communion as distinguishable and clearly defined constructs

Assessing Distinguishable Social Skills in Medical Admission - Does Construct-Driven Development Solve Validity Issues of Situational Judgment Tests?

Social skills are important for future physicians and are therefore increasingly considered in selection processes. One economic assessment method from which different social skills can be inferred are Situational Judgment Tests (SJT) in which applicants are asked to rate behavioral responses in context-relevant situations. However, traditional SJTs have so far failed to distinctively measure specified constructs. To address this shortcoming in the medical admission context, we applied a construct-driven approach of SJT development in which test development was deductively guided by agency and communion as target constructs. The final version of the construct-driven SJT includes 15 items per construct with three behavioral responses. Medical school applicants (N = 1,527) completed the construct-driven SJT, a traditional SJT, and an aptitude test under high-stakes condition as part of their application. A subsample (N = 575) participated in a subsequent voluntary online study with self-report measures of personality and past behavior. The proposed two-factor structure and internal consistency of the construct-driven SJT was confirmed. Communal SJT scores were positively associated with self-reported communal personality and communal behavior, yet effects were smaller than expected. Findings for agentic SJT scores were mixed with positive small associations to self-reported agentic personality scores and agentic behavior but unexpected negative relations to communal self-reported measures. Results suggest that construct-driven SJTs might overcome validity limitations of traditional SJTs, although their implementation is challenging. Despite first indicators of validity, future research needs to address practical points of application in high-stakes settings, inclusion of other constructs, and especially prediction of actual behavior before the application of construct-driven SJTs for selection purposes in medical admission can be recommended

Validity evidence for the Hamburg multiple mini-interview

Abstract Background Multiple mini-interviews (MMI) become increasingly popular for the selection of medical students. In this work, we examine the validity evidence for the Hamburg MMI. Methods We conducted three follow-up studies for the 2014 cohort of applicants to medical school over the course of two years. We calculated Spearman’s rank correlation (ρ) between MMI results and (1) emotional intelligence measured by the Trait Emotional Intelligence Questionnaire (TEIQue-SF) and the Situational Test of Emotion Management (STEM), (2) supervisors’ and practice team members’ evaluations of psychosocial competencies and suitability for the medical profession after a one-week 1:1 teaching in a general practice (GP) and (3) objective structured clinical examination (OSCE) scores. Results There were no significant correlations between MMI results and the TEIQue-SF (ρ = .07, p > .05) or the STEM (ρ = .05, p > .05). MMI results could significantly predict GP evaluations of psychosocial competencies (ρ = .32, p < .05) and suitability for the medical profession (ρ = .42, p < .01) as well as OSCE scores (ρ = .23, p < .05). The MMI remained a significant predictor of these outcomes in a robust regression model including gender and age as control variables. Conclusions Our findings suggest that MMIs can measure competencies that are relevant in a practical context. However, these competencies do not seem to be related to emotional intelligence as measured by self-report or situational judgement test

Measuring personal characteristics in applicants to German medical school: Piloting and online open-response Situational Judgement Test (Casper)

This project describes a pilot study of Casper, an online open-response Situational Judgement Test, in two cohorts of applicants to / students at German medical schools. We investigate its reliability, applicant acceptability, subgroup performance differences and validity to evaluate its use for the selection of German medical students

Additional file 1: of Validity evidence for the Hamburg multiple mini-interview

Appendix 1, Research questionnaire, English translation of the questionnaire which was developed for the general practice study (DOCX 30 kb

Additional file 2: of Validity evidence for the Hamburg multiple mini-interview

Appendix 2, Tests for control variables, Statistical analyses of the relationship between the control variables (gender, age, GPA and HAM-Nat) and study variables (MMI and outcome measures) (DOCX 14 kb