Scoring method of a Situational Judgment Test:influence on internal consistency reliability, adverse impact and correlation with personality?

textabstractSituational Judgment Tests (SJTs) are increasingly used for medical school selection. Scoring an SJT is more complicated than scoring a knowledge test, because there are no objectively correct answers. The scoring method of an SJT may influence the construct and concurrent validity and the adverse impact with respect to non-traditional students. Previous research has compared only a small number of scoring methods and has not studied the effect of scoring method on internal consistency reliability. This study compared 28 different scoring methods for a rating SJT on internal consistency reliability, adverse impact and correlation with personality. The scoring methods varied on four aspects: the way of controlling for systematic error, and the type of reference group, distance and central tendency statistic. All scoring methods were applied to a previously validated integrity-based SJT, administered to 931 medical school applicants. Internal consistency reliability varied between .33 and .73, which is likely explained by the dependence of coefficient alpha on the total score variance. All scoring methods led to significantly higher scores for the ethnic majority than for the non-Western minorities, with effect sizes ranging from 0.48 to 0.66. Eighteen scoring methods showed a significant small positive correlation with agreeableness. Four scoring methods showed a significant small positive correlation with conscientiousness. The way of controlling for systematic error was the most influential scoring method aspect. These results suggest that the increased use of SJTs for selection into medical school must be accompanied by a thorough examination of the scoring method to be used

The self-regulated learning of medical students in the clinical environment – a scoping review

Abstract Background Self-regulated learning is the individual’s ability to effectively use various strategies to reach their learning goals. We conducted this scoping review to explore what has been found regarding self-regulated learning in the clinical environment and how this was measured. Methods Using Arksey and O’Malley’s five-stage framework, we searched three medical and educational databases as well as Google Scholar for literature on the self-regulated learning of medical students in the clinical environment published between 1966 and February 2017. After results were screened and relevant studies were identified, the data was summarised and discursively reported. Results The search resulted in 911 articles, with 14 articles included in the scoping review after the inclusion criteria was applied. Self-regulated learning was explored in these studies in various ways including qualitative, quantitative and mixed methods. Three major findings were found: 1) levels of self-regulated learning change in the clinical environment, 2) self-regulated learning is associated with academic achievement, success in clinical skills and mental health and 3) various factors can support self-regulated learning levels in medical students. Conclusions Most of articles exploring the self-regulated learning of medical students during the clinical years have been published in the last 5 years, suggesting a growing interest in the area. Future research could explore the self-regulated learning levels of medical students during the clinical years using a longitudinal approach or through the use of novel qualitative approaches