Assessment of clinical reasoning is often challenging, as it is a complex process of thinking and decision making. Script Concordance Testing (SCT), using authentic clinical scenarios with diagnostic or management uncertainties, has been developed to assess clinical reasoning. As SCT is a relatively new assessment modality, more empirical evidence is needed to support the validity of SCT scores.
This thesis examines key aspects of the validity of SCT scores in the assessment of the clinical reasoning ability of medical undergraduates. A review of the current literature informs the use, design and standard setting of SCT, as well as evidence for its reliability and validity. Exploration of the response patterns of 5 cohorts of graduate-entry medical students in an Australian Medical School showed deliberate avoidance of extreme responses by the lowest quartile students. A post-hoc simulation study, testing the hypothesis that test-wise candidates’ SCT scores were inflated through deliberate avoidance of extreme response-options and selection of neutral response-options, generated an approach to optimising and balancing SCT items for improved SCT score validity.
In response to the paucity of empirical studies on the construct validity for SCT scores, the next study showed evidence of progression in SCT scores from medical students, to junior registrars, to experienced general practitioners. Finally, an investigation of candidates’ response process, using a ‘think-aloud’ approach, supported the response process validity of SCT scores.
In conclusion, this thesis has demonstrated that: 1) thoughtful design and balance of SCT items can mitigate some of the validity threats to medical student SCT scores; 2) the tendency of SCT scores to progress with increasing levels of clinical practice experience further supports the construct validity of SCT scores; and 3) use of the ‘think-aloud’ approach to explore students’ response process may enhance the utility and educational benefits of SCT. The research supports the validity of SCT in assessing clinical reasoning in undergraduate medical education, and presents practical approaches to enhance the design of the assessment instrument
