Standard setting: Comparison of two methods

BACKGROUND: The outcome of assessments is determined by the standard-setting method used. There is a wide range of standard – setting methods and the two used most extensively in undergraduate medical education in the UK are the norm-reference and the criterion-reference methods. The aims of the study were to compare these two standard-setting methods for a multiple-choice question examination and to estimate the test-retest and inter-rater reliability of the modified Angoff method. METHODS: The norm – reference method of standard -setting (mean minus 1 SD) was applied to the 'raw' scores of 78 4th-year medical students on a multiple-choice examination (MCQ). Two panels of raters also set the standard using the modified Angoff method for the same multiple-choice question paper on two occasions (6 months apart). We compared the pass/fail rates derived from the norm reference and the Angoff methods and also assessed the test-retest and inter-rater reliability of the modified Angoff method. RESULTS: The pass rate with the norm-reference method was 85% (66/78) and that by the Angoff method was 100% (78 out of 78). The percentage agreement between Angoff method and norm-reference was 78% (95% CI 69% – 87%). The modified Angoff method had an inter-rater reliability of 0.81 – 0.82 and a test-retest reliability of 0.59–0.74. CONCLUSION: There were significant differences in the outcomes of these two standard-setting methods, as shown by the difference in the proportion of candidates that passed and failed the assessment. The modified Angoff method was found to have good inter-rater reliability and moderate test-retest reliability

Summative and Formative Style Anatomy Practical Examinations: Do They Have Impact on Students’ Performance and Drive for Learning?

Anatomical knowledge is commonly assessed by practical examinations that are often administered in summative format. The format of anatomy practical examination was changed at the Lee Kong Chian School of Medicine in Singapore from summative (graded; must pass) to formative (ungraded; no pass/fail) in academic year (AY) 2017-2018. Both assessment formats were undertaken online, but the formative mode used a team-based learning activity comprising individual and team assessments. This gave an unique opportunity to investigate: (1) the impact of two different online assessment formats on student performance in practical examination; (2) the impact of new formative practical examination on students' performance in summative examinations; and (3) students' opinions of these two practical examination formats. The class of 2021 perceptions was obtained as they experienced both formats. A retrospective cohort study was also conducted to analyze the Year 2 students' performance in anatomy practical and year-end summative examinations of cohorts AY 2015-2016, AY 2016-2017 (summative format), and AY 2017-2018 (formative format). There were no significant differences in students' performance between two practical examination formats. The cohort who experienced the formative format, performed significantly better in summative examinations (mean ± SD: 82.32 ± 10.22%) compared with the cohort who experienced the summative format (73.77 ± 11.09%) (P < 0.001). Students highlighted positive features of the formative practical examination, including team reinforcement of learning, instant feedback, and enhanced learning. These findings indicate that students continue to study for anatomy practical examination without the need for external drivers. The team-based learning style practical examination enhances students' performance in summative examinations