Standard setting: Comparison of two methods

BACKGROUND: The outcome of assessments is determined by the standard-setting method used. There is a wide range of standard – setting methods and the two used most extensively in undergraduate medical education in the UK are the norm-reference and the criterion-reference methods. The aims of the study were to compare these two standard-setting methods for a multiple-choice question examination and to estimate the test-retest and inter-rater reliability of the modified Angoff method. METHODS: The norm – reference method of standard -setting (mean minus 1 SD) was applied to the 'raw' scores of 78 4th-year medical students on a multiple-choice examination (MCQ). Two panels of raters also set the standard using the modified Angoff method for the same multiple-choice question paper on two occasions (6 months apart). We compared the pass/fail rates derived from the norm reference and the Angoff methods and also assessed the test-retest and inter-rater reliability of the modified Angoff method. RESULTS: The pass rate with the norm-reference method was 85% (66/78) and that by the Angoff method was 100% (78 out of 78). The percentage agreement between Angoff method and norm-reference was 78% (95% CI 69% – 87%). The modified Angoff method had an inter-rater reliability of 0.81 – 0.82 and a test-retest reliability of 0.59–0.74. CONCLUSION: There were significant differences in the outcomes of these two standard-setting methods, as shown by the difference in the proportion of candidates that passed and failed the assessment. The modified Angoff method was found to have good inter-rater reliability and moderate test-retest reliability

Repairing Leaks in the Chemistry Teacher Pipeline: A Longitudinal Analysis of Praxis Chemistry Subject Assessment Examinees and Scores

Teachers play a critical role in the preparation of future science, technology, engineering, and mathematics majors and professionals. What teachers know about their discipline (i.e., content knowledge) has been identified as an important aspect of instructional effectiveness; however, studies have not yet assessed the content knowledge of aspiring chemistry teachers in the United States. The Praxis Chemistry Subject Assessment is the most nationally representative measure of teacher content knowledge, used in 39 U.S. states in the past decade. In the presented study, we report findings concerning (i) the demographics of Praxis Chemistry Subject Assessment examinees (i.e., prospective chemistry teachers); and (ii) the longitudinal trends in exam performance across several demographic test-taker characteristics. These findings reveal substantial differences in performance and pass rates among examinees of different genders, races/ethnicities, undergraduate majors, undergraduate GPAs, and geographic locales in which they intend to teach. We establish potential leaks in the teacher pipeline that may impact the quality and diversity of chemistry teachers in the United States and suggest ways to improve the chemistry teaching workforce