Multiple choice for honours-level students? A statistical evaluation

Most computer-aided assessments use non-subjective techniques, such as multiple-choice questions (MCQs). However, when preparing summative assessments for honours-level degree students in their final year, MCQs are rarely considered suitable; by and large we tend to use traditional essay-type questions which are not quite so amenable to computer assistance. We present here a technique for evaluating MCQs statistically. This evaluation can be used for: identifying how an MCQ can be improved; and proving that an MCQ test is a valid technique for assessing honourslevel students. We also present a new form of MCQ, called the Permutational Multiple Choice Question (PMCQ) or "Two-stem" MCQ. These questions have been trialled with honours-level students. Statistical analysis of the trials indicates that these PMCQs can be as good a predictor of overall performance as essaytype questions. The statistical analysis also supports practical suggestions for improving PMCQs. We conclude that statistical analysis is useful, and good computer aided assessment systems should assist the examiner in analysing test results statistically