Adaptive comparative judgement: Adapting adaptive assessment to assess the quality of students’ work

Abstract

It is increasingly acknowledged that adaptive assessment has a significant role to play in enhancing the assessment of and for learning. Computer adaptive testing (CAT) is a familiar method of delivering adaptive assessment, but it comes with a number of challenges that make it difficult to deploy, especially its reliance on questions (items) that can be presented and scored on computer, as well as the need for a large bank of items that have all been calibrated on large samples of suitable students. This paper focuses on a new form of adaptive assessment, where the adaptive nature of the approach is the very thing that makes it viable for large scale deployment. Adaptive comparative judgement (ACJ) is based on a historically sound approach developed by L.L. Thurstone in 1927 - the law of comparative judgement. The approach utilises paired comparisons to deliver highly reliable, non-subjective scaled ranking, but in its raw state, it relies on many paired comparisons to achieve a secure ranked result, making it impractical for use as an operational system. Through work carried out over several years, TAG Developments, working with Cambridge expert Alastair Pollitt, has developed a web-based application of Thurstone's law that is underpinned by an innovative adaptive algorithm which can generate a secure, scaled rank order of student work at least as efficiently as traditional marking. In addition, since assessors are not restricted to the types of task that can be marked reliably, they are free to use whatever methods they judge most authentic and valid. The system therefore delivers strong validity and exceptional reliability when compared to traditional criterion referenced assessment methods.In addition to outlining ACJ and its advantages and disadvantages, the paper also provides information on a number of research projects and live/pilot deployments where the approach has been used in an assessment context