Assessing mathematical problem solving using comparative judgement

There is an increasing demand from employers and universities for school leavers to be able to apply their mathematical knowledge to problem solving in varied and unfamiliar contexts. These aspects are however neglected in most examinations of mathematics and, consequentially, in classroom teaching. One barrier to the inclusion of mathematical problem solving in assessment is that the skills involved are difficult to define and assess objectively. We present two studies that test a method called comparative judgement (CJ) that might be well suited to assessing mathematical problem solving. CJ is an alternative to traditional scoring that is based on collective expert judgements of students’ work rather than item-by-item scoring schemes. In Study 1 we used CJ to assess traditional mathematics tests and found it performed validly and reliably. In Study 2 we used CJ to assess mathematical problem-solving tasks and again found it performed validly and reliably. We discuss the implications of the results for further research and the implications of CJ for the design of mathematical problem-solving tasks

Improved imputation of low-frequency and rare variants using the UK10K haplotype reference panel

Imputing genotypes from reference panels created by whole-genome sequencing (WGS) provides a cost-effective strategy for augmenting the single-nucleotide polymorphism (SNP) content of genome-wide arrays. The UK10K Cohorts project has generated a data set of 3,781 whole genomes sequenced at low depth (average 7x), aiming to exhaustively characterize genetic variation down to 0.1% minor allele frequency in the British population. Here we demonstrate the value of this resource for improving imputation accuracy at rare and low-frequency variants in both a UK and an Italian population. We show that large increases in imputation accuracy can be achieved by re-phasing WGS reference panels after initial genotype calling. We also present a method for combining WGS panels to improve variant coverage and downstream imputation accuracy, which we illustrate by integrating 7,562 WGS haplotypes from the UK10K project with 2,184 haplotypes from the 1000 Genomes Project. Finally, we introduce a novel approximation that maintains speed without sacrificing imputation accuracy for rare variants

Adaptive comparative judgement: Adapting adaptive assessment to assess the quality of students’ work

It is increasingly acknowledged that adaptive assessment has a significant role to play in enhancing the assessment of and for learning. Computer adaptive testing (CAT) is a familiar method of delivering adaptive assessment, but it comes with a number of challenges that make it difficult to deploy, especially its reliance on questions (items) that can be presented and scored on computer, as well as the need for a large bank of items that have all been calibrated on large samples of suitable students. This paper focuses on a new form of adaptive assessment, where the adaptive nature of the approach is the very thing that makes it viable for large scale deployment. Adaptive comparative judgement (ACJ) is based on a historically sound approach developed by L.L. Thurstone in 1927 - the law of comparative judgement. The approach utilises paired comparisons to deliver highly reliable, non-subjective scaled ranking, but in its raw state, it relies on many paired comparisons to achieve a secure ranked result, making it impractical for use as an operational system. Through work carried out over several years, TAG Developments, working with Cambridge expert Alastair Pollitt, has developed a web-based application of Thurstone's law that is underpinned by an innovative adaptive algorithm which can generate a secure, scaled rank order of student work at least as efficiently as traditional marking. In addition, since assessors are not restricted to the types of task that can be marked reliably, they are free to use whatever methods they judge most authentic and valid. The system therefore delivers strong validity and exceptional reliability when compared to traditional criterion referenced assessment methods.In addition to outlining ACJ and its advantages and disadvantages, the paper also provides information on a number of research projects and live/pilot deployments where the approach has been used in an assessment context

How presentation affects the difficulty of computational thinking tasks

This paper discusses how a few changes in some computational thinking tasks proposed during the Bebras challenge affected the solvers\u2019 performance. After the 2016 challenge held in November in our country (Italy), we interviewed some participants on the difficulties they had faced and we modified some of the tasks accordingly. We then proposed the whole set of tasks, with some of them modified, to pupils who had not participated to the challenge in November and compared performances in the two sessions. Using Item Response Theory, we measured the change in the distribution of difficulty and discrimination of the modified tasks. On the basis of the obtained results, we tried to better understand the many factors which influenced the difference in performances, both in the conceptual and cognitive task content and in its presentation (text, images, layout)