Five go marking an exam question: the use of Adaptive Comparative Judgement to manage subjective bias

Adaptive Comparative Judgement (ACJ) is an alternative to conventional marking in which the assessor (judge) merely compares two answers and chooses a winner. (Scripts are typically uploaded to the CompareAssess interface as pdf files and are presented side-by-side.) Repeated comparisons and application of the sorting algorithm leads to scripts sorted in order of merit. Boundaries are determined by separate review of scripts. A small pilot of ACJ in the fourth year of the Manchester Pharmacy programme is described. Twelve judges used ACJ to mark 64 scripts previously marked conventionally. 50 students peer-marked their own mock examination question using ACJ. Peer-marking was successful with students learning from the process, and delivering both marks and feedback within two weeks. There was very good consistency among the students acting as judges, and accuracy (as defined by Pollitt, 2012) of 0.94. Staff were similarly consistent, but the agreement with marks obtained by conventional marking was disappointing. While some discrepancies could be attributed to conventional marking failing under the stress of marking during teaching term, the worst discrepancies appeared to originate from inadequate judging criteria. We conclude that ACJ is a very promising method, especially for peer assessment, but that judging criteria require very careful consideration

Adaptive Comparative Judgement: A Tool to Support Students’ Assessment Literacy

Comparative judgment in assessment is a process whereby repeated comparison of two items (e.g., assessment answers) can allow an accurate ranking of all the submissions to be achieved. In adaptive comparative judgment (ACJ), technology is used to automate the process and present pairs of pieces of work over iterative cycles. An online ACJ system was used to present students with work prepared by a previous cohort at the same stage of their studies. Objective marks given to the work by experienced faculty were compared to the rankings given to the work by a cohort of veterinary students (n=154). Each student was required to review and judge 20 answers provided by the previous cohort to a free-text short answer question. The time that students spent on the judgment tasks was recorded, and students were asked to reflect on their experiences after engaging with the task. There was a strong positive correlation between student ranking and faculty marking. A weak positive correlation was found between the time students spent on the judgments and their performance on the part of their own examination that contained questions in the same format. Slightly less than half of the students agreed that the exercise was a good use of their time, but 78% agreed that they had learned from the process. Qualitative data highlighted different levels of benefit from the simplest aspect of learning more about the topic to an appreciation of the more generic lessons to be learned

Do academics doubt their own research?

When do experts doubt or question their own previously published research and why? An online survey was designed and distributed across academic staff and postgraduate research students at different universities in Great Britain. Respondents (n = 202 - 244) identified the likelihoods of six different (quasi) hypothetical occurrences causing them to doubt or question work they have published in peer reviewed journals. They are: two objective and two semi-objective citation based metrics, plus two semi-objective metrics based on verbalised reactions. Only limited support is found from this study to suggest that the authors of primary research would agree with any judgements made by others about their research based on these metrics. The occurrence most likely to cause respondents to doubt or question their previously published research was where the majority of citing studies suggested mistakes in their work. In a multivariate context, only age and nationality are significant determinants of doubt beyond average likelihoods. Understanding and acknowledging what makes authors of primary research doubt their own research could increase the validity of those who pass judgement

ComPAIR: A new online tool using adaptive comparative judgement to support learning with peer feedback

Peer feedback is a useful strategy in teaching and learning, but its effectiveness particularly in introductory courses can be limited by the relative newness of students to both the body of knowledge upon which they are being asked to provide feedback and the skill set involved in providing good feedback. This paper applies a novel approach to facilitating novice feedback: making use of students’ inherent ability to compare. The ComPAIR application discussed in this article scaffolds peer feedback through comparisons, asking students to choose the “better” of two answers in a series of pairings offered in an engaging online context. In contrast to other peer-feedback approaches that seek to train novices to be able to provide expert feedback (such as calibrated peer review) or to crowdsource grading, ComPAIR focuses upon the benefits to be gained from the critical process of comparison and ranking. The tool design is based on the longstanding psychological principle of comparative judgement, by which novices who may not yet have the compass to assess others’ work confidently can still rank content as “better” with accuracy. Data from 168 students in pilot studies in English, Physics and Math courses at the University of British Columbia are reviewed. Though the use of ComPAIR required little classroom time, students perceived this approach to increase their facility with course content, their ability assess their own work, and their capacity to provide feedback on the work of others in a collaborative learning environment

Effective Philanthropy: Towards a Research Agenda - A White Paper

Many people look at getting people to give more. Giving Evidence and the Social Enterprise Initiative at the University of Chicago Booth School of Business have been looking at getting donors to give better. Perhaps improving giving will achieve more than increasing it: For instance, the cost of raising capital for charities is about 20-40 per cent, against only about 3-5 per cent for companies, and charities turn away some donors who are fiddly to deal with. It may be easier to reduce that cost of capital than to raise the amount given. Plus, money doesn't always go where it's most needed: for example, about 90 per cent of global health spending goes on 10 percent of the disease burden -- maybe those donations can cheaply be re-directed. Our white paper looks at at (i)what good giving is, i.e., what donor behaviours produce the best outcomes, and (ii)how to persuade/enable/nudge donors to do those behaviours. It collates what is known on these topics, and lays out many unanswered questions which would form a strong research agenda. [The Chicago Booth School of Business was recently ranked by The Economist as the best business school in the world. And its leading centre on decision science is highly relevant since decisions are so integral to giving.] The white paper identifies questions which non-profits, funders and other practitioners want answered about making giving better, and aims to encourage researchers to address them

Mission drift in qualitative research, or moving toward a systematic review of qualitative studies, moving back to a more systematic narrative review

The paper argues that the systematic review of qualitative research is best served by reliance upon qualitative methods themselves. A case is made for strengthening the narrative literature review and using narrative itself as a method of review. A technique is proposed that builds upon recent developments in qualitative systematic review by the use of a narrative inductive method of analysis. The essence of qualitative work is described. The natural ability for issues of ethnicity and diversity to be investigated through a qualitative approach is elaborated. Recent developments in systematic review are delineated, including the Delphi and Signal and Noise techniques, inclusion of grey literature, scoping studies and meta-ethnography. A narrative inductive interpretive method to review qualitative research is proposed, using reflective teams to analyse documents. Narrative is suggested as a knowledge-generating method and its underlying hermeneutic approach is defended as providing validity and theoretical structure. Finally, qualities that distinguish qualitative research from more quantitative investigations are delineated. Starting points for reflecting on qualitative studies and their usefulness are listed. Key words: Qualitative Systematic Review, Evidence-Based Policy, Grey Literature, Scoping Studies, Delphi, ‘Signal and Noise’, Meta-ethnography, Narrative Review, Narrative Method, and Reflective Teams

Riconcettualizzare il feedback come processo interno e non esterno

Students are producing internal feedback all the time as they monitor, evaluate and regulate their own learning. When they receive external feedback information from a teacher it has to be turned into internal feedback if it is to have any impact on learning. Recent research on peer review shows that students can generate productive internal feedback by themselves without any teacher input. Specifically, as they produce written feedback on the work of peers, they simultaneously reflect on and generate internal feedback on their own work. Strengthening internal feedback develops the students’ capacity to think for themselves and to become independent self-regulating learners. This article therefore makes the case for internal feedback, illustrates its operation in peer review and points to its promise as a guiding concept for future research and for improvements in practice.Gli studenti producono continuamente feedback interni mentre monitorano, valutano e regolano il proprio apprendimento. Anche quando ricevono feedback esterno da un docente, esso deve essere trasformato in feedback interno se si vuole che abbia un impatto sull’apprendimento. La recente ricerca sulla peer review dimostra che gli studenti sono in grado di generare autonomamente un feedback interno produttivo senza alcun contributo da parte dell’insegnante. Nello specifico, producendo feedback scritto sui compiti prodottidai propri pari, essi simultaneamente riflettono e generano un feedback interno sul proprio lavoro. Il rafforzamento del feedback interno sviluppa la capacità degli studenti di pensare in modo autonomo e di autoregolare il proprio apprendimento. Questo articolo, analizza il feedback interno, ne illustra il funzionamento nella peer review e indica la sua dimensione innovativa come concetto guida per la ricerca futura e per i miglioramenti nella pratica

Learning beyond compliance: a comparative analysis of two cohorts undertaking a first year social work module

This paper addresses a current gap in education for sustainable development (ESD), an international educational movement, with a particular focus on teaching and learning innovations. Reflecting upon the mainstream 'business as usual' approaches in the ESD discourse, theories and practices of transformative social work are considered to make a significant contribution to that end. Empirical research was conducted to examine a new pedagogical approach introduced within an established module taught in 9 different groups to first year UK Social Work students during the academic year of 2007/8. The core change investigated was the replacement of detailed weekly instructions for teaching staff. The new guide articulated a pedagogical framework for the course and outlined themes and objectives, leaving detailed planning and delivery to individual teachers. Explorations were made through a comparative analysis of the responses of teaching staff and students for pre- 2007/8 academic years and 2007/8 year respectively. Data were collected using both qualitative and quantitative research methodologies. The research findings include students' positive view towards the classroom-based learning and some indications of deeper and wider understanding of social justice. Staff reported a renewed sense of professionalism. This research illuminates the potential for learning beyond compliance within existing curriculum frameworks