Exploring rater judging behaviour using the many-facet rasch model

Performance assessment, unlike the traditional fixed-response assessment, has features peculiar to its assessment setting (the task choice, the task processing conditions, the raters,the rating scale, and the rating procedures) that make it much more vulnerable to construct irrelevant variance (McNamara, 1997; Upshur & Turner, 1999). Of these potential sources of variability, those associated with raters are considered to be extensive and pose serious threats to the validity of ratings (Linacre, 1989; McNamara, 1996).For performance assessment to yield valid and reliable results, it is essential that these sources of variability are eliminated or minimized. This paper illustrates how sources of rater-related variability or rater effects can be identified and controlled for using the Many-facet Rasch Model.It also illustrates how the idiosyncratic rating behaviour of individual raters can be explicated through the use of this measurement model. In this illustration, the ratings of thirty-five English language instructors on 12 paragraphs written by new intake students at the Centre for Foundation Studies, IIUM were analysed using FACETS (Linacre, 2003), a computer application which implements the Many-facet Rasch Model. The results of the analysis indicate substantial differences in rater severity, and the presence of other rater effects

Cognitive processing of spatial relations in Euclidean diagrams

The cognitive processing of spatial relations in Euclidean diagrams is central to the diagram-based geometric practice of Euclid's Elements. In this study, we investigate this processing through two dichotomies among spatial relations—metric vs topological and exact vs co-exact—introduced by Manders in his seminal epistemological analysis of Euclid's geometric practice. To this end, we carried out a two-part experiment where participants were asked to judge spatial relations in Euclidean diagrams in a visual half field task design. In the first part, we tested whether the processing of metric vs topological relations yielded the same hemispheric specialization as the processing of coordinate vs categorical relations. In the second part, we investigated the specific performance patterns for the processing of five pairs of exact/co-exact relations, where stimuli for the co-exact relations were divided into three categories depending on their distance from the exact case. Regarding the processing of metric vs topological relations, hemispheric differences were found for only a few of the stimuli used, which may indicate that other processing mechanisms might be at play. Regarding the processing of exact vs co-exact relations, results show that the level of agreement among participants in judging co-exact relations decreases with the distance from the exact case, and this for the five pairs of exact/co-exact relations tested. The philosophical implications of these empirical findings for the epistemological analysis of Euclid's diagram-based geometric practice are spelled out and discussed

Responses to Ofsted’s consultation on further education and skills inspections from September 2012 : an evaluation report

"Beginning in September 2011, Ofsted consulted widely on proposals for new inspection arrangements for the further education and skills system. This included a published consultation document and a formal three-month consultation process. This evaluation report summarises the responses we received on our proposals for further education and skills inspections from September 2012" - front cover

Dominance attributions following damage to the ventromedial prefrontal cortex

Damage to the human ventromedial prefrontal cortex (VM) can result in dramatic and maladaptive changes in social behavior despite preservation of most other cognitive abilities. One important aspect of social cognition is the ability to detect social dominance, a process of attributing from particular social signals another person's relative standing in the social world. To test the role of the VM in making attributions of social dominance, we designed two experiments: one requiring dominance judgments from static pictures of faces, the second requiring dominance judgments from film clips. We tested three demographically matched groups of subjects: subjects with focal lesions in the VM (n=15), brain-damaged comparison subjects with lesions excluding the VM (n=11), and a reference group of normal individuals with no history of neurological disease (n=32). Contrary to our expectation, we found that subjects with VM lesions gave dominance judgments on both tasks that did not differ significantly from those given by the other groups. Despite their grossly normal performance, however, subjects with VM lesions showed more subtle impairments specifically when judging static faces: They were less discriminative in their dominance judgments, and did not appear to make normal use of gender and age of the faces in forming their judgments. The findings suggest that, in the laboratory tasks we used, damage to the VM does not necessarily impair judgments of social dominance, although it appears to result in alterations in strategy that might translate into behavioral impairments in real life

The Craft of Incentive Prize Design: Lessons from the Public Sector

In the last five years, incentive prizes have transformed from an exotic open innovation tool to a proven innovation strategy for the public, private and philanthropic sectors. This report offers practical lessons for public sector leaders and their counterparts in the philanthropic and private sectors to help understand what types of outcomes incentive prizes help to achieve, what design elements prize designers use to create these challenges and how to make smart design choices to achieve a particular outcome. It synthesizes insights from expert interviews and analysis of more than 400 prize

Experts' Judgments of Management Journal Quality:An Identity Concerns Model

Many lists that purport to gauge the quality of journals in management and organization studies (MOS) are based on the judgments of experts in the field. This article develops an identity concerns model (ICM) that suggests that such judgments are likely to be shaped by the personal and social identities of evaluators. The model was tested in a study in which 168 editorial board members rated 44 MOS journals. In line with the ICM, respondents rated journal quality more highly to the extent that a given journal reflected their personal concerns (associated with having published more articles in that journal) and the concerns of a relevant ingroup (associated with membership of the journal’s editorial board or a particular disciplinary or geographical background). However, judges’ ratings of journals in which they had published were more favorable when those journals had a low-quality reputation, and their ratings of journals that reflected their geographical and disciplinary affiliations were more favorable when those journals had a high-quality reputation. The findings are thus consistent with the view that identity concerns come to the fore in journal ratings when there is either a need to protect against personal identity threat or a meaningful opportunity to promote social identity