Using pairwise comparisons in the online social moderation of performance assessment

Assessing the performance of a student involves some form of judgement, and where more than one assessor is involved this usually requires some form of moderation to ensure consistent and fair results. Often this involves meetings or communication between assessors, which is referred to as social moderation. This paper reports on a study that investigated the use of online technologies to support a form of social moderation of artworks submitted for assessment in a senior secondary school course in Western Australia. Online systems were used to facilitate communications and provide access to digital representations of the submissions along with assessment tools. In particular a pairwise comparison judging online tool was used. This approach to social moderation was tested in a realistic context involving a sample of 12 teachers from rural schools for whom face-to-face meetings would be difficult. The aim was to investigate whether the use of these online systems would support good moderation outcomes and valuable professional learning for those involved. The study found that this approach to online social moderation was feasible, and participants perceived that it had improved the consistency of their judgements because they had developed an improved understanding of the assessment criteria and standard of work. However, analysis of scores and reliability data suggested some were not adequately consistent, and it was likely that this was due to their inexperience in assessing such work. Therefore some changes to the processes of this form of online social moderation were recommended

Research Conference 2022: Reimagining assessment: Proceedings and program

The focus of this year’s Research Conference is on the use of assessment to support improved teaching and learning. The conference is titled ‘reimagining assessment’ because we believe there is a need to transform the essential purposes of educational assessment to provide better information about the deep conceptual learning, skills, competencies, and personal attributes that teachers and schools now have as objectives for student learning and development. Reimagined assessments must now be focused on monitoring learning across this broader range of intended outcomes and provide quality information about the points individuals have reached in their long-term development

The effect of cost information on buyer-supplier negotiations in different power settings.

buyer-supplier negotiations; cost information;

A comparative judgement approach to teacher assessment

We report one teacher’s response to a top-down shift from external examinations to internal teacher assessment for summative purposes in the Republic of Ireland. The teacher adopted a comparative judgement approach to the assessment of secondary students’ understanding of a chemistry experiment. The aims of the research were to investigate whether comparative judgement can produce assessment outcomes that are valid and reliable without producing undue workload for the teachers involved. Comparative judgement outcomes correlated as expected both with test marks and with existing student achievement data, supporting the validity of the approach. Further analysis suggested that teacher judgement privileged scientific understanding, whereas marking privileged factual recall. The estimated reliability of the outcome was acceptably high, but comparative judgement was notably more time-consuming than marking. We consider how validity and efficiency might be improved and the contributions that comparative judgement might offer to summative assessment, moderation of teacher assessment and peer assessment

The question-behaviour effect: a theoretical and methodological review and meta-analysis

Research has demonstrated that asking people questions about a behaviour can lead to behaviour change. Despite many, varied studies in different domains, it is only recently that this phenomenon has been studied under the umbrella term of the question-behaviour effect (QBE) and moderators of the effect have been investigated. With a particular focus on our own contributions, this article: (1) provides an overview of QBE research; (2) reviews and offers new evidence concerning three theoretical accounts of the QBE (behavioural simulation and processing fluency; attitude accessibility; cognitive dissonance); (3) reports a new meta-analysis of QBE studies (k = 66, reporting 94 tests) focusing on methodological moderators. The findings of this meta-analysis support a small significant effect of the QBE (g = 0.14, 95% CI = 0.11, 0.18, p < .001) with smaller effect sizes observed in more carefully controlled studies that exhibit less risk of bias and (4) also considers directions for future research on the QBE, especially studies that use designs with low risk of bias and consider desirable and undesirable behaviour separately

Towards Conceptualization of "Fair Explanation": Disparate Impacts of anti-Asian Hate Speech Explanations on Content Moderators

Recent research at the intersection of AI explainability and fairness has focused on how explanations can improve human-plus-AI task performance as assessed by fairness measures. We propose to characterize what constitutes an explanation that is itself "fair" -- an explanation that does not adversely impact specific populations. We formulate a novel evaluation method of "fair explanations" using not just accuracy and label time, but also psychological impact of explanations on different user groups across many metrics (mental discomfort, stereotype activation, and perceived workload). We apply this method in the context of content moderation of potential hate speech, and its differential impact on Asian vs. non-Asian proxy moderators, across explanation approaches (saliency map and counterfactual explanation). We find that saliency maps generally perform better and show less evidence of disparate impact (group) and individual unfairness than counterfactual explanations. Content warning: This paper contains examples of hate speech and racially discriminatory language. The authors do not support such content. Please consider your risk of discomfort carefully before continuing reading!Comment: EMNLP 2023 Main Conference (Long Paper

Exploring Predictors of Teamwork Performance in an Interprofessional Education Setting

Abstract EXPLORING PREDICTORS OF TEAMWORK PERFORMANCE IN AN INTERPROFESSIONAL EDUCATION SETTING By Danah M. Alsane, MS. A thesis submitted in partial fulfillment of the requirements for the degree of Master of Pharmaceutical Science at Virginia Commonwealth University Virginia Commonwealth University, 2016 Advisor: Patricia Slattum, Pharm.D., Ph.D. Professor and Director of the Geriatric Pharmacotherapy Program Department of Pharmacotherapy and Outcomes Science Objectives: The primary objective of this study was to explain how individual characteristics influence teamwork development. In addition, it evaluated how teamwork development, in conjunction with content knowledge, impact students’ performance on a team-based project in an Interprofessional Quality Improvement and Patient Safety (IPQIPS) course. Methods: This cross sectional study included medical, pharmacy, and nursing students enrolled in an IPQIPS course offered for the first time at VCU. Predictors of teamwork development examined included collective orientation (measured using the Collective Orientation Scale, which included dominance and affiliation subscales), and prior interprofessional teamwork experience (measured using self-report). The Team Development Measure (TDM) was used to measure teamwork development. The Statistical Process Control Quiz (SPCQ) was used to assess content knowledge acquired during the course. The final project score was used to evaluate students’ performance on a team-based project. Structural equation modeling was used to test study hypotheses. Results: Among the proposed predictors (dominance, affiliation, and interprofessional teamwork experience), only dominance was related to TDM. No significant relationship was found between teamwork development combined with content knowledge and successful accomplishment of team-based project. Conclusion: This study was the first to our knowledge to simultaneously assess the impact of individual characteristics on teamwork development, and how teamwork development (combined with individual student knowledge) influences students’ performance on team-based project in an interprofessional education setting. Although findings were not conclusive, several potential avenues for future study are highlighted

Information Delivery, User Decision Approach, and Choice Environment: Examining the Effectiveness of Non-Compensatory and Customization-based Online Decision Support.

Decision support research has largely focused on the mechanics of tool design, with less attention paid to the way the alternatives are presented to the user - that is, the format of the output, how the decision tool design can play a role in it, and the output content (characteristics). Furthermore, little research has examined specific decision contexts and user’s cognitive aspects pertinent to the choice task, and their role during an online purchase. This study addresses these issues by investigating the impact of output format and content of a non-compensatory (NC) tool and a customization-based tool on user’s decision quality in the context of a health insurance purchase. It also examines the moderating role of context (perceived risk) and user’s decision approach (price heuristics) – both salient in a health plan choice. Drawing from risk perception, decoy effect, price order effect, and options framing, this research carries out 2 studies: 2x3x2 full factorial between subjects experiments. Study 1 examines the effect of NC Descending (price High-Low) choice sets with asymmetrically dominated alternatives, while Study 2 examines NC Descending, NC Ascending, and customization-based tools. Both studies also investigate the roles of perceived risk (high vs low), and user’s decision approach (price heuristics-driven strong vs weak). Results of Study 1 demonstrate that output content characterized by price anchoring differentially affects user’s decision quality. These dynamics change for users under different levels of perceived risk and with disparate decision approaches. Study 2 indicate that by subjecting the user to reference dependence, usage of NC Descending tool can have a negative impact on decision quality (highest price paid), and usage of NC Ascending and Financial tool have a positive impact (lower price paid). Usage of a customization-based tool, as per the design delineated here, mitigates the negative impact of NC Descending, and further lowers, the influence of NC Ascending tools, by enforcing cost-utility analysis, adopting base-level reference point, and enabling more flexible item composition. The study contributes to: a) information systems, by uncovering detailed dynamics of the interactions between information delivery and the user; and b) boundaries of reference dependence, thus, loss aversion

The effects of time of day and circadian rhythm on performance during variable levels of cognitive workload

The present study examined the effects of time of day of testing on a simulated aviation task. The tasks required the participants to engage in multitasking while electroencephalogram (EEG) data was collected to objectively measure participants’ workload. Task demands were altered throughout the testing period to expose participants to both high and low workload conditions. Additionally, individual differences in circadian rhythm were explored by assessing participants’ circadian typology. No significant differences in performance were found resulting from time of day differences. However, performance and EEG differences were found based on phase of testing and workload manipulations. Subjective workload measures were influenced by time of day, with a moderating effect of circadian typology. Implications are discussed