Measurement Integrity in Peer Prediction: A Peer Assessment Case Study

We propose measurement integrity, a property related to ex post reward fairness, as a novel desideratum for peer prediction mechanisms in many natural applications. Like robustness against strategic reporting, the property that has been the primary focus of the peer prediction literature, measurement integrity is an important consideration for understanding the practical performance of peer prediction mechanisms. We perform computational experiments, both with an agent-based model and with real data, to empirically evaluate peer prediction mechanisms according to both of these important properties. Our evaluations simulate the application of peer prediction mechanisms to peer assessment -- a setting in which ex post fairness concerns are particularly salient. We find that peer prediction mechanisms, as proposed in the literature, largely fail to demonstrate significant measurement integrity in our experiments. We also find that theoretical properties concerning robustness against strategic reporting are somewhat noisy predictors of empirical performance. Further, there is an apparent trade-off between our two dimensions of analysis. The best-performing mechanisms in terms of measurement integrity are highly susceptible to strategic reporting. Ultimately, however, we show that supplementing mechanisms with realistic parametric statistical models can, in some cases, improve performance along both dimensions of our analysis and result in mechanisms that strike the best balance between them.Comment: The code for our experiments is hosted in the following GitHub repository: https://github.com/burrelln/Measurement-Integrity-and-Peer-Assessment. Version 2 (uploaded on 9/22/22) introduces experiments with real peer grading data alongside significant changes to the framing of the paper and presentation of the result

Measuring What’s Valued Or Valuing What’s Measured? Knowledge Production and the Research Assessment Exercise

Power is everywhere. But what is it and how does it infuse personal and institutional relationships in higher education? Power, Knowledge and the Academy: The Institutional is Political takes a close-up and critical look at both the elusive and blatant workings and consequences of power in a range of everyday sites in universities. Authors work with multi-layered conceptions of power to disturb the idea of the academy as a haven of detached reason and instead reveal the ways in which power shapes personal and institutional relationships, the production of knowledge and the construction of academic careers. Chapters focus on, among other areas, student-supervisor relationships, personal PhD journeys, power in research teams, networking, the Research Assessment Exercise in the UK, and the power to construct knowledge in literature reviews. This chapter does not address which mechanism of research assessment provides a more truthful account of the value of a set of ‘research outputs’. Instead, it focuses on the power of any such mechanism to reinforce particular values and to inscribe hierarchies regarding knowledge. Regardless of what replaces it, the UK's RAE will have been productive, not just reflective of academic values. Some of the negative consequences of the RAE for UK academic life are considered, focusing on the operation of power through processes of knowledge production

CIVE 440: Reinforced Concrete Design I: Faculty-led Inquiry into Reflective and Scholarly Teaching (FIRST) Course Portfolio

This Faculty-led Inquiry into Reflective and Scholarly Teaching (FIRST) Course Portfolio documents the instructor’s teaching practices and student learning for Reinforced Concrete Design I (CIVE 440) course. The contents of this course portfolio captures the CIVE 440 course that was taught on the Lincoln campus during the semester of Fall 2022. CIVE 440 is a structural engineering design elective offered every year in the Department of Civil and Environmental Engineering on both Lincoln and Omaha campus at University of Nebraska-Lincoln (UNL). The course objective is to help students to be able to use theory and experience to proportion and detail reinforced concrete members such as beams, columns, and one-way slabs. In order to achieve this course objective, teaching methods, course materials, and assessment strategies have been carefully designed and continuously modified throughout multiple semesters. In addition to lectures, peer discussions in class with active learning tools (Plickers; a free paper based online clicker tool), solving and reviewing examples/problems as a group, and in-class reading were utilized as teaching methods. These unique methods are not the conventional way many engineering design courses are typically taught and this course portfolio documents these particular aspects. A mix of formative assessments (recall questions, peer instruction, minute papers, and muddiest point) and summative assessments (fourteen assignments and four exams) were designed and completed for each module in this course to assess the student learning. In addition, for Fall 2022 semester, a modified “specifications grading” of pass/fail grading scheme was used for the assignments in this course which allowed a second attempt if the first trial was failed. This was rather to help students achieve the goal of mastering the learning objectives of each lesson and improve their student learning experiences rather than focusing only on assignment and test scores only. The course assessment data analysis summarizes how these unique teaching methods (peer instruction with Plickers, group activities solving design problems, in-class reading) and different assessment strategies (specifications grading, minute papers, and muddiest points) influenced the student learning. Positive changes are observed in student learning based on their improved formative and summative assessment results compared to the assessments in previous semesters

Expanding the metaphor: a pragmatic application of hospitality theory to the field of writing studies

This dissertation examines the relationship between hospitality theory and Writing Studies. Contemporary Writing Studies scholarship approaches hospitality through a traditional lens viewing it separately as either theory, practice or pedagogy for the composition classroom (Dale Jacobs, Richard Haswell, Janice Haswell, Glen Blalock, Matthew Heard, Joanna Lin Want). As the practice and pedagogy of hospitality are promoted in Writing Studies, the binary metaphor of the host and guest provides the dominant way of discussing this work. In this binary metaphor the instructor is often designated the host, and the students the guests. This configuration obscures the important influence of the university upon the classroom relationship. I argue that recognizing the additional influence of the university on the relationship between the instructor and students is necessary because it impacts the instructor's ability to act as a host. Following pragmatic influences like William James and Ann E. Berthoff, I argue for a disruption of the binary metaphor of hospitality. Returning to hospitality theory I focus on Levinas' identification of a third position in the hospitable metaphor. Using this third position, which for this conversation I call the "Preparer," to apply a triadic metaphor of hospitality to the composition classroom reveals how the institution must create the conditions necessary for the instructor to act as the host. The triadic metaphor of hospitality supplies an analytical perspective to be applied beyond the classroom to the additional work of Writing Studies as well. Viewing the position of the writing program administrator in the context of the triadic metaphor untangles the multiple, often conflicting, positions the administrator occupies. The peripheral position of the writing center at the edges of the university provides a space in which the writing center administrator can create an environment in which a hospitable encounter between the consultant and writer is possible

Experiences of Grade Inflation at an Online University in the United States: An Autoethnography

Grade inflation is a problem at universities in the United States. To understand the cultural effect of grade inflation at a regionally accredited online university in the United States, I conducted autoethnographic research as a participant and observer. In this autoethnographic study, the purpose of my research was to explore my experiences being immersed in a grade inflation culture. I addressed a gap of autoethnographic research related to a culture of grade inflation existing at an online university in the United States. I provided seven themes serving as my discoveries related to my observations and participation as a faculty member. My discoveries supported my assumptions that a culture of grade inflation likely exists at the online university. My discoveries also contribute to the overriding theme in the extant literature that grade inflation exists. My discoveries also support the concept that grade inflation is not limited to on ground universities but also extends to online universities in the United States

Managing group work: the impact of peer assessment on student engagement

This study investigates the impact of peer assessment on students’ engagement in their learning in a group work context. The study used regression analysis and was complemented by qualitative responses from a survey of 165 first-year undergraduates in a UK university. Findings suggest that students' perception of their contribution to group work fosters engagement and enhances their learning in a group. Also, that students’ perception and the overall experience of rating their peers’ work impact their engagement within a group. The study contributes to the literature by focusing on the assessment of the entire learning journey within a group rather than the final group output. In particular, the study highlights the significant contributions of peer assessment in managing student engagement in modules and/or assessments for large cohorts

Final MA Portfolio

This is the portfolio submission for my Master\u27s in English with a specialization in English Teaching

PUBH 4132: Health Promotion Program Planning I

Provides the student with the theory and practical applications of planning, developing, implementing and evaluating health promotion programs in a variety of settings. The focus will be on a global approach to planning with emphasis on the 5 settings for health promotion programming which include school, community, clinical, college, and worksite settings

Examining Graphic Design Students’ Attitudes toward Participative Assessment in Studio-Based Learning

This study examines the potential impact of the development and implementation of innovative assessment practices in graphic design studio to improve teaching and learning and to foster creativity development among communication design students in a higher education. The data were collected from communication design students at the Kwame Nkrumah University of Science and Technology in Ghana. An action research which used qualitative method to capture students’ perceptions of the use of peer assessment in various aspects of studio critique. The conclusion reached is that while the introduction of peer assessment option may be time consuming for staff to develop, the benefits of an enhanced student-centered approach to assessment may be well worth this investment in time. The results should be of interest to those academics who are concerned with assessment of creative product in art and design schools and its impact on students’ achievement. Keywords: Participative assessment, assessment, peer assessment, studio-based learning, qualitative stud