The anonymous reviewer:the relationship between perceived expertise and the perceptions of peer feedback in higher education

Peer feedback often has positive effects on student learning processes and outcomes. However, students may not always be honest when giving and receiving peer feedback as they are likely to be biased due to peer relations, peer characteristics and personal preferences. To alleviate these biases, anonymous peer feedback was investigated in the current research. Research suggests that the expertise of the reviewer influences the perceived usefulness of the feedback. Therefore, this research investigated the relationship between expertise and the perceptions of peer feedback in a writing assignment of 41 students in higher education with a multilevel analysis. The results show that students perceive peer feedback as more adequate when knowing the reviewer perceives him/herself to have a high level of expertise. Furthermore, the results suggest that students who received feedback from a peer who perceives their expertise as closer to the reviewee’s own perceived expertise was more willing to improve his or her own assignment

Weighted coverage based reviewer assignment

Peer reviewing is a standard process for assessing the quality of submissions at academic conferences and journals. A very important task in this process is the assignment of reviewers to papers. However, achieving an appropriate assignment is not easy, because all reviewers should have similar load and the subjects of the assigned papers should be consistent with the reviewers' expertise. In this paper, we propose a generalized framework for fair reviewer assignment. We first extract the domain knowledge from the reviewers' published papers and model this knowledge as a set of topics. Then, we perform a group assignment of reviewers to papers, which is a generalization of the classic Reviewer Assignment Problem (RAP), considering the relevance of the papers to topics as weights. We study a special case of the problem, where reviewers are to be found for just one paper (Journal Assignment Problem) and propose an exact algorithm which is fast in practice, as opposed to brute-force solutions. For the general case of having to assign multiple papers, which is too hard to be solved exactly, we propose a greedy algorithm that achieves a 1/2-approximation ratio compared to the exact solution. This is a great improvement compared to the 1/3-approximation solution proposed in previous work for the simpler coverage-based reviewer assignment problem, where there are no weights on topics. We theoretically prove the approximation bound of our solution and experimentally show that it is superior to the current state-of-the-art.postprin

An Earth Science Scrapbook Project as an Alternative Assessment Tool

"Scrapbooking" is a popular hobby and as such, has found its way into educational settings, primarily in middle and elementary school levels. This article describes a scrapbook project that is used both as a means of demonstrating the connections between geology and students' daily lives and as an alternative form of assessment. The project was developed for an introductory Earth Science class for middle school and high school pre-service teachers. Educational levels: Graduate or professional

Multivariate Fairness for Paper Selection

Peer review is the process by which publishers select the best publications for inclusion in a journal or a conference. Bias in the peer review process can impact which papers are selected for inclusion in conferences and journals. Although often implicit, race, gender and other demographics can prevent members of underrepresented groups from presenting at major conferences. To try to avoid bias, many conferences use a double-blind review process to increase fairness during reviewing. However, recent studies argue that the bias has not been removed completely. Our research focuses on developing fair algorithms that correct for these biases and select papers from a more demographically diverse group of authors. To address this, we present fair algorithms that explicitly incorporate author diversity in paper recommendation using multidimensional author profiles that include five demographic features, i.e., gender, ethnicity, career stage, university rank, and geolocation. The Overall Diversity method ranks papers based on an overall diversity score whereas the Multifaceted Diversity method selects papers that fill the highest-priority demographic feature first. We evaluate these algorithms with Boolean and continuous-valued features by recommending papers for SIGCHI 2017 from a pool of SIGCHI 2017, DIS 2017 and IUI 2017 papers and compare the resulting set of papers with the papers accepted by the conference. Both methods increase diversity with small decreases in utility using profiles with either Boolean or continuous feature values. Our best method, Multifaceted Diversity, recommends a set of papers that match demographic parity, selecting authors who are 42.50% more diverse with a 2.45% gain in utility. This approach could be applied when selecting conference papers, journal papers, grant proposals, or other tasks within academia

Multivariate Fairness for Paper Selection

Peer review is the process by which publishers select the best publications for inclusion in a journal or a conference. Bias in the peer review process can impact which papers are selected for inclusion in conferences and journals. Although often implicit, race, gender and other demographics can prevent members of underrepresented groups from presenting at major conferences. To try to avoid bias, many conferences use a double-blind review process to increase fairness during reviewing. However, recent studies argue that the bias has not been removed completely. Our research focuses on developing fair algorithms that correct for these biases and select papers from a more demographically diverse group of authors. To address this, we present fair algorithms that explicitly incorporate author diversity in paper recommendation using multidimensional author profiles that include five demographic features, i.e., gender, ethnicity, career stage, university rank, and geolocation. The Overall Diversity method ranks papers based on an overall diversity score whereas the Multifaceted Diversity method selects papers that fill the highest-priority demographic feature first. We evaluate these algorithms with Boolean and continuous-valued features by recommending papers for SIGCHI 2017 from a pool of SIGCHI 2017, DIS 2017 and IUI 2017 papers and compare the resulting set of papers with the papers accepted by the conference. Both methods increase diversity with small decreases in utility using profiles with either Boolean or continuous feature values. Our best method, Multifaceted Diversity, recommends a set of papers that match demographic parity, selecting authors who are 42.50% more diverse with a 2.45% gain in utility. This approach could be applied when selecting conference papers, journal papers, grant proposals, or other tasks within academia

Building and Executing MOOCs: A Practical Review of Glasgow's First Two MOOCs (Massive Open Online Courses)

No abstract available

Knowledge aggregation in people recommender systems : matching skills to tasks

People recommender systems (PRS) are a special type of RS. They are often adopted to identify people capable of performing a task. Recommending people poses several challenges not exhibited in traditional RS. Elements such as availability, overload, unresponsiveness, and bad recommendations can have adverse effects. This thesis explores how people’s preferences can be elicited for single-event matchmaking under uncertainty and how to align them with appropriate tasks. Different methodologies are introduced to profile people, each based on the nature of the information from which it was obtained. These methodologies are developed into three use cases to illustrate the challenges of PRS and the steps taken to address them. Each one emphasizes the priorities of the matching process and the constraints under which these recommendations are made. First, multi-criteria profiles are derived completely from heterogeneous sources in an implicit manner characterizing users from multiple perspectives and multi-dimensional points-of-view without influence from the user. The profiles are introduced to the conference reviewer assignment problem. Attention is given to distribute people across items in order reduce potential overloading of a person, and neglect or rejection of a task. Second, people’s areas of interest are inferred from their resumes and expressed in terms of their uncertainty avoiding explicit elicitation from an individual or outsider. The profile is applied to a personnel selection problem where emphasis is placed on the preferences of the candidate leading to an asymmetric matching process. Third, profiles are created by integrating implicit information and explicitly stated attributes. A model is developed to classify citizens according to their lifestyles which maintains the original information in the data set throughout the cluster formation. These use cases serve as pilot tests for generalization to real-life implementations. Areas for future application are discussed from new perspectives.Els sistemes de recomanació de persones (PRS) són un tipus especial de sistemes recomanadors (RS). Sovint s’utilitzen per identificar persones per a realitzar una tasca. La recomanació de persones comporta diversos reptes no exposats en la RS tradicional. Elements com la disponibilitat, la sobrecàrrega, la falta de resposta i les recomanacions incorrectes poden tenir efectes adversos. En aquesta tesi s'explora com es poden obtenir les preferències dels usuaris per a la definició d'assignacions sota incertesa i com aquestes assignacions es poden alinear amb tasques definides. S'introdueixen diferents metodologies per definir el perfil d’usuaris, cadascun en funció de la naturalesa de la informació necessària. Aquestes metodologies es desenvolupen i s’apliquen en tres casos d’ús per il·lustrar els reptes dels PRS i els passos realitzats per abordar-los. Cadascun destaca les prioritats del procés, l’encaix de les recomanacions i les seves limitacions. En el primer cas, els perfils es deriven de variables heterogènies de manera implícita per tal de caracteritzar als usuaris des de múltiples perspectives i punts de vista multidimensionals sense la influència explícita de l’usuari. Això s’aplica al problema d'assignació d’avaluadors per a articles de conferències. Es presta especial atenció al fet de distribuir els avaluadors entre articles per tal de reduir la sobrecàrrega potencial d'una persona i el neguit o el rebuig a la tasca. En el segon cas, les àrees d’interès per a caracteritzar les persones es dedueixen dels seus currículums i s’expressen en termes d’incertesa evitant que els interessos es demanin explícitament a les persones. El sistema s'aplica a un problema de selecció de personal on es posa èmfasi en les preferències del candidat que condueixen a un procés d’encaix asimètric. En el tercer cas, els perfils dels usuaris es defineixen integrant informació implícita i atributs indicats explícitament. Es desenvolupa un model per classificar els ciutadans segons els seus estils de vida que manté la informació original del conjunt de dades del clúster al que ell pertany. Finalment, s’analitzen aquests casos com a proves pilot per generalitzar implementacions en futurs casos reals. Es discuteixen les àrees d'aplicació futures i noves perspectives.Postprint (published version