The PeerRank Method for Peer Assessment

We propose the PeerRank method for peer assessment. This constructs a grade for an agent based on the grades proposed by the agents evaluating the agent. Since the grade of an agent is a measure of their ability to grade correctly, the PeerRank method weights grades by the grades of the grading agent. The PeerRank method also provides an incentive for agents to grade correctly. As the grades of an agent depend on the grades of the grading agents, and as these grades themselves depend on the grades of other agents, we define the PeerRank method by a fixed point equation similar to the PageRank method for ranking web-pages. We identify some formal properties of the PeerRank method (for example, it satisfies axioms of unanimity, no dummy, no discrimination and symmetry), discuss some examples, compare with related work and evaluate the performance on some synthetic data. Our results show considerable promise, reducing the error in grade predictions by a factor of 2 or more in many cases over the natural baseline of averaging peer grades.Comment: To appear in Proc. of ECAI 201

Preference Learning

This report documents the program and the outcomes of Dagstuhl Seminar 14101 “Preference Learning”. Preferences have recently received considerable attention in disciplines such as machine learning, knowledge discovery, information retrieval, statistics, social choice theory, multiple criteria decision making, decision under risk and uncertainty, operations research, and others. The motivation for this seminar was to showcase recent progress in these different areas with the goal of working towards a common basis of understanding, which should help to facilitate future synergies

Trust-Based Community Assessment

In this paper we present Community Assessment (COMAS), a trust-based assessment service that helps compute group opinion from the perspective of a specific community member. We apply COMAS in the context of communities of learners, and we compute the group opinion from the perspective of the teacher. Specifically, our model relies on \emph{teacher assessments}, aggregations of \emph{student assessments} and \emph{trust measures} derived from student assessments to suggest marks to assignments that have not been assessed by the teacher. The proposed model intends to support intelligent online learning applications by 1) encouraging students to assess one another, and 2) benefiting from students' assessments. We believe the task of assessing massive numbers of students is of special interest to online learning communities, such as Massive Open Online Courses (MOOCs). Experimental results were conducted on a real classroom datasets as well as simulated data that considers different social network topologies (where we say students assess some assignments of socially connected students). Results show that our method 1) is sound, i.e. the error of the suggested assessments decreases for increasing numbers of teacher assessments; and 2) scales for large numbers of students. 2015 Elsevier Ltd. All rights reservedThis work is supported by the CollectiveMind project (Spanish Ministry of Economy and Competitiveness, under grant number TEC2013-49430-EXP), the MILESS project (Spanish Ministry of Economy and Competitiveness, under grant num- ber TIN2013-45039-P) and the PRAISE project (funded by the European Commission, under grant number 388770)Peer reviewe

Aggregating partial rankings with applications to peer grading in massive online open courses

We investigate the potential of using ordinal peer grading for the evaluation of students in massive online open courses (MOOCs). According to such grading schemes, each student receives a few assignments (by other students) which she has to rank. Then, a global ranking (possibly translated into numerical scores) is produced by combining the individual ones. This is a novel application area for social choice concepts and methods where the important problem to be solved is as follows: how should the assignments be distributed so that the collected individual rankings can be easily merged into a global one that is as close as possible to the ranking that represents the relative performance of the students in the assignment? Our main theoretical result suggests that using very simple ways to distribute the assignments so that each student has to rank only k of them, a Borda-like aggregation method can recover a 1 - O(1/k) fraction of the true ranking when each student correctly ranks the assignments she receives. Experimental results strengthen our analysis further and also demonstrate that the same method is extremely robust even when students have imperfect capabilities as graders. Our results provide strong evidence that ordinal peer grading cam be a highly effective and scalable solution for evaluation in MOOCs

Analysis of the PeerRank Method for Peer Grading

Peer grading can have many benefits in education, including a reduction in the time instructors spend grading and an opportunity for students to learn through their analysis of others work. However, when not handled properly, peer grading can be unreliable and may result in grades that are vastly different from those which a student truly deserves. Therefore, any peer grading system used in a classroom must consider the potential for graders to generate inaccurate grades. One such system is the PeerRank rule proposed by Toby Walsh, which uses an iterative, linear algebra based process reminiscent of the Google PageRank algorithm in order to produce grades by weighting the peer grades with the graders accuracies. However, this system has certain properties which make it less than ideal for peer grading in the classroom. We propose a modification of PeerRank that attempts to make the system more applicable in a classroom environment by incorporating the concept of “ground truth” to provide a basis for accuracy. We then perform a series of experiments to compare the accuracy of our method to that of PeerRank. We conclude that, in cases where a grader’s accuracy in grading others is a reflection of their own grade, our method produces grades with a similar accuracy to PeerRank. However, in cases where a grader’s accuracy and grade are unrelated, our method performs more accurately than PeerRank

Economic behavior of information acquisition: Impact of peer grading in MOOCs

A critical issue in operating massive open online courses (MOOCs) is the scalability of providing feedback. Because it is not feasible for instructors to grade a large number of students’ assignments, MOOCs use peer grading systems. This study investigates the efficacy of that practice when student graders are rational economic agents. We characterize grading as a process of (a) acquiring information to assess an assignment’s quality and (b) reporting a score. This process entails a tradeoff between the cost of acquiring information and the benefits of accurate grading. Because the true quality is not observable, any measure of inaccuracy must reference the actions of other graders, which motivates student graders to behave strategically. We present the unique equilibrium information level and reporting strategy of a homogeneous group of student graders and then examine the outcome of peer grading. We show how both the peer grading structure and the nature of MOOC courses affect peer grading accuracy, and we identify conditions under which the process fails. There is a systematic grading bias toward the mean, which discourages students from learning. To improve current practice, we introduce a scale-shift grading scheme, theoretically examine how it can improve grading accuracy and adjust grading bias and discuss how it can be practically implemented

PeerNomination : a novel peer selection algorithm to handle strategic and noisy assessments

In peer selection a group of agents must choose a subset of themselves, as winners for, e.g., peer-reviewed grants or prizes. We take a Condorcet view of this aggregation problem, assuming that there is an objective ground-truth ordering over the agents. We study agents that have a noisy perception of this ground truth and give assessments that, even when truthful, can be inaccurate. Our goal is to select the best set of agents according to the underlying ground truth by looking at the potentially unreliable assessments of the peers. Besides being potentially unreliable, we also allow agents to be self-interested, attempting to influence the outcome of the decision in their favour. Hence, we are focused on tackling the problem of impartial (or strategyproof) peer selection -- how do we prevent agents from manipulating their reviews while still selecting the most deserving individuals, all in the presence of noisy evaluations? We propose a novel impartial peer selection algorithm, PeerNomination, that aims to fulfil the above desiderata. We provide a comprehensive theoretical analysis of the recall of PeerNomination and prove various properties, including impartiality and monotonicity. We also provide empirical results based on computer simulations to show its effectiveness compared to the state-of-the-art impartial peer selection algorithms. We then investigate the robustness of PeerNomination to various levels of noise in the reviews. In order to maintain good performance under such conditions, we extend PeerNomination by using weights for reviewers which, informally, capture some notion of reliability of the reviewer. We show, theoretically, that the new algorithm preserves strategyproofness and, empirically, that the weights help identify the noisy reviewers and hence to increase selection performance

Graph Models for Rational Social Interaction

A fundamental issue in multi-agent systems is to extract a consensus from a group of agents with different perspectives. Even if the bilateral relationships (reflecting the outcomes of disputes, product comparisons, or evaluation of political candidates) are rational, the collective output may be irrational (e.g., intransitivity of group preferences). This motivates AI’s research for devising social outcomes compatible with individual positions. Frequently, such situations are modeled as graphs. While the preponderance of formal theoretical studies of such graph based-models has addressed semantic concerns for defining a desirable output in order to formalize some high-level intuition, results relating to algorithmic and computational complexity are also of great significance from the computational point of view. The first Part of this thesis is devoted to combinatorial aspects of Argumentation Frameworks related to computational issues. These abstract frameworks, introduced by Dung in 1995, are directed graphs with nodes interpreted as arguments and the directed edges as attacks between the arguments. By designing a conflict-resolution formalism to make distinction among acceptable and unacceptable arguments, Dung initiated an important area of research in Artificial Intelligence. I prove that any argumentation framework can be syntactically augmented into a normal form preserving the semantic properties of the original arguments, by using a cubic time rewriting technique. I introduce polyhedral labellings for an argumentation frameworks, which is a polytope with the property that its integral points are exactly the incidence vectors of specific types of Dung’s outcome. Also, a new notion of acceptability of arguments is considered – deliberative acceptability – and I provide it’s time computational complexity analysis. This part extends and improves some of the results from the my Master thesis. In the second Part, I introduce a novel graph-based model for aggregating preferences. By using graph operations to describe properties of the aggregators, axiomatic characterizations of aggregators corresponding to usual majority or approval & disapproval rule are given. Integrating Dung’s semantics into our model provides a novel qualitative approach to classical social choice: argumentative aggregation of individual preferences. Also, a functional framework abstracting many-to-many two-sided markets is considered: the study of the existence of a Stable Choice Matching in a Bipartite Choice System is reduced to the study of the existence of Stable Common Fixed Points of two choice functions. A generalization of the Gale-Shapley algorithm is designed and, in order to prove its correctness, a new characterization of path independence choice functions is given. Finally, in the third Part, we extend Dung’s Argumentation Frameworks to Opposition Frameworks, reducing the gap between Structured and Abstract Argumentation. A guarded attack calculus is developed, giving proper generalizations of Dung’s extensions.Ein grundlegendes Problem von Multiagentensystemen ist, eine Gruppe von Agenten mit unterschiedlichen Perspektiven zum Konsens zu bringen.W¨ahrend die bilaterale Ergebnisse von Rechtsstreitigkeiten, Produktvergleichen sowie die Bewertung von politischen Kandidaten wiederspiegelnden Beziehungen rational sein sollten, k¨onnte der kollektive Ausgang irrational sein z.B. durch die Intransitivit¨at von Pr¨aferenzen der Gruppe. Das motiviert die KI-Forschung zur Entwicklung von sozialen Ergebnissen, welche mit individuellen Einstellungen kompatibel sind. H¨aufig werden solche Situationen als Graphen modelliert. W¨ahrend die meisten formalen theoretischen Studien von Graphmodellen sich mit semantischen Aspekten f¨ur die Definition eines w¨unschenswerten Ausgangs zur Formalisierung auf hohem Intuitionsniveau besch¨aftigen, ist es ebenfalls von großer Bedeutung, die Komplexit¨at von Algorithmen und Berechnungen zu verstehen. Der erste Teil der vorliegenden Arbeit widmet sich den kombinatorischen Aspekten von Argumentation Frameworks im Zusammenhang mit rechnerischen Fragen. Diese von Dung in 1995 eingef ¨uhrten abstrakten Frameworks sind gerichtete Graphen mit als Argumenten zu interpretierenden Knoten, wobei die gerichteten Kanten Angriffe zwischen den Argumenten sind. Somit hat Dung mit seiner Gestaltung eines Konfliktl¨osungsformalismus zur Unterscheidung zwischen akzeptablen und inakzeptablen Argumenten f¨ur einen wichtigen Bereich von Forschung in KI den Grundstein gelegt. Die Verfasserin hat bewiesen, dass jedes Argumentation Framework sich in einer die semantischen Eigenschaften der originalen Argumente bewahrenden normalen Form syntaktisch erweitern l¨asst, indem man eine mit kubischer Laufzeit umwandelnde Technik verwendet. Neu eingef¨urt werden hier Polyhedrische Etiketten f¨ur Argumentation Frameworks. Dabei handelt es sich um einen Polytop, wessen ganze Punkte genau die Inzidenzvektoren von bestimmten Arten von Dungs Ausgabe sind. Weiterhin wird ein neuer Begriff der Akzeptanz von Argumenten gepr¨agt, n¨amlich - deliberative Akzeptanz - und dessen Komplexit¨at analysiert. Dieser Teil erweitert und verfeinert einige ihrer Ergebnisse aus der Masterarbeit. Im zweiten Teil wurde ein neuartiges graphenbasiertes Modell f¨ur die Aggregation von Pr¨aferenzen erarbeitet. Hier werden axiomatische Charakterisierungen von Aggregatoren neu eingef¨uhrt, und zwar durch die Verwendung von Graphoperationen zur Beschreibung der Eigenschaften von Aggregatoren. Sie entsprechen dem ¨ublichen Mehrheitsprinzip bzw. der Genehmigungs- & Ablehnungsregel. Einen neuartigen, qualitativen Ansatz im Vergleich zu der klassischen Sozialwahltheorie bietet die Integration der Semantik von Dung in dem neuen Modell, und zwar argumentative Aggregation individueller Pr¨aferenzen. Desweiteren wird ein funktionales many to many zweiseitige M¨arkte abstrahierendes Framework untersucht, indem statt die Existenz einer Stabilen Wahl Matching in einem Bipartite Wahlsystem zu studieren, wird die Existenz von Stable Common Fixed Points auf zwei Wahlfunktionen erforscht. Im n¨achsten Schritt wird eine neue Verallgemeinerung des Gale-Shapley Algorithmus entworfen und eine neue Charakterisierung der Wegunabh¨angigkeitsfunktion gegeben, die einen Korrektheitsbeweis f¨ur den Algorithmus erm¨oglicht. Im dritten Teil werden schließlich Dungs Argumentation Frameworks auf Opposition Frameworks erweitert und dadurch die in der gegenw¨artigen Forschung bestehende L¨ucke zwischen strukturierter und abstrakter Argumentation verringert. Daf¨ur wird ein bewachter Angriffskalk¨ul entwickelt, welches strikten Verallgemeinerungen von Dungs echten Erweiterungen f¨uhrt