Methods for Ordinal Peer Grading

MOOCs have the potential to revolutionize higher education with their wide outreach and accessibility, but they require instructors to come up with scalable alternates to traditional student evaluation. Peer grading -- having students assess each other -- is a promising approach to tackling the problem of evaluation at scale, since the number of "graders" naturally scales with the number of students. However, students are not trained in grading, which means that one cannot expect the same level of grading skills as in traditional settings. Drawing on broad evidence that ordinal feedback is easier to provide and more reliable than cardinal feedback, it is therefore desirable to allow peer graders to make ordinal statements (e.g. "project X is better than project Y") and not require them to make cardinal statements (e.g. "project X is a B-"). Thus, in this paper we study the problem of automatically inferring student grades from ordinal peer feedback, as opposed to existing methods that require cardinal peer feedback. We formulate the ordinal peer grading problem as a type of rank aggregation problem, and explore several probabilistic models under which to estimate student grades and grader reliability. We study the applicability of these methods using peer grading data collected from a real class -- with instructor and TA grades as a baseline -- and demonstrate the efficacy of ordinal feedback techniques in comparison to existing cardinal peer grading methods. Finally, we compare these peer-grading techniques to traditional evaluation techniques.Comment: Submitted to KDD 201

Building functional neuromarkers from resting state fMRI to describe physiopathological traits

2016 - 2017The overarching goal of this work has been that of devising novel methods for building functional neuromarkers from resting-state fMRI data to describe healthy and pathological human behaviour. Observing spontaneous uctuations of the BOLD signal, resting-state fMRI allows to have an insight into the functional organisation of the brain and to detect functional networks that are consistent across subjects. Studying how patterns of functional connectivity vary both in healthy subjects and in subjects a ected by a neurodegenerative disease is a way to shed light on the physiological and pathological mechanisms governing our nervous system. The rst part of this thesis is devoted to the description of fully data-driven feature extraction techniques based on clustering aimed at supporting the diagnosis of neurodegenerative diseases (e.g., amyotrophic lateral sclerosis and Parkinson's disease). The high-dimensional nature of resting state fMRI data implies the need of suitable feature selection techniques. Traditional univariate techniques are fast and straightforward to interpret, but are unable to unveil relationships among multiple features. For this reason, this work presents a methodology based on consensus clustering, a particular approach to the clustering problem that consists in combining di erent partitions of the same data set to produce more stable solutions. One of the objectives of fMRI data analysis is to determine regions that show an abnormal activity with respect to a healthy brain and this is often attained with comparative statistical models applied to single voxels or brain parcels within one or several functional networks. Here, stochastic rank aggregation is applied to identify brain regions that exhibit a coherent behaviour in groups of subjects a ected by the same disorder. The proposed methodology was validated on real data and the results are consistent with previous literature, thus indicating that this approach might be suitable to support early diagnosis of neurodegenerative diseases... [edited by Author]XXX cicl

Ordinal ranking aggregation in bibliometric analysis

This paper reviews standard ranking aggregation approaches in bibliometric analysis. These include the arithmetic and the harmonic mean. We also present two less-well known aggregation schemes, lexicographic and graphicolexic, which are based on the order of the rankings. Finally, we introduce two recently proposed ranking aggregation approaches which are based on stochastic aggregation. We describe all approaches and give a small illustrative and an empirical example to highlight the differences

Ordinal ranking aggregation in bibliometric analysis

This paper reviews standard ranking aggregation approaches in bibliometric analysis. These include the arithmetic and the harmonic mean. We also present two less-well known aggregation schemes, lexicographic and graphicolexic, which are based on the order of the rankings. Finally, we introduce two recently proposed ranking aggregation approaches which are based on stochastic aggregation. We describe all approaches and give a small illustrative and an empirical example to highlight the differences

Veto values in Group Decision Making within MAUT: aggregating complete rankings derived from dominance intensity measures

We consider a groupdecision-making problem within multi-attribute utility theory, in which the relative importance of decisionmakers (DMs) is known and their preferences are represented by means of an additive function. We allow DMs to provide veto values for the attribute under consideration and build veto and adjust functions that are incorporated into the additive model. Veto functions check whether alternative performances are within the respective veto intervals, making the overall utility of the alternative equal to 0, where as adjust functions reduce the utilty of the alternative performance to match the preferences of other DMs. Dominance measuring methods are used to account for imprecise information in the decision-making scenario and to derive a ranking of alternatives for each DM. Specifically, ordinal information about the relative importance of criteria is provided by each DM. Finally, an extension of Kemeny's method is used to aggregate the alternative rankings from the DMs accounting for the irrelative importance

A systematic review on multi-criteria group decision-making methods based on weights: analysis and classification scheme

Interest in group decision-making (GDM) has been increasing prominently over the last decade. Access to global databases, sophisticated sensors which can obtain multiple inputs or complex problems requiring opinions from several experts have driven interest in data aggregation. Consequently, the field has been widely studied from several viewpoints and multiple approaches have been proposed. Nevertheless, there is a lack of general framework. Moreover, this problem is exacerbated in the case of experts’ weighting methods, one of the most widely-used techniques to deal with multiple source aggregation. This lack of general classification scheme, or a guide to assist expert knowledge, leads to ambiguity or misreading for readers, who may be overwhelmed by the large amount of unclassified information currently available. To invert this situation, a general GDM framework is presented which divides and classifies all data aggregation techniques, focusing on and expanding the classification of experts’ weighting methods in terms of analysis type by carrying out an in-depth literature review. Results are not only classified but analysed and discussed regarding multiple characteristics, such as MCDMs in which they are applied, type of data used, ideal solutions considered or when they are applied. Furthermore, general requirements supplement this analysis such as initial influence, or component division considerations. As a result, this paper provides not only a general classification scheme and a detailed analysis of experts’ weighting methods but also a road map for researchers working on GDM topics or a guide for experts who use these methods. Furthermore, six significant contributions for future research pathways are provided in the conclusions.The first author acknowledges support from the Spanish Ministry of Universities [grant number FPU18/01471]. The second and third author wish to recognize their support from the Serra Hunter program. Finally, this work was supported by the Catalan agency AGAUR through its research group support program (2017SGR00227). This research is part of the R&D project IAQ4EDU, reference no. PID2020-117366RB-I00, funded by MCIN/AEI/10.13039/ 501100011033.Peer ReviewedPostprint (published version

Stochastic Rank Aggregation for the Identification of Functional Neuromarkers

The main challenge in analysing functional magnetic resonance imaging (fMRI) data from extended samples of subject (N > 100) is to extract as much relevant information as possible from big amounts of noisy data. When studying neurodegenerative diseases with resting-state fMRI, one of the objectives is to determine regions with abnormal background activity with respect to a healthy brain and this is often attained with comparative statistical models applied to single voxels or brain parcels within one or several functional networks. In this work, we propose a novel approach based on clustering and stochastic rank aggregation to identify parcels that exhibit a coherent behaviour in groups of subjects affected by the same disorder and apply it to default-mode network independent component maps from resting-state fMRI data sets. Brain voxels are partitioned into parcels through k-means clustering, then solutions are enhanced by means of consensus techniques. For each subject, clusters are ranked according to their median value and a stochastic rank aggregation method, TopKLists, is applied to combine the individual rankings within each class of subjects. For comparison, the same approach was tested on an anatomical parcellation. We found parcels for which the rankings were different among control subjects and subjects affected by Parkinson's disease and amyotrophic lateral sclerosis and found evidence in literature for the relevance of top ranked regions in default-mode brain activity. The proposed framework represents a valid method for the identification of functional neuromarkers from resting-state fMRI data, and it might therefore constitute a step forward in the development of fully automated data-driven techniques to support early diagnoses of neurodegenerative diseases