Multi-Rater Consensus Learning for Modeling Multiple Sparse Ratings of Affective Behaviour

The use of multiple raters to label datasets is an established practice in affective computing. The principal goal is to reduce unwanted subjective bias in the labelling process. Unfortunately, this leads to the key problem of identifying a ground truth for training the affect recognition system. This problem becomes more relevant in a sparsely-crossed annotation where each rater only labels a portion of the full dataset to ensure a manageable workload per rater. In this paper, we introduce a Multi-Rater Consensus Learning (MRCL) method which learns a representative affect recognition model that accounts for each rater's agreement with the other raters. MRCL combines a multitask learning (MTL) regularizer and a consensus loss. Unlike standard MTL, this approach allows the model to learn to predict each rater's label while explicitly accounting for the consensus among raters. We evaluated our approach on two different datasets based on spontaneous affective body movement expressions for pain behaviour detection and laughter type recognition respectively. The two naturalistic datasets were chosen for the different forms of labelling (different in affect, observation stimuli, and raters) that they together offer for evaluating our approach. Empirical results demonstrate that MRCL is effective for modelling affect from datasets with sparsely-crossed multi-rater annotation

Multi-Rater Consensus Learning for Modeling Multiple Sparse Ratings of Affective Behaviour

The use of multiple raters to label datasets is an established practice in affective computing. The principal goal is to reduce unwanted subjective bias in the labelling process. Unfortunately, this leads to the key problem of identifying a ground truth for training the affect recognition system. This problem becomes more relevant in a sparsely-crossed annotation where each rater only labels a portion of the full dataset to ensure a manageable workload per rater. In this paper, we introduce a Multi-Rater Consensus Learning (MRCL) method which learns a representative affect recognition model that accounts for each rater’s agreement with the other raters. MRCL combines a multitask learning (MTL) regularizer and a consensus loss. Unlike standard MTL, this approach allows the model to learn to predict each rater’s label while explicitly accounting for the consensus among raters. We evaluated our approach on two different datasets based on spontaneous affective body movement expressions for pain behaviour detection and laughter type recognition respectively. The two naturalistic datasets were chosen for the different forms of labelling (different in affect, observation stimuli, and raters) that they together offer for evaluating our approach. Empirical results demonstrate that MRCL is effective for modelling affect from datasets with sparsely-crossed multi-rater annotation

Analyzing evolution of rare events through social media data

Recently, some researchers have attempted to find a relationship between the evolution of rare events and temporal-spatial patterns of social media activities. Their studies verify that the relationship exists in both time and spatial domains. However, few of those studies can accurately deduce a time point when social media activities are most highly affected by a rare event because producing an accurate temporal pattern of social media during the evolution of a rare event is very difficult. This work expands the current studies along three directions. Firstly, we focus on the intensity of information volume and propose an innovative clustering algorithm-based data processing method to characterize the evolution of a rare event by analyzing social media data. Secondly, novel feature extraction and fuzzy logic-based classification methods are proposed to distinguish and classify event-related and unrelated messages. Lastly, since many messages do not have ground truth, we execute four existing ground-truth inference algorithms to deduce the ground truth and compare their performances. Then, an Adaptive Majority Voting (Adaptive MV) method is proposed and compared with two of the existing algorithms based on a set containing manually-labeled social media data. Our case studies focus on Hurricane Sandy in 2012 and Hurricane Maria in 2017. Twitter data collected around them are used to verify the effectiveness of the proposed methods. Firstly, the results of the proposed data processing method not only verify that a rare event and social media activities have strong correlations, but also reveal that they have some time difference. Thus, it is conducive to investigate the temporal pattern of social media activities. Secondly, fuzzy logic-based feature extraction and classification methods are effective in identifying event-related and unrelated messages. Lastly, the Adaptive MV method deduces the ground truth well and performs better on datasets with noisy labels than other two methods, Positive Label Frequency Threshold and Majority Voting

On the study of crowdsourced labelled data and annotators: beyond noisy labels.

123 p.La presente tesis incluye 3 contribuciones al área llamada "learning from crowds", que estudia los métodos de aprendizaje basados en datos etiquetados por medio del "crowdsourcing". Estas etiquetas se caracterizan por tener una incertidumbre asociada debido a que la fiabilidad de las personas anotadoras no está garantizada. En primer lugar, se propone un nuevo método de "label aggregation", llamado"domain aware voting", una extensión del popular y simple método "majority voting" que tiene en cuenta la variable descriptiva, obteniendo resultados mejores especialmente cuando hay una mayor escasez de etiquetas. La segunda contribución consiste en la propuesta de un nuevo marco de etiquetado, "candidate labelling", que permite a las personas anotadoras expresar sus dudas acerca de las etiquetas que otorgan,pudiendo otorgar varias etiquetas a cada instancia. Se proponen 2 métodos de "label aggregation"asociados a este tipo de etiquetado, y se muestra, mediante un marco experimental que aúna el etiquetado tradicional y el propuesto, que el "candidate labelling" consigue extraer más información con un mismo número de personas anotadoras. Por último, se desarrolla un modelo de persona anotadora y 2 métodos de aprendizaje adaptados a este nuevo etiquetado, basados en el algoritmo EM, que obtienen mejores resultados en general que los métodos análogos en el marco de etiquetado tradicional.bcam Excelencia Severo Ocho