Exploiting `Subjective' Annotations

Many interesting phenomena in conversation can only be annotated as a subjective task, requiring interpretative judgements from annotators. This leads to data which is annotated with lower levels of agreement not only due to errors in the annotation, but also due to the differences in how annotators interpret conversations. This paper constitutes an attempt to find out how subjective annotations with a low level of agreement can profitably be used for machine learning purposes. We analyse the (dis)agreements between annotators for two different cases in a multimodal annotated corpus and explicitly relate the results to the way machine-learning algorithms perform on the annotated data. Finally we present two new concepts, namely `subjective entity' classifiers resp. `consensus objective' classifiers, and give recommendations for using subjective data in machine-learning applications.\u

Subjective Annotations for Vision-Based Attention Level Estimation

Attention level estimation systems have a high potential in many use cases, such as human-robot interaction, driver modeling and smart home systems, since being able to measure a person's attention level opens the possibility to natural interaction between humans and computers. The topic of estimating a human's visual focus of attention has been actively addressed recently in the field of HCI. However, most of these previous works do not consider attention as a subjective, cognitive attentive state. New research within the field also faces the problem of the lack of annotated datasets regarding attention level in a certain context. The novelty of our work is two-fold: First, we introduce a new annotation framework that tackles the subjective nature of attention level and use it to annotate more than 100,000 images with three attention levels and second, we introduce a novel method to estimate attention levels, relying purely on extracted geometric features from RGB and depth images, and evaluate it with a deep learning fusion framework. The system achieves an overall accuracy of 80.02%. Our framework and attention level annotations are made publicly available.Comment: 14th International Conference on Computer Vision Theory and Application

Gestures as an interface of performers’ intentionality : a case study of Western Embodiment of Karnatic Music in piano performance

Background Music performance has a strong corporeal dimension, involving different types of gestures (technical gestures, expressive gestures, etc.) that performers employ to transform a written score into live music (Leman 2007). This transformation is based on the musical intentions that arise from the performers’ personal interpretation of the composition as an outcome of their artistic praxis, which leads to decisions on how to play the music in terms of its structure, articulation of the phrases, dynamics, timber and the necessary motor strategies to realize these decisions in the sounding music. Aims This research wants to investigate musical gestures as an interface of performers’ intentionality, i.e. an outcome of the artistic praxis and the process of embodiment, in the light of the recent theories on enactment and embodied music cognition (Leman 2016). For this reason, we considered a case study based on the interpretation of a piece that includes the acquisition and embodiment of musical knowledge quite knew to the performer in order to map the modifications in the corporeal engagement from an intuitive approach to a conscious approach. The composition chosen was a contemporary piano piece based on a non-western music tradition: the Karnatic modes from South India. Method To assist the performer in (re)framing the phases of her artistic process, a methodology, called performer based analysis method (Caruso et al 2016), was developed to establish also the procedures of a performative experiment where the performance of the 8th cycle from the 72 Etudes Karnatiques pour piano by Jacques Charpentier (b.1933) was taken as a case study. The performative experiment required a period of preparation, which concerns the performer/researcher’s artistic praxis (to embody the piece) and the self-observation of a video recording archive of her performances in order to map and describe the artistic praxis. The pianist conducted a musicological research on the influence of Indian music in the French contemporary piano repertoire to enrich specifically her current competences in Karnatic Music and had a collaborative three years practice with the composer and with two experts in Karnatic music, a singer and a dancer (see the Re-Orient project: http://re-orient.wixsite.com/indiandream). A retrospective thinking-aloud procedure (Van den Haak & De Jong, 2003) was used during the experiment to allow the performer in rendering explicit and systematic the artistic reflections. The experiment was recorded by a video camera, a microphone and the Motion Capture System. Results To catch the development between the initial intuitive performance and the final embodied performance, two recordings of one fragment from the piece played with these two different approaches (intuitive and conscious) were compared. The analysis of these fragments was based on an alignment between qualitative data acquired through subjective descriptions - based on a performance model and a score annotation - and quantitative data (objective measurements) produced by the audio-video and motion capture recordings. The qualitative and quantitative data (audio and video) were processed through the ELAN software. Gestural similarities and differences between the intuitive and conscious versions were detected by comparing the kinematic and audio measurements (quantitative data) with the performer’s subjective annotations (qualitative data) concerning the motor strategies and the interpretative cues. The results show a different corporeal engagement of the pianist related to the different intentions through a parallel configuration of these two different subjective and objective layers. Conclusions The actual investigation wants to present musical gestures as vehicles of idiosyncratic intentions and expressions by linking performers’ corporeal engagement to the embodiment of their interpretation in order to better understand the connection between musical intentions, goal actions and sound. The role of the technology-mediated approach (thirdperson’s perspective) gives the opportunity to study, as in a mirror-like tool, some aspects, which imply the performer’s subjective involvement (first-person’s perspective). This method provides specifically to musicians/researchers an easier access to music performance analysis. Furthermore, with the implementation of a transdisciplinary and collaborative practice (with the composer, the Indians singer and dancer) plus the aid of technology with the motion and audio analysis, the actual study adds an alternative perspective concerning the exploration and the design of new paths within the field of artistic research

How Crowd Worker Factors Influence Subjective Annotations: A Study of Tagging Misogynistic Hate Speech in Tweets

Crowdsourced annotation is vital to both collecting labelled data to train and test automated content moderation systems and to support human-in-the-loop review of system decisions. However, annotation tasks such as judging hate speech are subjective and thus highly sensitive to biases stemming from annotator beliefs, characteristics and demographics. We conduct two crowdsourcing studies on Mechanical Turk to examine annotator bias in labelling sexist and misogynistic hate speech. Results from 109 annotators show that annotator political inclination, moral integrity, personality traits, and sexist attitudes significantly impact annotation accuracy and the tendency to tag content as hate speech. In addition, semi-structured interviews with nine crowd workers provide further insights regarding the influence of subjectivity on annotations. In exploring how workers interpret a task - shaped by complex negotiations between platform structures, task instructions, subjective motivations, and external contextual factors - we see annotations not only impacted by worker factors but also simultaneously shaped by the structures under which they labour.Comment: Accepted to the 11th AAAI Conference on Human Computation and Crowdsourcing (HCOMP 2023

Crowdsourcing subjective annotations using pairwise comparisons reduces bias and error compared to the majority-vote method

How to better reduce measurement variability and bias introduced by subjectivity in crowdsourced labelling remains an open question. We introduce a theoretical framework for understanding how random error and measurement bias enter into crowdsourced annotations of subjective constructs. We then propose a pipeline that combines pairwise comparison labelling with Elo scoring, and demonstrate that it outperforms the ubiquitous majority-voting method in reducing both types of measurement error. To assess the performance of the labelling approaches, we constructed an agent-based model of crowdsourced labelling that lets us introduce different types of subjectivity into the tasks. We find that under most conditions with task subjectivity, the comparison approach produced higher $f_1$ scores. Further, the comparison approach is less susceptible to inflating bias, which majority voting tends to do. To facilitate applications, we show with simulated and real-world data that the number of required random comparisons for the same classification accuracy scales log-linearly $O(N \log N)$ with the number of labelled items. We also implemented the Elo system as an open-source Python package.Comment: Accepted for publication at ACM CSCW 202

Using EEG-validated Music Emotion Recognition Techniques to Classify Multi-Genre Popular Music for Therapeutic Purposes

Music is observed to possess significant beneficial effects to human mental health, especially for patients undergoing therapy and older adults. Prior research focusing on machine recognition of the emotion music induces by classifying low-level music features has utilized subjective annotation to label data for classification. We validate this approach by using an electroencephalography-based approach to cross-check the predictions of music emotion made with the predictions from low-level music feature data as well as collected subjective annotation data. Collecting 8-channel EEG data from 10 participants listening to segments of 40 songs from 5 different genres, we obtain a subject-independent classification accuracy for EEG test data of 98.2298% using an ensemble classifier. We also classify low-level music features to cross-check music emotion predictions from music features with the predictions from EEG data, obtaining a classification accuracy of 94.9774% using an ensemble classifier. We establish links between specific genre preference and perceived valence, validating individualized approaches towards music therapy. We then use the classification predictions from the EEG data and combine it with the predictions from music feature data and subjective annotations, showing the similarity of the predictions made by these approaches, validating an integrated approach with music features and subjective annotation to classify music emotion. We use the music feature-based approach to classify 250 popular songs from 5 genres and create a musical playlist application to create playlists based on existing psychological theory to contribute emotional benefit to individuals, validating our playlist methodology as an effective method to induce positive emotional response

On Interpretability of Deep Learning based Skin Lesion Classifiers using Concept Activation Vectors

Deep learning based medical image classifiers have shown remarkable prowess in various application areas like ophthalmology, dermatology, pathology, and radiology. However, the acceptance of these Computer-Aided Diagnosis (CAD) systems in real clinical setups is severely limited primarily because their decision-making process remains largely obscure. This work aims at elucidating a deep learning based medical image classifier by verifying that the model learns and utilizes similar disease-related concepts as described and employed by dermatologists. We used a well-trained and high performing neural network developed by REasoning for COmplex Data (RECOD) Lab for classification of three skin tumours, i.e. Melanocytic Naevi, Melanoma and Seborrheic Keratosis and performed a detailed analysis on its latent space. Two well established and publicly available skin disease datasets, PH2 and derm7pt, are used for experimentation. Human understandable concepts are mapped to RECOD image classification model with the help of Concept Activation Vectors (CAVs), introducing a novel training and significance testing paradigm for CAVs. Our results on an independent evaluation set clearly shows that the classifier learns and encodes human understandable concepts in its latent representation. Additionally, TCAV scores (Testing with CAVs) suggest that the neural network indeed makes use of disease-related concepts in the correct way when making predictions. We anticipate that this work can not only increase confidence of medical practitioners on CAD but also serve as a stepping stone for further development of CAV-based neural network interpretation methods.Comment: Accepted for the IEEE International Joint Conference on Neural Networks (IJCNN) 202