Sentiment and behaviour annotation in a corpus of dialogue summaries

This paper proposes a scheme for sentiment annotation. We show how the task can be made tractable by focusing on one of the many aspects of sentiment: sentiment as it is recorded in behaviour reports of people and their interactions. Together with a number of measures for supporting the reliable application of the scheme, this allows us to obtain sufficient to good agreement scores (in terms of Krippendorf's alpha) on three key dimensions: polarity, evaluated party and type of clause. Evaluation of the scheme is carried out through the annotation of an existing corpus of dialogue summaries (in English and Portuguese) by nine annotators. Our contribution to the field is twofold: (i) a reliable multi-dimensional annotation scheme for sentiment in behaviour reports; and (ii) an annotated corpus that was used for testing the reliability of the scheme and which is made available to the research community

Annotating for Hate Speech: The MaNeCo Corpus and Some Input from Critical Discourse Analysis

This paper presents a novel scheme for the annotation of hate speech in corpora of Web 2.0 commentary. The proposed scheme is motivated by the critical analysis of posts made in reaction to news reports on the Mediterranean migration crisis and LGBTIQ+ matters in Malta, which was conducted under the auspices of the EU-funded C.O.N.T.A.C.T. project. Based on the realization that hate speech is not a clear-cut category to begin with, appears to belong to a continuum of discriminatory discourse and is often realized through the use of indirect linguistic means, it is argued that annotation schemes for its detection should refrain from directly including the label 'hate speech,' as different annotators might have different thresholds as to what constitutes hate speech and what not. In view of this, we suggest a multi-layer annotation scheme, which is pilot-tested against a binary +/- hate speech classification and appears to yield higher inter-annotator agreement. Motivating the postulation of our scheme, we then present the MaNeCo corpus on which it will eventually be used; a substantial corpus of on-line newspaper comments spanning 10 years.Comment: 10 pages, 1 table. Appears in Proceedings of the 12th edition of the Language Resources and Evaluation Conference (LREC'20

Predicting continuous conflict perception with Bayesian Gaussian processes

Conflict is one of the most important phenomena of social life, but it is still largely neglected by the computing community. This work proposes an approach that detects common conversational social signals (loudness, overlapping speech, etc.) and predicts the conflict level perceived by human observers in continuous, non-categorical terms. The proposed regression approach is fully Bayesian and it adopts Automatic Relevance Determination to identify the social signals that influence most the outcome of the prediction. The experiments are performed over the SSPNet Conflict Corpus, a publicly available collection of 1430 clips extracted from televised political debates (roughly 12 hours of material for 138 subjects in total). The results show that it is possible to achieve a correlation close to 0.8 between actual and predicted conflict perception

Interpersonal stance in police interviews: content analysis

A serious game for learning the social skills required for effective police interviewing is a challenging idea. Building artificial conversational characters that play the role of a suspect in a police interrogation game requires computational models of police interviews as well as of the internal psychological mechanisms that determine the behaviour of suspects in this special type of dialogues. Leary's interactional circumplex is used in police interview training as a theoretical framework to understand how suspects take stance during an interview and how this is related to the stance and the strategy that the interviewer takes. Interactional stance is a fuzzy notion. The question that we consider here is whether different observers of police nterviews agree on the type of stance that suspect and policemen take and express in a face-to-face interview. We analyzed police interviews and report about a stance annotation exercise. We conclude that although inter-annotator agreement on stance labeling on the level of speech segments is low, a majority voting meta-annotator" is able to reveal the important dynamics in stance taking in a police interview. Then we explore the relation between the stance taken by the suspect and turn-taking behaviour, overlaps, interruptions, pauses and silences. Our findings contribute to building computational models of non-player characters that allow more natural turn-taking behaviour in serious games instead of the one-at-a-time regime in interview training games

Interpersonal stance in police interviews: content analysis

A serious game for learning the social skills required for effective police interviewing is a challenging idea. Building artificial conversational characters that play the role of a suspect in a police interrogation game requires computational models of police interviews as well as of the internal psychological mechanisms that determine the behaviour of suspects in this special type of dialogues. Leary's interactional circumplex is used in police interview training as a theoretical framework to understand how suspects take stance during an interview and how this is related to the stance and the strategy that the interviewer takes. Interactional stance is a fuzzy notion. The question that we consider here is whether different observers of police nterviews agree on the type of stance that suspect and policemen take and express in a face-to-face interview. We analyzed police interviews and report about a stance annotation exercise. We conclude that although inter-annotator agreement on stance labeling on the level of speech segments is low, a majority voting meta-annotator" is able to reveal the important dynamics in stance taking in a police interview. Then we explore the relation between the stance taken by the suspect and turn-taking behaviour, overlaps, interruptions, pauses and silences. Our findings contribute to building computational models of non-player characters that allow more natural turn-taking behaviour in serious games\ud instead of the one-at-a-time regime in interview training games

The recognition of acted interpersonal stance in police interrogations and the influence of actor proficiency

This paper reports on judgement studies regarding the perception of interpersonal stances taken by humans playing the role of a suspect in a police interrogation setting. Our project aims at building believable embodied conversational characters to play the role of suspects in a serious game for learning interrogation strategies. The main question we ask is: do human judges agree on the way they perceive the various aspects of stance taking, such as friendliness and dominance? Four types of stances were acted by eight amateur actors. Short recordings were shown in an online survey to subjects who were asked to describe them using a selection of a number of adjectives. Results of this annotation task are reported in this paper. We explain how we computed the inter-rater agreement with Krippendorff’s alpha statistics using a set theoretical distance metric. Results show that for some of the stance types observers agreed more than for others. Some actors are better than others, but validity (recognizing the intended stance) and inter-rater agreement do not always go hand in hand. We further investigate the effect the expertise of actors has on the perception of the stance that is acted. We compare the fragments from amateur actors to fragments from professional actors taken from popular TV-shows

University Students’ Perceptions of Videotaping as a Teaching Tool in a Public Speaking Course

The purpose of this study was to investigate the effect of videotaping on college students’ public speaking skills development in English as a Foreign Language (EFL) from the learners’ perspective. Twenty Moroccan freshmen students majoring in Engineering, Business, and Humanities at Al Akhawayn University in Ifrane, Morocco, participated in the study. Using 60 videotaped extemporaneous speeches, pre and post-videotaping surveys together with self-reflection essays, the researcher reports on students’ perceptions of and attitudes towards the effectiveness of videotaping on their public speaking competence development. Results revealed that the students’ public speaking skills improved over the course of a semester in terms of content, followed by non-verbal communication, verbal communication, organization, and language. In line with some previous research, this study confirms that a combination of videotaping and self-reflection has a major effect on improving students' public speaking skills, developing confidence of EFL learners, and fostering independent learnin